A data management method and program product

Abstract

The application provides a data management method and a program product. Binary log files are generated by calculating detailed data, and the binary log files are stored in preset directory storage locations. Data of each binary log file is read from the directory storage locations in sequence, and the read data of the binary log files is stored in a buffer corresponding to the binary log files, and then the binary log file data in the buffer is stored in an embedded database. While improving read-write performance, the application has good scalability and universality.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data management method and a program product. Background Technology

[0002] In application scenarios that address the traceability, interpretability, and complete recording of intermediate computation processes in large-scale computing tasks, existing technologies generally employ a method of writing detailed computation process data into a server-type database in a structured manner to form a complete computation process record, thereby enabling subsequent visualization analysis, problem reproduction, and anomaly investigation.

[0003] However, when existing technologies are applied in low-latency scenarios, they need to record a large number of intermediate results and place high demands on storage performance, query performance, and scalability. Therefore, how to improve the storage performance, query performance, and scalability of large-scale data in low-latency scenarios has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a data management method and program product to address the shortcomings of the prior art, thereby solving the problems of poor storage performance, query performance, and scalability of large-scale data in low-latency scenarios in the prior art.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] Firstly, this application provides a data management method, the method comprising:

[0007] Upon receiving a data calculation request, data to be written is generated based on the data calculation request;

[0008] At least one binary log file is generated based on the data to be written, and the at least one binary log file is stored in a preset directory storage location;

[0009] Data from each of the binary log files is read sequentially from the directory storage location, and the read binary log file data is stored in the buffer corresponding to the binary log file;

[0010] If the amount of data in the buffer reaches a preset data volume threshold, the data of the binary log file in the buffer will be written to the embedded database.

[0011] Optionally, generating at least one binary log file based on the data to be written includes:

[0012] Determine at least one file generation thread corresponding to the data to be written based on the business type of the data to be written;

[0013] Each file generation thread writes the data to be written into a file sequentially according to the order of the data to be written, thereby obtaining at least one binary log file.

[0014] Optionally, reading data from each of the binary log files from the directory storage location includes:

[0015] Based on a preset periodic scan of the directory storage location, if a new target binary log file exists in the directory storage location, the data of the target binary log file is read by a file reading thread, and the data of the target binary log file is stored in the buffer corresponding to the target binary log file.

[0016] Optionally, the step of reading data from the target binary log file via a file reading thread and storing the data of the target binary log file in the buffer corresponding to the target binary log file includes:

[0017] Create a target state machine for the target binary log file, wherein the initial state of the target state machine is the writing state;

[0018] If there are any newly added binary log files after the target binary log file, then the state of the target state machine is set to the write-complete state;

[0019] Read the data of the target binary log file sequentially. If the currently read data is a preset termination identifier, then set the state of the target state machine to the read complete state.

[0020] If the target state machine is in the read complete state, then the target binary log file is deleted from the directory storage location.

[0021] Optionally, when reading data from the target binary log file sequentially, the method further includes:

[0022] If the target binary log file reading fails, the state of the target state machine is set to an abnormal state, and the target binary log file is stored in a preset data storage location.

[0023] Optionally, writing the binary log file data in the buffer to the embedded database includes:

[0024] The binary log file data in the buffer is parsed to obtain the data to be written.

[0025] The data to be written is fragmented to obtain at least one fragment of data;

[0026] Determine the target write thread corresponding to each of the data fragments;

[0027] Each of the target write threads writes the fragmented data into the embedded database.

[0028] Optionally, parsing the binary log file data in the buffer to obtain the data to be written includes:

[0029] The binary log file data is parsed to obtain parsed data;

[0030] Write the parsed data into the file before writing;

[0031] When the file before writing meets the preset conditions, the parsed data in the file before writing is used as the data to be written.

[0032] Optionally, determining the target write thread corresponding to each of the data shards includes:

[0033] The fragment number of each fragment data is determined based on the service sequence number of each fragment data.

[0034] The write thread corresponding to each of the aforementioned shard numbers is used as the target write thread corresponding to the shard data.

[0035] Optionally, the method further includes:

[0036] Receive a data query request, parse the data query request, and obtain data query parameters, which include: the date and business sequence number of the data to be queried;

[0037] The data to be queried is obtained from the embedded database based on the data query parameters.

[0038] Optionally, obtaining the data to be queried from the embedded database according to the data query parameters includes:

[0039] Based on the date of the data to be queried, at least one candidate shard data is determined in the embedded database;

[0040] Based on the service sequence number, determine the target shard data from the candidate shard data;

[0041] The data to be queried corresponding to the data query parameters is obtained from the target fragment data.

[0042] Secondly, embodiments of this application provide a data management device, including:

[0043] The generation module is used to generate data to be written based on the data calculation request when a data calculation request is received;

[0044] Used to generate at least one binary log file based on the data to be written, and to store the at least one binary log file in a preset directory storage location;

[0045] The reading module is used to sequentially read data from each of the binary log files from the directory storage location, and store the read binary log file data in the buffer corresponding to the binary log file;

[0046] The writing module is used to write the data of the binary log file in the buffer to the embedded database when the data volume in the buffer reaches a preset data volume threshold.

[0047] Thirdly, embodiments of this application provide an electronic device, including:

[0048] Memory, used to store one or more programs;

[0049] processor;

[0050] When the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0051] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0052] Fifthly, embodiments of this application provide a program product that, when executed by a processor, implements the method as described in any one of the first aspects above.

[0053] Compared to existing technologies, the data management method and program product provided in this application reduce data encoding and decoding overhead and improve data read and write performance by storing computational detail data as a binary log file. Sequential reading and writing of the binary log file solves the performance degradation problem caused by large amounts of random reading and writing. Employing an embedded database also reduces network I / O, improves data writing and querying performance, and since the embedded database supports standard SQL queries, this application does not require modification of the database source code when adding new business type data tables; instead, only the data table configuration information needs to be added. Therefore, it also has good scalability and versatility.

[0054] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This diagram illustrates an application scenario of a risk control calculation system.

[0057] Figure 2 This diagram illustrates the architecture of a data management method provided by an embodiment of the present invention applied in a financial risk control system.

[0058] Figure 3 A flowchart illustrating a data management method provided by an embodiment of the present invention is shown;

[0059] Figure 4 This diagram illustrates a process for generating binary log files according to an embodiment of the present invention.

[0060] Figure 5 A schematic diagram of the architecture of threads and embedded database in a calculation details plugin is shown;

[0061] Figure 6 This invention provides a schematic diagram of a process for reading a binary log file according to an embodiment of the present invention.

[0062] Figure 7 This diagram illustrates the state transitions of a file state machine according to an embodiment of the present invention.

[0063] Figure 8 This illustration shows a flowchart of writing data in a buffer to an embedded database according to an embodiment of the present invention;

[0064] Figure 9 This diagram illustrates a directory structure in an embedded database according to an embodiment of the present invention.

[0065] Figure 10 This diagram illustrates a flowchart of a method for querying target data according to an embodiment of the present invention.

[0066] Figure 11 This illustrates another flowchart for querying target data provided in an embodiment of the present invention;

[0067] Figure 12 This diagram illustrates the structure of a data management device 20 provided in an embodiment of the present invention.

[0068] Figure 13A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0071] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0072] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] To improve the traceability and interpretability of computation results and processes in large-scale computing tasks, existing technologies employ a method of writing intermediate variables and dependent data into a server-side database in a structured manner, forming a complete and queryable record of computation details. When data queries are needed on the front end, the server-side database can be queried using the business sequence number, thereby reconstructing the entire computation process.

[0074] To facilitate understanding, the following explanation uses a risk control calculation system in the financial industry as an example to illustrate the data management process. Figure 1 This is a schematic diagram of an application scenario for a risk control calculation system. The risk control calculation system communicates and connects with the transaction system and front-end devices. The risk control calculation system includes a risk control calculation engine, a server-type database, and a risk control management system.

[0075] Reference Figure 1The trading system can call the calculation interface provided by the risk control calculation system to trigger the pre-verification logic. After the risk control calculation engine completes a series of calculations, the intermediate process variables and dependent data are written to the server-type database through INSERT SQL (Structured Query Language) operations.

[0076] The front-end system initiates a query request based on the business sequence number to retrieve the corresponding calculation details data. At this time, the risk control management system accesses the database through standard SQL query statements to obtain complete calculation process data and returns it to the front-end interface for display.

[0077] However, in application systems that require recording large-scale intermediate process variables and their dependent data, continuous, high-frequency, and structured recording of intermediate process data is often necessary to ensure traceability, interpretability, and complete record-keeping of data processing. When processing such ultra-large-scale data, the aforementioned... Figure 1 The existing technology shown has many problems that need to be solved.

[0078] First, large-scale data computing scenarios place high demands on database storage capacity. Traditional server-based databases not only need to store the raw data, but also need to maintain a large amount of additional content for intermediate process data, resulting in a significant increase in actual space usage.

[0079] Secondly, in large-scale computing scenarios, some business operations have high-frequency computing modules with very high write rates, which requires the system to have a data persistence capability that is no less than the write rate. However, a large number of concurrent write operations can easily cause the device to fall into a write bottleneck, which may lead to excessive memory usage, excessive disk usage, or even data loss. In severe cases, it may even cause core modules to be blocked or the system to malfunction.

[0080] Third, low-latency applications have high requirements for real-time performance. However, traditional server-based databases are limited by index maintenance overhead and full table scan efficiency when dealing with extremely large table structures, making it difficult to achieve a low-latency, high-concurrency query experience.

[0081] Fourth, in the traditional architecture, if it is necessary to add new business dimensions or expand new detailed record tables, developers usually need to modify the code and redeploy the system. This approach is cumbersome, inflexible, and lacks versatility, and it also incurs high development costs.

[0082] Fifth, in some system architectures, intermediate process data is only written to log files and not synchronized to the database. Therefore, after the calculation task is completed, the front-end system cannot obtain relevant detailed data in real time, making it difficult to guarantee the timeliness of the system response.

[0083] In summary, when handling large-scale computing tasks, database storage capacity and write performance are prone to bottlenecks, and there are also problems such as limited query performance, insufficient system scalability and versatility, and difficulty in guaranteeing low-latency response.

[0084] In view of the problems existing in the above-mentioned technologies when dealing with large-scale computing tasks, in order to solve the impact of data processing on database storage capacity and computer performance, improve the scalability and versatility of the system, improve data query performance, and ensure low-latency response.

[0085] Specifically, Figure 2 This is a schematic diagram of the architecture of a data management method provided by an embodiment of the present invention applied in a financial risk control system.

[0086] Reference Figure 2 The risk control calculation engine performs data calculations based on instructions from the transaction system to obtain detailed calculation data. This application achieves a data persistence mechanism decoupled from the core logic by asynchronously writing the detailed calculation data to a binary log file. Simultaneously, by batch sequentially reading the binary log file and using multi-threaded calls to the embedded database interface, it achieves batch and efficient writing of structured data to a local database file. The front-end device reads the detailed calculation data from the embedded database through the risk control management system.

[0087] Among them, the risk control calculation system saves the calculated detailed data as a binary log file, compared to Figure 1 The calculation details are stored in a server-type database in text form. In the technical solution of this application, it is not necessary to convert binary data into text form for storage. Instead, it is stored directly in binary form, which reduces the consumption of data encoding and decoding and improves the performance of data reading and writing.

[0088] The calculation details plugin includes an embedded database. This embedded database supports queries using most standard SQL statements, enhancing the versatility of the risk control calculation system. Adding new data tables is straightforward; configuration information can be added directly to the calculation details plugin without modifying the system source code, demonstrating high scalability.

[0089] The embedded database SDK (Software Development Kit) can be compiled into the calculation details plugin. The risk control calculation system does not need to install any file data-related content separately, nor does it need to configure the database. Therefore, the hardware cost and operation and maintenance deployment cost are lower.

[0090] When the computational detail plugin calls the embedded database, it can use the embedded database's SDK to transform query and management requests into SQL query calls to the local embedded database. Current technologies typically implement database queries and management by calling dynamic link libraries, requiring the transmission of query requests and results over a network, which can introduce latency even within a local area network. However, by compiling the embedded database's SDK into the computational detail plugin, the plugin can directly call the in-memory data interface, eliminating the overhead of inter-process communication and protocol parsing, thus improving the performance of data query and management.

[0091] It should be understood that this application only provides an illustrative example of data management for large-scale computing tasks, using the business scenario of computational data management in a financial risk control system. However, the technical solutions of this application are not limited to this scenario. Based on the examples described below, those skilled in the art can apply the data management methods of this application to other data management systems without any inventive effort, and this application does not impose any restrictions here.

[0092] Combination Figure 2 The architecture used in the financial risk control system shown is as follows. Figure 3 This is a flowchart illustrating a data management method provided in an embodiment of the present invention, referred to below. Figure 3 The method includes:

[0093] Step 100: Upon receiving a data calculation request, generate the data to be written based on the data calculation request.

[0094] The data to be written includes detailed calculation data and result data generated when processing calculation tasks based on data calculation requests.

[0095] Combination Figure 2 The trading system can initiate a risk control call to the risk control calculation engine through the interface provided by the risk control calculation system, and pass in a data calculation request. The data calculation request includes the trading instructions and order information. The risk control calculation engine performs risk control calculations based on the data calculation request and generates data to be written.

[0096] Step 101: Generate at least one binary log file based on the data to be written, and store at least one binary log file in a preset directory storage location.

[0097] See Figure 2 After generating the data to be written, the risk control calculation engine can write the data to a binary file to obtain at least one binary log file.

[0098] The binary log file can have a pre-set file size limit. When the file size limit is exceeded, the data is written to a new binary file, and a new binary log file is generated.

[0099] It should be noted that the calculation of the data to be written in step 100 and the generation of the binary log file based on the data to be written in step 101 can be performed asynchronously. For example, the main thread executes step 100, while other threads execute step 101, so as not to affect the efficiency of data calculation and processing.

[0100] Step 102: Read the data of each binary log file sequentially from the directory storage location, and store the read binary log file data in the buffer corresponding to the binary log file.

[0101] The steps of reading and generating binary log files can be performed synchronously using different threads, thus decoupling data reading from data writing. For example, reading data from a binary log file can be done with a single thread, while generating the binary log file from the data to be written can be done in parallel using multiple threads.

[0102] Each binary log file has a corresponding buffer. Data read from the binary log file can be temporarily stored in the buffer. If the buffer is not full after a single read, data can continue to be read from the binary log file into the buffer until the buffer is full.

[0103] Step 103: If the amount of data in the buffer reaches the preset data volume threshold, the binary log files in the buffer are written to the embedded database in parallel through multiple threads.

[0104] The data volume threshold of the buffer can be determined based on the data volume of the binary log file. Different binary log files have different data volumes, so their corresponding buffer data volume thresholds are also different.

[0105] The correspondence between buffers and binary log files can be one binary log file corresponding to one buffer, or one binary log file of the same business type corresponding to one buffer, or all binary log files corresponding to one buffer.

[0106] When a buffer is provided for binary log files corresponding to the same type of business data, if multiple binary log files are provided for the same type of business data, the data of each binary log file can be read into the buffer according to the file number or the order of file generation time.

[0107] After reading the data from the binary log files into the buffer, the size of the buffer corresponding to each binary log file can be determined, and the data from each buffer can be written into the embedded database in sequence.

[0108] The data management method provided in this application reduces data encoding / decoding overhead and improves data read / write performance by storing computational detail data in binary log files. Sequential reading and writing of the binary log files solves the performance degradation caused by large amounts of random reading and writing. Employing an embedded database also reduces network I / O, improves data writing and querying performance, and because the embedded database supports standard SQL queries, this application does not require modification of the database source code when adding new business type data tables; only the data table configuration information needs to be added. Therefore, it also has good scalability and versatility.

[0109] The following provides a possible implementation method for generating binary log files. Figure 3 On this basis, Figure 4 This is a schematic diagram of a process for generating a binary log file according to an embodiment of the present invention. See also: Figure 4 Step 101, specifically:

[0110] Step 101-1: Determine at least one file generation thread corresponding to the data to be written based on the business type of the data to be written.

[0111] In one possible implementation, binary files can be written at the table level, and binary log files can be generated. Each business type can be processed by a separate file generation thread for data writing. When multiple business types exist, the file generation threads can perform data writing in parallel.

[0112] Each type of business data corresponds to a table. Each table is written to by a file generation thread. The file size can be a preset value. After all the data to be written is written to the file, at least one binary log file can be generated.

[0113] Step 101-2: The file generation threads write the data to be written to the files in the order of the data to be written, so as to obtain at least one binary log file.

[0114] If the detailed data includes a type of business data, a single file generation thread writes the data to be written to the file in ascending order, resulting in at least one binary log file corresponding to the data to be written.

[0115] For example, if the business data unit stock is a holdings table, the business data unit stock can be written to a file. The maximum size of each file is a preset value (e.g., 500M). If the file size exceeds the maximum size, a new file will be generated, and the file sequence number will be incremented. When the amount of business data is very large, multiple binary log files can be obtained, such as unitstock_1.log, unitstock_2.log, unitstock_3.log, and unitstock_4.log.

[0116] If the detailed data includes multiple types of business data, it can be written concurrently by multiple threads in a thread pool. Each file writing thread is used to write one type of business data to a binary file, resulting in at least one binary log file corresponding to each type of data in the data to be written.

[0117] It should be noted that conventional log files, such as text files in UTF-8 format, require the variables in memory to be converted to UTF-8 encoding before writing the text file, which incurs some performance overhead. However, in this embodiment of the invention, by writing the data to be written sequentially to obtain a binary log file, the step of transcoding the data to be written is omitted, thus improving system performance during the data writing process.

[0118] Step 102, specifically:

[0119] Step 102-1: Scan the directory storage location based on a preset period. If a new target binary log file exists in the directory storage location, read the data of the target binary log file through the file reading thread and store the data of the target binary log file in the buffer corresponding to the target binary log file.

[0120] The target binary log file can be a newly added binary log file in the directory storage location.

[0121] See Figure 2 After startup, the calculation details plugin can periodically scan the directory storage location to determine whether there are any newly added binary log files in the directory storage location.

[0122] In one possible implementation, the calculation details plugin can record the file number of the last binary log file in the directory storage location. When the file number of the latest file in the directory storage location is different from the previously saved file number, it can be considered that a new binary log file has appeared in the directory storage location.

[0123] Figure 5 This is a schematic diagram of the architecture of threads and an embedded database in a calculation detail plugin. See also... Figure 5The threads include file reading threads, a file insertion thread pool, and data query threads. The file reading thread can read data from the binary log file and send the read data to the file insertion thread pool, which then inserts the data into the embedded database.

[0124] The file reading thread is used to read binary log file data, and the file writing thread pool is used to write binary log file data to the embedded database, thereby decoupling binary log file reading from data writing to the database.

[0125] When reading data from a binary log file using a file reading thread, the reading process can be managed using a binary file state machine.

[0126] The following provides a possible implementation method for reading data from a binary log file. Figure 3 On this basis, Figure 6 This is a schematic diagram of a process for reading a binary log file according to an embodiment of the present invention. See also... Figure 6 Step 102-1 is as follows:

[0127] Step 102-10: Create the target state machine for the target binary log file. The initial state of the target state machine is the writing state.

[0128] In one possible implementation, the file state of the binary log file during the reading process can be recorded using a file state machine, with each binary log file corresponding to a file state machine. Figure 7 This is a schematic diagram of the state transition of a file state machine provided in an embodiment of the present invention. The file states include writing state, writing completed state, reading completed state, and abnormal state.

[0129] If a new binary log file appears in the directory storage location, a target state machine for the target binary file can be created, and the initial state of the target state machine can be set to the writing state.

[0130] Step 102-11: If there are any newly added binary log files after the target binary log file, then set the state of the target state machine to the write-complete state.

[0131] In one possible implementation, the binary log files generated from data to be written for the same business type have the same name and are arranged in ascending order of file number.

[0132] If a binary log file with the same name as the target binary log file and a larger sequence number exists in the directory storage location, the state of the target binary log file's state machine can be modified to the write-complete state.

[0133] If no binary log file with a greater sequence number than the target binary log file exists in the directory storage location, the state of the target state machine will not be modified even if the data in the target binary log file has been completely read.

[0134] Steps 102-12: Read the data from the target binary log file sequentially. If the currently read data is a preset termination identifier, set the state of the target state machine to the read complete state.

[0135] The termination identifier is placed after the last data in the target binary log file, for example, it could be EOF.

[0136] The file reading thread can use a streaming approach, meaning the stream is not closed after each read from the target binary log file. After reading data, the offset is recorded, and subsequent reads continue based on this recorded offset until a termination marker is encountered, at which point the file stream is closed. Streaming the target binary log file can improve data reading efficiency.

[0137] Steps 102-13: If the target state machine is in the read complete state, then delete the target binary log file from the directory storage location.

[0138] See Figure 7 When the read data is a termination identifier, it means that the target binary log file has been read completely. At this time, the state of the target state machine can be changed to the read complete state, and the target binary log file can be deleted from the directory storage location to free up disk space in the directory storage location.

[0139] Steps 102-12 also include:

[0140] Steps 102-14: If the target binary log file reading fails, set the target state machine to the abnormal state and store the target binary log file in the preset data storage location.

[0141] The preset data storage location is used to store the target binary log file for data read errors.

[0142] See Figure 7 When reading the target binary log file, the target state machine may be in the writing state or the writing completed state. If an error occurs while reading the data, the target state machine can be modified to an abnormal state, and the target binary log file can be moved from the directory storage location to the data storage location.

[0143] In this embodiment of the invention, the data reading process of the binary log file is managed by a state machine, which can ensure the orderliness and coordination of the binary log file writing and data reading processes. By storing the binary log files in abnormal states separately in the data storage location, it is beneficial to improve the efficiency of subsequent problem investigation.

[0144] In one possible implementation, a configuration file can be pre-deployed in the computation details plugin. This configuration file indicates the directory storage location of the binary log files, the root directory of the embedded database, and the path to the computation details data entity description files. See the following plugin configuration XML code snippet, which is a code example of a configuration file in a computation details plugin.

[0145] <plugin lib="fsc_calcdetail" getinfo="GetCalcDetailInfo" load_level="2" thread_num="1" note="计算明细插件">

[0146] <args db_path=" / howe / risk / ufr / workspace / calcdetaildb" / / Root directory of the embedded database

[0147] log_path=" / home / risk / ufr / workspace / caltofile" / / Directory of the binary log file

[0148] table_filed_cfg="cal_todb_tabe.xml" / / Path of the calculation detail metadata entity description file

[0149] buffer_size_row_num="100" / / Buffer size for file reading

[0150] dir_polling_time="1000" / / Directory polling interval

[0151] <uftdb_info svr name="hsfund,***,core" svr idx="0" plugin id="hsuftdb" / >

[0152] < / plugin>

[0153] The embedded database's root directory stores the database files for computational detail data. The directory storing the binary log files contains the binary log files themselves. The computational detail data entity description file path stores the configuration information for each business type's computational detail data, including table name, table length, and table type. By adding new configuration information to the file within the computational detail entity description file path, automatic data entry and retrieval can be achieved without modifying the code.

[0154] See Figure 5 After the file reading thread reads the data from the binary log file into the buffer, the threads in the file writing thread pool can write the data in the buffer into the embedded database in parallel.

[0155] Optionally, since the calculation details maintained by this solution need to support SQL-based query requirements as much as possible, in one possible implementation, the aforementioned embedded database involved in the above example can be an SQLite database. The SQLite database is characterized by supporting most SQL queries, thereby enabling the various examples of this application to have good versatility.

[0156] The SQLite database is a relational database, specifically an embedded database. This SQLite database is lightweight. In one possible implementation, the SQLite database in this application can be implemented as follows: Compared to the usual approach of implementing the SQLite database by installing a dynamic link library, this application compiles the SQLite database source code and embeds it into the calculation detail plugins provided in the various examples of this application. This reduces the workload of system deployment and lowers maintenance costs. Furthermore, compiling the SQLite database source code into the calculation detail plugin allows for modification or version upgrades of the SQLite database source code, thereby improving database performance.

[0157] Optionally, in addition to using an SQLite database, to further improve the performance of writing data to the SQLite database, various examples in this application utilize binary log files to store the calculation details data. Referring to the foregoing embodiments, this application uses binary log files to store the calculation details data generated by the risk control calculation engine. Compared to directly writing the data to the SQLite database after encoding and decoding, this application first sequentially writes the calculation details data to the binary log file, then sequentially reads the binary log file, and finally writes it to the SQLite database, resulting in faster writing efficiency. Furthermore, using the binary log file as a buffer, compared to using memory for buffering, can prevent memory from being instantly filled in calculation scenarios with a large instantaneous data volume, thereby improving the stability of system operation.

[0158] Optionally, referring to the foregoing embodiments, the data calculation plugin and the risk control calculation engine can be independent of each other in terms of business logic. That is, the risk control calculation engine is responsible for data calculation, and the data calculation plugin is responsible for data storage, thereby decoupling data calculation from data storage and improving the versatility of the solution in this application. (Refer to...) Figure 5 After the risk control calculation engine generates detailed calculation data and writes it to a binary log file, the detailed calculation plugin handles the process of writing the detailed calculation data to an SQLite database. Therefore, in one possible implementation, the detailed calculation plugin of this application can also connect to other risk control calculation engines or other applications and provide the function of writing data to an SQLite database.

[0159] Furthermore, based on writing computational detail data using binary log files, this application can also utilize sharding to implement the function of writing data blocks of the binary log files in parallel to the SQLite database in the form of sharded data. For example... Figure 5 As shown, the data blocks of binary log file n are processed by sharding, and corresponding threads 7 to 15 are written in parallel, thereby improving the write performance of each binary log file.

[0160] The following is combined with Figure 5 This provides a possible implementation method for writing data from a buffer to an embedded database. Figure 3 On this basis, Figure 8 This is a schematic diagram of a process for writing data in a buffer to an embedded database, provided by an embodiment of the present invention.

[0161] Step 103, specifically:

[0162] Step 103-1: Parse the binary log file data in the buffer to obtain the data to be written.

[0163] Optionally, the configuration information for calculating detailed data also includes descriptive information, which indicates the parsing rules for binary log file data, including the actual meaning represented by each byte of data.

[0164] The binary log file is parsed based on the description information in the configuration information to obtain the calculation details data, which is then used as the data to be written.

[0165] In one possible implementation, the binary log file can be parsed, and the parsed result can be assembled into an INSERT SQL statement, which is then used as the data to be written.

[0166] Step 103-2: Perform fragmentation on the data to be written to obtain at least one fragment of data.

[0167] Optionally, the data to be written can be sharded based on its business sequence number, thus grouping data with the same business sequence number into the same shard. In actual business scenarios, detailed calculation data can be continuously added to the binary log file, thus continuously generating new data to be written. As long as the new data has the same business sequence number, it can be sharded into the same shard.

[0168] By sharding the data to be written, large table data can be split into multiple shards, thus reducing the amount of data that would otherwise need to store billions of rows to multiple smaller shards. Furthermore, multi-threaded concurrent writing of the data to be written can be used, thereby improving the performance of writing data to the embedded database.

[0169] Meanwhile, through the aforementioned data sharding mechanism, the data to be written can be split and stored in multiple shards. When performing data queries, compared to the original need to retrieve data from a large data table with billions of rows, this solution can first route to the shards and find the target data in the shards with smaller data volumes, effectively improving the performance of data queries.

[0170] Figure 9 This is a schematic diagram of a directory structure in an embedded database provided by an embodiment of the present invention. See also... Figure 9 The root directory contains multiple date subdirectories, each recording sharded data. Data shards for the same date can be grouped together based on their corresponding dates and dates, allowing each shard to be written to be stored in the same date directory. Each shard has its own dedicated thread responsible for writing it to the embedded database. For example... Figure 9 Thread 0 is responsible for writing shard data _0 to the embedded database, and thread 1 is responsible for writing shard data _1 to the embedded database.

[0171] The number of data fragments to be written can be determined based on the amount of data to be written, or it can be preset based on actual business needs. No specific method is restricted here.

[0172] Step 103-3: Determine the target write thread corresponding to each data shard.

[0173] Step 103-4: Each target write thread writes each shard of data into the embedded database.

[0174] The file input thread pool includes multiple file writing threads, and the target writing thread can be any file writing thread in the file input thread pool.

[0175] In one possible implementation, the file insertion thread pool can allocate a target write thread for each data shard, and each target write thread is used to write each data shard into the embedded database.

[0176] In another possible implementation, the fragmented data can be grouped into multiple fragmented groups, and a target write thread can be assigned to each fragmented group to write the fragmented data in the fragmented group into the embedded database.

[0177] In this embodiment of the invention, by performing data fragmentation on the data to be written and allocating a corresponding thread to each fragment for data writing, and writing each fragment to the embedded database through the target writing thread corresponding to each fragment, a single file can be split into multiple sub-files that can be written to the embedded database concurrently. This transforms the single-threaded writing of a single file into the concurrent writing of multiple sub-files, breaking through the upper limit of single file writing performance and improving the performance of data writing to the embedded database.

[0178] Steps 103-1 to 103-4 above are one implementation method given in the embodiment of the present invention. It should be understood that data parsing, fragmentation processing and data writing processing can also be combined in various ways in specific implementation.

[0179] As one possible implementation, when performing steps 103-1 to 103-4 above, the binary log file can be first segmented by the file reading thread to obtain the data of each segment, and the data of each segment can be bound to the target writing thread in the file writing thread pool. At this time, the target writing thread first parses the data of each segment to obtain the parsed data of each segment, and then writes the parsed data of each segment into the embedded database.

[0180] The following is combined with Figure 5 The overall process of the data management method in the embodiments of the present invention is described.

[0181] See Figure 5 After the trading system calls the risk control calculation engine to generate detailed calculation data, it can write the detailed calculation data to a binary log file, such as... Figure 5 The binary log file shown is 1, 2, ..., N, where N is a positive integer. The file reading thread reads the data from the binary log files sequentially and stores the read data in the corresponding buffer of each binary log file.

[0182] If the data in the buffer reaches a preset data volume threshold, the file reading thread can segment the data to obtain at least one data segment. A corresponding thread is created for each data segment, and each thread parses the data segment and writes the parsed data into the embedded database in parallel. For example... Figure 5 As shown, after the data block of binary log file n is divided into multiple data shards _0 to _7, each data shard can be parsed by threads 7 to 15 and then written to the embedded database in parallel.

[0183] In one possible implementation, the number of threads in the file insertion thread pool can be determined based on the number of shards. For example, for each shard of data generated, a thread corresponding to the shard of data is created, and the shard of data and the thread are bound together. The bound thread then parses the shard of data and performs data insertion processing.

[0184] For embedded databases, one possible implementation could be a file directory structure to store fragmented data for each binary log file. See also Figure 5For a root directory, it can store corresponding sharded data based on a date span using database files (DB files). For example, binary log file n obtained on April 29, 2025, has sharded data _0 to sharded data _7 read into a buffer. Then, by traversing this sharded data and obtaining the shard number of each shard according to the business sequence number (e.g., shard number = [risk_serial_no] % sharding_num, where risk_serial_no represents the business sequence number and sharding_num represents the number of shards), data with the same sequence number are assigned to the same group. Each group of sharded data is mapped to a corresponding thread through a Map, and finally, the thread performs a batch database insertion operation. For example, see [link to relevant documentation]. Figure 5 Thread 7 writes the sharded data _0 to the corresponding DB file of the embedded database. Figure 5 (Not shown in the text)

[0185] For the same day, there can be multiple points in time for writing sharded data. Therefore, in one possible implementation, threads can be bound to sharded data to ensure that the same sharded data (with the same sequence number) from different points in time can be written to the same database file by the same thread. This ensures that the data insertion operation for a single shard is ordered and that concurrent operations do not occur, thereby reducing errors.

[0186] Optionally, for writing sharded data from different days, the binding between the thread and the sharded data can be maintained. For example, in the data block of binary log file n on April 30, 2025, sharded data _0 and thread 7 still have a binding relationship. The specific method depends on the actual business scenario and is not limited here.

[0187] The file ingestion thread pool can manage threads based on the Least Recently Used (LRU) strategy. Until the entire binary log file is written, the threads corresponding to each data shard are retained in the file ingestion thread pool after creation. It should be understood that in real-world applications, the binary log file and each data shard will continuously receive new data. During this process, the threads bound to the data shards can continuously parse and ingest the data. When system resources are scarce, if the thread corresponding to a data shard has not processed the data for an extended period, the thread can be destroyed to release resources.

[0188] See Figure 5 When a thread writes sharded data to an embedded database, it can store the data based on the time of the sharded data, for example, by storing the sharded data according to the date.

[0189] In existing methods, data is typically synchronized to the database using the delete mode. Before each data write, a database write lock is acquired, and then the changed data is written to the .db file. This requires not only writing the data but also writing the data index.

[0190] In this embodiment of the invention, a WAL mechanism can be used to write data to the embedded database. Specifically, data from the binary log file can be first written to a .wal file, and then the data in the .wal file can be synchronized to the embedded database.

[0191] Table 1 shows a performance comparison between delete mode and WAL mechanism when writing data.

[0192] Table 1 Performance Comparison Table

[0193]

[0194] Referring to Table 1, the delete mode suffers from problems such as random data writing, poor concurrency performance, and severe read-write conflicts; for example, read operations can be blocked by write operations. In contrast, the WAL mechanism performs read and write operations asynchronously, resulting in better concurrency performance, and since data is written sequentially, its data writing performance is also better.

[0195] The following provides a possible implementation method for parsing binary log files.

[0196] Step 103-1, specifically:

[0197] Step 103-10: Parse the binary log file data to obtain the parsed data.

[0198] Step 103-11: Write the parsed data to the file before writing.

[0199] Step 103-12: When the file before writing meets the preset conditions, the parsed data in the file before writing is used as the data to be written.

[0200] The pre-write file can be a log file in an embedded database, used to record data change operations in the binary log file. The pre-write file includes transaction identifiers, data change records, and auxiliary information, such as checkpoint mechanisms and verification information.

[0201] When the data in the buffer reaches a preset data volume threshold, a transaction can be committed to trigger the data writing process to the embedded database and generate a transaction identifier. After parsing the binary log file data to obtain the parsed data, the corresponding data change records and transaction identifiers can be written to the pre-write file.

[0202] If the parsed data is a data write operation, the parsed data can be appended sequentially to the end of the file before writing.

[0203] If the parsing operation is a data read operation, then data can be read from the newly added content of the file before writing.

[0204] The pre-write file also includes a checkpoint mechanism to check the amount of data in the pre-write file and monitor whether the pre-write file meets the preset trigger conditions. If the amount of data in the pre-write file is greater than the preset threshold, or if the pre-write file meets the preset trigger conditions, the checkpoint mechanism is executed and the data in the pre-write file is written to the embedded database.

[0205] It should be noted that in steps 103-10 to 103-12 above, writing the parsed data to the pre-write file and the checkpoint mechanism can be implemented asynchronously, that is, the checkpoint mechanism does not block the normal reading and writing of parsed data.

[0206] In this embodiment of the invention, the parsed data is first written to a pre-write file, and then the parsed data is synchronized to the embedded database through the pre-write file. This not only decouples the writing of data to the embedded database from the reading and parsing of binary log file data, but also improves the database's concurrency performance and write efficiency.

[0207] The following provides a possible implementation method for determining the target write thread corresponding to each data shard.

[0208] Step 103-3, specifically:

[0209] Steps 103-30: Determine the fragment number of each fragment of data based on the service sequence number of each fragment.

[0210] Step 103-31: Use the write thread corresponding to each shard sequence number as the target write thread corresponding to the shard data.

[0211] The number of shards can be preset. By performing a modulo operation on the business sequence number of each shard and the number of shards, the modulo result can be used as the shard sequence number. Each shard corresponds to a write thread.

[0212] For example, if there are 8 shards, the data to be written can be split into 8 shards, and 8 write threads can write the data to the embedded database.

[0213] After writing the data from the binary log file to the embedded database based on the above steps, see [link to documentation]. Figure 2 and Figure 5 The front-end device can initiate a data query request through the risk control management system and obtain the target data from the embedded database through the data query thread in the calculation details plugin.

[0214] The following provides a possible implementation method for retrieving target data from an embedded database. Figure 10 This is a schematic diagram of a process for querying target data provided in an embodiment of the present invention.

[0215] After step 103, the data query process performed by the front-end device includes:

[0216] Step 104: Receive the data query request, parse the data query request, and obtain the data query parameters, which include the date and business sequence number of the data to be queried.

[0217] The embedded database categorizes and stores data according to date, and the sharding process determines the sharding number based on the data's business sequence number. Therefore, based on the date and business sequence number in the data query request, the storage location of the data to be queried in the embedded database can be determined.

[0218] See Figure 3 The front-end device can initiate a data query request through the risk control management system. The risk control management system converts the data query request into an RPC call to the calculation detail plugin. The calculation detail plugin then retrieves the data to be queried from the embedded database based on the data query parameters through a data query thread. [chofn-1]

[0219] Step 105: Based on the data query parameters, retrieve the data to be queried from the embedded database.

[0220] See Figure 5 and Figure 9 The embedded database stores data based on date and shard number. The data query thread can start from the root directory to query the data according to the date and business number of the data to be queried, determine the shard data corresponding to the data query parameters, and retrieve the data to be queried from the shard data.

[0221] The following provides a possible approach to retrieving target data based on data query parameters. Figure 10 On this basis, Figure 11 This is a schematic diagram of another process for querying target data provided in an embodiment of the present invention.

[0222] Step 105, specifically:

[0223] Step 105-1: Based on the date of the data to be queried, determine at least one candidate shard data in the embedded database.

[0224] See Figure 9 Based on the date of the data to be queried, the fragmented data under the date subdirectory corresponding to the date of the data to be queried in the embedded database can be used as the candidate fragmented data.

[0225] Step 105-2: Determine the target shard data from the candidate shard data according to the business sequence number.

[0226] Step 105-3: Query the data to be queried corresponding to the data query parameters in the target fragment data.

[0227] Performing the same processing as steps 103-30 above on the business sequence number can yield the shard sequence number, thereby using the candidate shard data indicated by the shard sequence number as the target shard data.

[0228] For example, the business sequence number and the number of shards can be moduloed, and the modulo result can be used as the shard sequence number. Each shard sequence number corresponds to a shard data. For example, the suffix of the shard data includes the shard sequence number. Therefore, the candidate shard data with the suffix including the shard sequence number can be used as the target shard data.

[0229] After determining the target data shard, the SQL query statement in the data query request can be executed in the target data shard to obtain the data to be queried corresponding to the data query parameters, and then the data to be queried can be returned to the front-end device.

[0230] Experiments have shown that the system performance is significantly improved when storing and querying detailed data based on the method of this application.

[0231] Taking writing 2 billion rows of data to a single table as an example, with a hardware configuration of 32 cores and 64GB of memory, when performing data queries based on the method of this application, the latency from initiating the query to returning the result can be controlled within 10ms, and the write performance can reach 400,000 rows / second.

[0232] By sharding the data to be written, data query performance and data writing performance are also greatly improved.

[0233] Specifically, taking a scenario of writing 2 billion rows of table data as an example, where the original requirement was to store the 2 billion rows of table data in a single file, the file can now be split into multiple sub-files, for example, based on... Figure 9 The architecture is split into 8 sub-files, each of which only needs to store 2 billion / 8 = 25 million rows of data. Originally, the write performance of writing 2 billion rows of data in a single thread was 50,000 rows / second. Now, by splitting the architecture, parallel write processing can be achieved by 8 threads, improving the write performance from 50,000 rows / second to 400,000 rows / second.

[0234] Taking a scenario with 2 billion rows of data as an example, what used to require searching through 2 billion rows of data can now be routed by date and shard number, requiring only a search through 25 million rows of data, thus greatly improving the performance of data query.

[0235] Based on the same inventive concept, this invention also provides a data management device corresponding to the data management method. Since the principle of the device in this invention is similar to that of the data management method described above, the implementation of the device can refer to the implementation of the method, and repeated details will not be repeated.

[0236] Figure 12 A schematic diagram of a data management device 20 provided in an embodiment of the present invention is shown. The device 20 includes a generation module 200, a reading module 201, and a writing module 202.

[0237] The generation module 200 is used to generate data to be written according to the calculation request when a data calculation request is received;

[0238] Generate at least one binary log file based on the data to be written, and store at least one binary log file in a preset directory storage location;

[0239] The reading module 201 is used to read data from each binary log file sequentially from the directory storage location and store the read binary log file data in the buffer corresponding to the data file;

[0240] The writing module 202 is used to write the data of the binary log file in the buffer to the embedded database when the amount of data in the buffer reaches a preset data amount threshold.

[0241] Optionally, the generation module 200 is specifically used for:

[0242] Determine at least one file generation thread corresponding to the data to be written based on the business type of the data to be written;

[0243] Each file generation thread writes the data to be written to the file sequentially according to the order of the data to be written, thus obtaining at least one binary log file.

[0244] Optionally, the reading module 201 is specifically used for:

[0245] Based on a preset periodic scan of the directory storage location, if a new target binary log file exists in the directory storage location, the data of the target binary log file is read by the file reading thread and stored in the buffer corresponding to the target binary log file.

[0246] Optionally, the reading module 201 is specifically used for:

[0247] Create the target state machine for the target binary log file. The initial state of the target state machine is the writing state.

[0248] If there are any new binary log files added after the target binary log file, then set the state of the target state machine to the write complete state;

[0249] Read the data from the target binary log file sequentially. If the currently read data is a preset termination identifier, set the state of the target state machine to the read complete state.

[0250] If the target state machine is in the read complete state, then the target binary log file is deleted from the directory storage location.

[0251] Optionally, the reading module 201 is also used for:

[0252] If the target binary log file reading fails, the target state machine is set to an abnormal state, and the target binary log file is stored in a preset data storage location.

[0253] Optionally, the write module 202 is specifically used for:

[0254] The binary log file data in the buffer is parsed to obtain the data to be written;

[0255] The data to be written is fragmented to obtain at least one fragment of data.

[0256] Determine the target write thread for each data shard;

[0257] Each target write thread writes the data of each shard to the embedded database.

[0258] Optionally, the write module 202 is specifically used for:

[0259] The binary log file data is parsed to obtain the parsed data;

[0260] Write the parsed data to the file before writing;

[0261] When the pre-write file meets the preset conditions, the parsed data in the pre-write file will be used as the data to be written.

[0262] Optionally, the write module 202 is specifically used for:

[0263] The shard number of each shard is determined based on the service sequence number of each shard.

[0264] The write thread corresponding to each shard sequence number is used as the target write thread for the shard data.

[0265] Optionally, the device further includes a query module for:

[0266] Receive data query requests, parse the data query requests, and obtain data query parameters, which include: the date of the data to be queried and the business sequence number;

[0267] Based on the data query parameters, the data to be queried is obtained from the embedded database.

[0268] Optionally, the query module is specifically used for:

[0269] Based on the date of the data to be queried, at least one candidate data shard is determined in the embedded database;

[0270] Based on the business sequence number, determine the target shard data from the candidate shard data;

[0271] The query data corresponding to the data query parameters is obtained from the target data shards.

[0272] The invention also provides an electronic device that can perform all the steps of the examples described above in the embodiments of the invention to achieve the corresponding technical effects. Specifically, Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. See also: Figure 13 The electronic device 30 includes: a memory 301 and a processor 300;

[0273] Memory 301 is used to store one or more programs;

[0274] Processor 300;

[0275] When one or more programs are executed by a processor, the electronic device 30 can achieve the steps and corresponding technical effects when it performs the steps shown in the above-described method examples.

[0276] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0277] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0278] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a program product. This program product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0279] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0280] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A data management method, characterized in that, include: Upon receiving a data calculation request, data to be written is generated based on the data calculation request, and the data to be written is business data; At least one binary log file is generated based on the data to be written, and the at least one binary log file is stored in a preset directory storage location. The binary log file stores the calculation details data output by the risk control calculation engine. Data from each of the binary log files is read sequentially from the directory storage location, and the read binary log file data is stored in the buffer corresponding to the binary log file; If the amount of data in the buffer reaches a preset data volume threshold, the data of the binary log file in the buffer will be written to the embedded database. The step of writing the binary log file data in the buffer to the embedded database includes: The binary log file data in the buffer is parsed to obtain the data to be written. The data to be written is fragmented to obtain at least one fragment of data, wherein data with the same service sequence number are in the same fragment of data. Determine the target write thread corresponding to each of the data fragments; Each of the target write threads writes the fragmented data into the embedded database.

2. The method according to claim 1, characterized in that, The process of generating at least one binary log file based on the data to be written includes: Determine at least one file generation thread corresponding to the data to be written based on the business type of the data to be written; Each file generation thread writes the data to be written into a file sequentially according to the order of the data to be written, thereby obtaining at least one binary log file.

3. The method according to claim 1, characterized in that, The step of reading data from each of the binary log files from the directory storage location includes: Based on a preset periodic scan of the directory storage location, if a new target binary log file exists in the directory storage location, the data of the target binary log file is read by a file reading thread, and the data of the target binary log file is stored in the buffer corresponding to the target binary log file.

4. The method according to claim 3, characterized in that, The step of reading data from the target binary log file via a file reading thread and storing the data from the target binary log file in the buffer corresponding to the target binary log file includes: Create a target state machine for the target binary log file, wherein the initial state of the target state machine is the writing state; If there are any newly added binary log files after the target binary log file, then the state of the target state machine is set to the write-complete state; Read the data of the target binary log file sequentially. If the currently read data is a preset termination identifier, then set the state of the target state machine to the read complete state. If the target state machine is in the read complete state, then the target binary log file is deleted from the directory storage location.

5. The method according to claim 4, characterized in that, When reading data from the target binary log file sequentially, the method further includes: If the target binary log file reading fails, the state of the target state machine is set to an abnormal state, and the target binary log file is stored in a preset data storage location.

6. The method according to claim 1, characterized in that, The step of parsing the binary log file data in the buffer to obtain the data to be written includes: The binary log file data is parsed to obtain parsed data; Write the parsed data into the file before writing; When the file before writing meets the preset conditions, the parsed data in the file before writing is used as the data to be written.

7. The method according to claim 1, characterized in that, Determining the target write thread corresponding to each of the data shards includes: The fragment number of each fragment data is determined based on the service sequence number of each fragment data. The write thread corresponding to each of the aforementioned shard numbers is used as the target write thread corresponding to the shard data.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Receive a data query request, parse the data query request, and obtain data query parameters, which include: the date and business sequence number of the data to be queried; The data to be queried is obtained from the embedded database based on the data query parameters.

9. The method according to claim 8, characterized in that, The step of retrieving the data to be queried from the embedded database based on the data query parameters includes: Based on the date of the data to be queried, at least one candidate shard data is determined in the embedded database; Based on the service sequence number, determine the target shard data from the candidate shard data; The data to be queried corresponding to the data query parameters is obtained from the target fragment data.

10. A data management device, characterized in that, include: The generation module is used to generate data to be written according to the data calculation request when a data calculation request is received, wherein the data to be written is business data; At least one binary log file is generated based on the data to be written, and the at least one binary log file is stored in a preset directory storage location. The binary log file stores the calculation details data output by the risk control calculation engine. The reading module is used to sequentially read data from each of the binary log files from the directory storage location, and store the read binary log file data in the buffer corresponding to the binary log file; The writing module is used to write the data of the binary log file in the buffer to the embedded database when the data volume in the buffer reaches a preset data volume threshold. The step of writing the binary log file data in the buffer to the embedded database includes: The binary log file data in the buffer is parsed to obtain the data to be written. The data to be written is fragmented to obtain at least one fragment of data, wherein data with the same service sequence number are in the same fragment of data. Determine the target write thread corresponding to each of the data fragments; Each of the target write threads writes the fragmented data into the embedded database.

11. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of claims 1-9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-9.

13. A program product, characterized in that, When the program product is executed by the processor, it implements the method as described in any one of claims 1-9.

Images