Data processing method, apparatus, device, medium, and program product

Abstract

This application provides a data processing method, apparatus, device, storage medium, and program product, which can be applied to the fields of distributed technology and financial technology. The data processing method includes: receiving a data processing instruction for a target product from a scheduling module; sending a data processing request to a data synchronization component, causing the data synchronization component to extract data to be updated associated with the target product from multiple database shards of a persistent database, and aggregating and storing the data to be updated in a memory database corresponding to the target transaction node; and processing the data to be updated in the memory database to obtain updated target data.

Description

Technical Field

[0001] This application relates to the fields of distributed technology and financial technology, and specifically to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the deep integration of internet technology and financial services, financial institutions are experiencing an explosive growth in the amount of data they need to process. To cope with the pressure of storing and accessing massive amounts of data, distributed databases are often used, and the data is horizontally sharded to improve the system's storage capacity and high-concurrency access capabilities. However, due to the decentralized nature of distributed databases, even if the amount of data to be processed is small, it can lead to frequent accesses across different shards of the persistent database, resulting in duplicated network transmission overhead, wasted database connection resources, and reduced data processing efficiency. Summary of the Invention

[0003] In view of the above problems, this application provides data processing methods, apparatus, devices, media and program products to improve data processing efficiency.

[0004] According to a first aspect of this application, a data processing method is provided, comprising: receiving a data processing instruction for a target product from a scheduling module; sending a data processing request to a data synchronization component, such that the data synchronization component extracts data to be updated associated with the target product from multiple database shards of a persistent database, and aggregates and stores the data to be updated in a memory database corresponding to a target transaction node; and performing data processing on the data to be updated in the memory database to obtain updated target data.

[0005] According to an embodiment of this application, the data processing instruction is generated by a scheduling module. The process of the scheduling module generating the data processing instruction includes: obtaining target task definition information associated with the target product from a pre-configured task definition table, wherein the target task definition information is used to characterize the execution timing of the data processing task, and the execution timing includes timed execution rules or event triggering conditions; obtaining target task configuration information associated with the target product from a task detail table, wherein the target task configuration information is used to characterize the data content fields to be processed and the identification information of the target transaction node, wherein the data content fields to be processed include business data items related to the target product, and the identification information includes node number or node address; and generating a data processing instruction for the target product based on the execution timing in the target task definition information, the data content fields to be processed in the target task configuration information, and the identification information of the target transaction node.

[0006] According to an embodiment of this application, before obtaining the target task configuration information associated with the target product from the task details table, the method further includes: querying a distributed registry center to obtain multiple transaction nodes that are currently available; determining the target transaction node to perform the data processing task based on a predefined scheduling strategy and the multiple transaction nodes that are currently available; and associating the identification information of the target transaction node with the configuration information associated with the target product in the task details table.

[0007] According to an embodiment of this application, the method further includes: distributing the target data to their respective corresponding database shards to complete the update processing of the data in each database shard.

[0008] According to an embodiment of this application, distributing target data to their respective corresponding database shards includes: calling a file splitting component to split the target data into multiple target files; determining the correspondence between the multiple target files and the multiple database shards based on a hash algorithm; and writing the target data from the multiple target files into their respective multiple database shards according to the correspondence.

[0009] According to an embodiment of this application, extracting data to be updated associated with a target product from multiple database shards of a persistent database includes: extracting data to be updated from the corresponding database shards based on the data content fields contained in the data processing request; aggregating the extracted data to be updated and storing it in an in-memory database corresponding to the target transaction node.

[0010] According to an embodiment of this application, the memory database is an embedded database.

[0011] A second aspect of this application provides a data processing apparatus, comprising: a receiving module for receiving a data processing instruction for a target product from a scheduling module; a sending module for sending a data processing request to a data synchronization component, such that the data synchronization component extracts data to be updated associated with the target product from multiple database shards of a persistent database and aggregates and stores the data to be updated in a memory database corresponding to a target transaction node; and a processing module for processing the data to be updated in the memory database to obtain updated target data.

[0012] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] According to embodiments of this application, by configuring corresponding in-memory databases for each trading node and using a data synchronization component to retrieve relevant data, each trading node can have its own data processing space. The data synchronization component retrieves target product-related data from each shard of the distributed database in batches and stores it centrally in the corresponding in-memory database. On the one hand, this avoids frequent and multiple accesses to shards, reducing resource consumption (such as IO and connection count) of the sharded database and minimizing performance loss and resource waste from multi-shard access in a distributed environment. On the other hand, storing data centrally in an in-memory database leverages the high-speed read / write characteristics of in-memory storage to improve the efficiency of data aggregation, calculation, and updating operations. This is particularly suitable for high-frequency business scenarios such as batch updates of customer holdings for financial products and centralized calculation of dividends, shortening the time required for large-scale data processing and, to some extent, solving the efficiency bottleneck of distributed data processing. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0018] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown schematically.

[0019] Figure 3 This illustration schematically shows a diagram illustrating the determination of a target transaction node according to an embodiment of this application;

[0020] Figure 4 This illustration schematically shows a data write-back process according to an embodiment of the present application;

[0021] Figure 5 A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown; and

[0022] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] It should be noted that the data processing method and apparatus of this application embodiment can be used in the fields of distributed technology and financial technology, and can also be used in any field other than the fields of distributed technology and financial technology. The application fields of the data processing method and apparatus of this application embodiment are not limited.

[0028] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided in this application all provide users with corresponding operation entry points for users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process enters the expert decision-making process.

[0030] Embodiments of this application provide a data processing method, which involves receiving a data processing instruction for a target product from a scheduling module; sending a data processing request to a data synchronization component, so that the data synchronization component extracts data to be updated associated with the target product from multiple database shards of a persistent database, and aggregates and stores the data to be updated in a memory database corresponding to the target transaction node; and performing data processing on the data to be updated in the memory database to obtain updated target data.

[0031] Figure 1 The diagram illustrates an application scenario of the data processing method according to an embodiment of this application.

[0032] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a scheduling module 110, a transaction node 120, a data synchronization component 130, and a memory database 140.

[0033] The scheduling module 110, transaction node 120, and data synchronization component 130 can transmit data via wired or wireless means. The scheduling module 110 interacts with the transaction node 120, issuing scheduling instructions to the transaction node 120 and allocating transaction tasks to be processed.

[0034] Transaction node 120 is the execution unit that actually processes transaction business. After receiving instructions from the scheduling module, it completes the specific transaction logic processing. It is connected to the scheduling module 110 to receive scheduling instructions; it establishes a bidirectional connection with the data synchronization component 130. When processing transactions, it uses the data synchronization component 130 to call data in each database shard for processing, and after processing the data, it writes the updated data back to the database shard.

[0035] The in-memory database 140 is used to temporarily store the data to be updated extracted by the data synchronization component 130, in order to support the rapid data processing and interaction of transaction nodes.

[0036] Shards 1 to N are data storage units. The overall data is split according to preset rules and stored on the data shards respectively, which is used to improve the efficiency of data storage and access, and also facilitates the horizontal expansion of the system.

[0037] It should be understood that Figure 1 The number of scheduling modules, transaction nodes, and data synchronization components shown is merely illustrative. Depending on implementation needs, it can include any number of terminal devices, large models, and rule engines.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-4 The data processing method according to the embodiments of this application will be described in detail.

[0039] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown.

[0040] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S230, and the transaction processing method can be executed by the target transaction node.

[0041] During operation S210, data processing instructions for the target product are received from the scheduling module.

[0042] According to embodiments of this application, the target product represents a specific financial product for which data processing is required, such as a wealth management product, fund, or deposit product. The data processing instruction is a specific task instruction issued by the scheduling module, containing information such as the target product, the data type to be processed, the range of data to be processed, and the processing time. For example, wealth management product A needs to update the holdings of all customers and calculate the daily dividend at 2:00 AM every day. At this time, the scheduling module will automatically generate a data processing instruction at 2:00 AM: "Execute holdings update and dividend calculation for product A," and send this instruction to the data processing node responsible for that product, i.e., the target transaction node. The target transaction node accepts the data processing instruction.

[0043] In operation S220, a data processing request is sent to the data synchronization component, so that the data synchronization component extracts the data to be updated associated with the target product from multiple database shards of the persistent database, and aggregates and stores the data to be updated in the in-memory database corresponding to the target transaction node.

[0044] According to embodiments of this application, a data processing request is a specific data operation request related to the target product generated by the target transaction node after receiving a scheduling instruction. The data synchronization component is an intermediate component responsible for data extraction, aggregation, and synchronization, ensuring accurate data flow between different storage locations. A persistent database is used for persistent data storage. The characteristic of a persistent database is that data will not be lost due to system restarts; it is commonly used to store core business data, such as customer holding records and transaction logs. Database sharding is the process of dividing a large database into smaller data blocks according to rules to improve performance, such as sharding by customer number range: customers 1-1000 exist in shard 1, and customers 1001-2000 exist in shard 2. The data to be updated is the original data that needs to be processed, such as the customer's current shareholding and historical dividend records. An in-memory database can be an embedded database, such as an H2 database, characterized by fast access speed, suitable for temporary storage of data requiring high-frequency processing.

[0045] For example, after receiving the instruction, the target trading node sends a request to the data synchronization component: "Please extract all customer holdings of Product A." The data synchronization component will filter out all customer holding records related to Product A from multiple shards of the persistent database, i.e., the data to be updated. Then, it will aggregate these scattered data and store them uniformly in the in-memory database corresponding to the trading node, so that the target trading node can access the in-memory database to perform calculations and other processing on the data to be updated.

[0046] In operation S230, data processing is performed on the data to be updated in the in-memory database to obtain the updated target data. According to the embodiments of this application, data processing refers to the specific business logic of performing data processing tasks on the data to be updated, such as calculation, verification, and updating. The updated target data is the final result data obtained after processing, such as the updated customer holdings or the calculated dividend amount.

[0047] like Figure 1 As shown, transaction node 1 reads the current holdings (data to be updated) of all customers of product A from the in-memory database 1, updates their latest holdings, such as customer A originally holding 1000 units, and subscribing to 500 units on the same day, updating to 1500 units; based on the product's net asset value on the day and the updated holdings, the daily dividend for each customer is calculated. The final "latest holdings of customers" and "daily dividend amount" are the updated target data.

[0048] According to embodiments of this application, by configuring corresponding in-memory databases for each trading node and using a data synchronization component to retrieve relevant data, each trading node can have its own data processing space. The data synchronization component retrieves target product-related data from each shard of the distributed database in batches and stores it centrally in the corresponding in-memory database. On the one hand, this avoids frequent and multiple accesses to shards, reducing resource consumption (such as IO and connection count) of the sharded database and minimizing performance loss and resource waste from multi-shard access in a distributed environment. On the other hand, storing data centrally in an in-memory database leverages the high-speed read / write characteristics of in-memory storage to improve the efficiency of data aggregation, calculation, and updating operations. This is particularly suitable for high-frequency business scenarios such as batch updates of customer holdings for financial products and centralized calculation of dividends, shortening the time required for large-scale data processing and, to some extent, solving the efficiency bottleneck of distributed data processing.

[0049] According to embodiments of this application, the memory database is an embedded database, such as the H2 database.

[0050] According to an embodiment of this application, extracting data to be updated associated with a target product from multiple database shards of a persistent database includes: extracting data to be updated from the corresponding database shards based on the data content fields contained in the data processing request; aggregating the extracted data to be updated and storing it in an in-memory database corresponding to the target transaction node.

[0051] After receiving a data processing request from the target trading node, the data synchronization component first parses the fields of the data to be processed contained in the request, such as business data items related to the target product, such as customer identification information (ID), holding shares, and holding duration. It then locates the corresponding database shard storing these data fields according to the preset database sharding rules, such as splitting into multiple database shards based on customer number ranges or regions. From each relevant shard, it extracts the original data to be updated that is only associated with the target product, avoiding redundant extraction of irrelevant data. The data synchronization component then aggregates and integrates the scattered data extracted from multiple shards to ensure data integrity and consistency. Aggregation and integration may include merging related holding records for the same customer and removing duplicate data. Finally, the aggregated data to be updated is uniformly stored in the in-memory database corresponding to the target trading node, providing high-speed data support for the target trading node to quickly access data and efficiently perform computational processing, while avoiding resource consumption and performance degradation caused by the trading node frequently accessing multiple database shards directly.

[0052] When a target transaction node needs to process data, it first pulls the target data, which is scattered across different shards, from the sharded cluster of the persistent database in batches through a data synchronization device. This scattered data is then aggregated and centrally stored in a secondary cache layer, i.e., based on the H2 database's Multi-Version (MV) mode. Leveraging the file system's input / output (IO) processing capabilities and H2's high-performance read / write features, batch computations are completed in a memory-level environment. After processing, the results are written back to the corresponding persistent database shards. By utilizing the high-performance read / write advantages of the in-memory database and combining it with the file system's IO processing capabilities, efficient batch data computation is achieved, reducing data processing time. For example, batch tasks such as updating customer holdings for financial products and calculating dividends can be completed in a short time. By aggregating scattered data from different shards into the file system and storing it in the in-memory database for centralized processing, the complex multi-shard data access logic and low cross-shard computation efficiency in distributed sharding scenarios are solved. This allows the business layer to focus solely on data processing logic without needing to concern itself with the underlying shard distribution details.

[0053] According to an embodiment of this application, the data processing instruction is generated by a scheduling module. The process of the scheduling module generating the data processing instruction includes: obtaining target task definition information associated with the target product from a pre-configured task definition table, wherein the target task definition information is used to characterize the execution timing of the data processing task, and the execution timing includes timed execution rules or event triggering conditions; obtaining target task configuration information associated with the target product from a task detail table, wherein the target task configuration information is used to characterize the data content fields to be processed and the identification information of the target transaction node, wherein the data content fields to be processed include business data items related to the target product, and the identification information includes node number or node address; and generating a data processing instruction for the target product based on the execution timing in the target task definition information, the data content fields to be processed in the target task configuration information, and the identification information of the target transaction node.

[0054] The task definition table is used to indicate the task types and dependencies between multiple data processing tasks. Different tasks can be derived from the task definition table, and their types and dependencies can be defined. For example, in a financial batch processing scenario, the task definition table can be configured so that the "Wealth Management Product Dividend Calculation Task" is of type aggregate calculation (reduce), and specify that it can only be executed after the "Customer Holdings Data Synchronization Task" is completed.

[0055] The task details table indicates the specific execution procedures and steps included in each data processing task. It defines the specific execution steps and content of each task, such as which procedures each task includes, for example, data synchronization, transaction processing, and data persistence. The task details table breaks down a single task into multiple execution procedures, such as data synchronization, transaction processing, and data persistence, clearly defining the "execution flow details" of the task. For example, in a wealth management product dividend scenario, the task details table can record the three steps configured for the dividend calculation task: first, the data synchronization program retrieves customer holdings data; second, the transaction processing program calculates the dividend; and finally, the data persistence program writes the results to the database.

[0056] First, the scheduling module uses the target product as an index to filter out the target task definition information associated with it from the pre-configured task definition table of the system, clarifying the execution time of the data processing task and ensuring that the task is triggered at the correct time, such as 2:00 AM every day, the last working day of the month, etc. It can also be triggered by events based on specific conditions, such as when the product's net value fluctuation reaches a threshold, or after the customer completes the subscription operation, thus providing time-dimensional constraints for task execution.

[0057] Secondly, the scheduling module continues to use the target product as a benchmark, extracting the corresponding target task configuration information from the task details table, namely the specific content and execution entity of the task. On the one hand, it clarifies the data content fields to be processed, namely the specific business data items that need to participate in the calculation, such as the customer's product holding share, holding period, historical transaction records, and the product's net asset value on that day, all of which are directly related to the target product. On the other hand, it determines the identification information of the target trading node responsible for executing the task, locating the execution unit through the node number (e.g., node 005) or node address (e.g., the IP address of a specific server), ensuring that the task is assigned to the correct processing node.

[0058] Finally, the scheduling module generates a data processing instruction for the target product based on information such as the execution timing in the target task definition, the data content field to be processed in the target task configuration, and the target transaction node identifier. This instruction contains all the key parameters required for task execution, directly guiding subsequent data flow and business processing, ensuring the accurate and orderly progress of the data processing task.

[0059] By associating the target product with pre-configured task definition tables and task detail tables, the system accurately obtains the timing rules or event triggering conditions for data processing, the fields of business data to be processed, and the identifiers of target transaction nodes, ensuring the relevance and completeness of instruction generation. Based on these key elements, the system integrates and generates exclusive data processing instructions for the target product, clearly defining the dimensions of "when to execute, what data to process, and which node to execute" for data processing tasks, avoiding execution deviations caused by ambiguous instructions. At the same time, relying on the association logic between the pre-configured tables and the target product, the system simplifies the instruction generation process, reduces manual intervention, improves the efficiency and accuracy of instruction generation, effectively supports multi-product parallel processing scenarios, and enhances the overall task scheduling standardization and operational stability of the system.

[0060] Figure 3 A schematic diagram illustrating the determination of a target transaction node according to an embodiment of this application is shown.

[0061] According to an embodiment of this application, before obtaining the target task configuration information associated with the target product from the task details table, the data processing method further includes the following steps executed by the scheduling module: querying the distributed registry center to obtain multiple transaction nodes that are currently available; determining the target transaction node to perform the data processing task based on a predefined scheduling strategy and the multiple transaction nodes that are currently available; and associating the identification information of the target transaction node with the configuration information associated with the target product in the task details table.

[0062] like Figure 3 As shown, each transaction node registers or submits its availability status to the distributed registry center through proactive reporting, such as whether the transaction node is online and its resource load. The scheduling module obtains a real-time status list of all transaction nodes from the distributed registry center through a preset query interface or protocol, thereby filtering out the transaction nodes that are currently available, and then determining the target transaction node based on the scheduling strategy. Specifically, it queries the distributed registry center to obtain a list of multiple transaction nodes that are currently available, ensuring that the nodes subsequently allocated have the ability to execute tasks. According to predefined scheduling strategies, such as allocation based on node load balancing, allocation based on regional proximity, and allocation based on node processing capacity priority, the most suitable node for executing the current target product data processing task is selected from the above available nodes, i.e., the target transaction node is determined. It is required to ensure the rationality of task allocation and avoid node resource waste or overload. The identification information of the selected target transaction node (such as node number, IP address, etc.) is associated and stored with the configuration information corresponding to the target product in the task details table. When retrieving target task configuration information from the task details table later, it can directly include the target transaction node identifier unique to that task, ensuring that the data processing task can be accurately directed to the identified available node.

[0063] The scheduling module queries the distributed registry center to obtain a list of available transaction nodes. Combined with predefined scheduling strategies (such as load balancing and priority allocation), it accurately selects target transaction nodes and associates their identification information with the configuration information of the target product in the task details table. This ensures that the nodes assigned to tasks have execution capabilities and comply with resource optimization rules. This avoids execution failures caused by assigning tasks to unavailable nodes and ensures the rational utilization of node resources through scheduling strategies, preventing single-node overload or resource waste. Furthermore, by pre-associating node identifiers with task configurations, it provides accurate execution entity information for subsequent data processing instruction generation, reducing node verification overhead during instruction generation and improving the reliability, resource utilization, and overall process efficiency of task scheduling in a distributed environment.

[0064] According to an embodiment of this application, the data processing method further includes: distributing the target data to their respective corresponding database shards to complete the update processing of the data in each database shard.

[0065] The data synchronization component receives the target data transmitted by the target transaction node and identifies the original storage shard corresponding to each piece of target data according to the sharding rules of the persistent database. For example, the updated data of customer A corresponds to shard 1, and the updated data of customer B corresponds to shard 2, ensuring that the data can be accurately written back to the initial storage location. The data synchronization component distributes the split target data to their respective database shards and initiates a data update request to each shard, triggering the replacement or supplementation of the original old data in the shard. For example, it overwrites the original historical share data in the shard with the customer's latest share, or adds the current day's dividend record to the corresponding customer's business data. The entire distribution process strictly follows the data consistency protocol to ensure that the data received by each shard is complete and error-free, avoiding data loss, corruption, or incomplete updates.

[0066] Temporary target data obtained from in-memory computation is ultimately persisted to the corresponding shard in the persistent database, completing the closed loop from data processing to result persistence. This ensures the long-term secure storage of core business data, preventing loss due to system restarts or memory release. Furthermore, the precise shard-based distribution avoids the performance pressure caused by writing all data, ensuring the efficiency and accuracy of data updates in each shard and providing reliable data source support for subsequent business queries.

[0067] Figure 4 The schematic diagram illustrates a data write-back process according to an embodiment of this application.

[0068] According to an embodiment of this application, distributing target data to their respective corresponding database shards includes: calling a file splitting component to split the target data into multiple target files; determining the correspondence between the multiple target files and the multiple database shards based on a hash algorithm; and writing the target data from the multiple target files into their respective multiple database shards according to the correspondence.

[0069] like Figure 4 As shown, the file splitting component is invoked to split the complete target data output by the target transaction node. Considering the data volume, sharding storage rules, and subsequent write efficiency requirements, the massive target data is divided into multiple well-structured and balanced target files. For example, it can be split by customer ID, with each 1000 customer data entries forming one target file. This avoids transmission delays or write failures caused by excessively large single file sizes, while also improving the feasibility of parallel writing across multiple shards.

[0070] The core identifier information of each target file is hashed using a pre-defined hash algorithm (such as MD5, SHA-1, etc.) to obtain the corresponding hash value. For example, the core identifier information of a target file might include a customer ID prefix or a unique file ID. Based on a pre-defined mapping rule between the hash value and database shards, the corresponding database shard for each target file is determined, thus establishing a correspondence between target files and database shards. For example, the result of modulo the hash value is matched with the shard number: a hash value modulo 1 corresponds to shard 1, a hash value modulo 2 corresponds to shard 2, and so on. This hash algorithm-based mapping method ensures even data distribution, preventing load imbalance caused by a single shard carrying too much data, and guarantees that data with the same core identifier is always mapped to the same shard, maintaining data storage consistency.

[0071] Based on the established correspondence, a multi-threaded or distributed write mechanism is initiated to synchronously write the target data from each target file to its corresponding database shard. During the write process, data update operations are performed on each shard to ensure the accuracy of the written data. For example, the latest holdings data in the target file overwrites the historical data in the shard, and new dividend records are added to the corresponding customer's data entries. The entire process improves transmission and write efficiency through file splitting and ensures the rationality and consistency of data allocation through hash algorithms, achieving efficient and accurate distribution of target data to multiple database shards and completing a full update of the persistent database.

[0072] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0073] Figure 5 A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0074] like Figure 5 As shown, the data processing apparatus 500 of this embodiment includes a receiving module 510, a sending module 520, and a processing module 530.

[0075] The receiving module 510 is used to receive data processing instructions for the target product from the scheduling module. In one embodiment, the receiving module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0076] The sending module 520 is used to send a data processing request to the data synchronization component, so that the data synchronization component extracts the data to be updated associated with the target product from multiple database shards of the persistent database, and aggregates and stores the data to be updated in the memory database corresponding to the target transaction node. In one embodiment, the sending module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0077] The processing module 530 is used to process the data to be updated in the memory database to obtain the updated target data. In one embodiment, the processing module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0078] According to an embodiment of this application, the data processing instruction is generated by a scheduling module, and the apparatus for the scheduling module to generate the data processing instruction includes a first acquisition module, a second acquisition module, and a generation module.

[0079] The first acquisition module is used to obtain target task definition information associated with the target product from a pre-configured task definition table, based on the target product. The target task definition information is used to characterize the execution timing of the data processing task, including timed execution rules or event triggering conditions. The second acquisition module is used to obtain target task configuration information associated with the target product from a task detail table, which is used to characterize the data content fields to be processed and the identification information of the target transaction node. The data content fields to be processed include business data items related to the target product, and the identification information includes node number or node address. The generation module is used to generate data processing instructions for the target product based on the execution timing in the target task definition information, the data content fields to be processed in the target task configuration information, and the identification information of the target transaction node.

[0080] According to embodiments of this application, before obtaining target task configuration information associated with the target product from the task details table, the device further includes a query module, a determination module, and an association module.

[0081] The query module queries the distributed registry center to obtain multiple transaction nodes that are currently available; the determination module determines the target transaction node for executing the data processing task based on a predefined scheduling strategy and the multiple transaction nodes that are currently available; and the association module associates the identification information of the target transaction node with the configuration information associated with the target product in the task details table.

[0082] According to an embodiment of this application, the data processing apparatus further includes: a distribution module, used to distribute target data to their respective corresponding database shards to complete the update processing of data in each database shard.

[0083] According to embodiments of this application, the distribution module includes a calling submodule, a determining submodule, and a writing submodule.

[0084] The `call` submodule is used to call the file splitting component to split the target data into multiple target files; the `determine` submodule is used to determine the correspondence between multiple target files and multiple database shards based on a hash algorithm; and the `write` submodule is used to write the target data from the multiple target files to their respective database shards according to the correspondence.

[0085] According to an embodiment of this application, the sending module 520 includes an extraction submodule and a storage submodule.

[0086] The extraction submodule is used to extract the data to be updated from the corresponding database shards based on the data content fields contained in the data processing request; the storage submodule is used to aggregate the extracted data to be updated and store it in the in-memory database corresponding to the target transaction node.

[0087] According to embodiments of this application, the memory database is an embedded database. According to embodiments of this application, any multiple modules among the receiving module 510, transmitting module 520, and processing module 530 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some functions of one or more of these modules can be combined with at least some functions of other modules and implemented in one module. According to embodiments of this application, at least one of the receiving module 510, transmitting module 520, and processing module 530 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the receiving module 510, transmitting module 520, and processing module 530 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0088] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application.

[0089] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0090] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0091] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 606 including a network interface card such as a LAN card, modem, etc. The communication section 606 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0092] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0093] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0094] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this application.

[0095] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0096] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 606, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0097] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 606, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0098] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0101] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A data processing method, characterized by, The method includes: Receive data processing instructions for the target product from the scheduling module; The data processing request is sent to the data synchronization component, so that the data synchronization component extracts the data to be updated associated with the target product from multiple database shards of the persistent database, and aggregates and stores the data to be updated in the memory database corresponding to the target transaction node; The data to be updated in the memory database is processed to obtain the updated target data.

2. The method of claim 1, wherein, The data processing instruction is generated by the scheduling module, and the process by which the scheduling module generates the data processing instruction includes: Based on the target product, obtain the target task definition information associated with the target product from the pre-configured task definition table. The target task definition information is used to characterize the execution timing of the data processing task. The execution timing includes timed execution rules or event triggering conditions. Obtain the target task configuration information associated with the target product from the task details table. The target task configuration information is used to characterize the data content fields to be processed and the identification information of the target transaction node. The data content fields to be processed include business data items related to the target product, and the identification information includes node number or node address. Based on the execution timing in the target task definition information, the pending data content field in the target task configuration information, and the identification information of the target transaction node, a data processing instruction for the target product is generated.

3. The method of claim 2, wherein, Before retrieving the target task configuration information associated with the target product from the task details table, the method further includes: Query the distributed registry center to obtain multiple transaction nodes that are currently available; Based on a predefined scheduling strategy and the multiple transaction nodes that are currently available, the target transaction node for executing the data processing task is determined. The identification information of the target transaction node is associated with the configuration information related to the target product in the task details table.

4. The method of claim 1, wherein, The method further includes: The target data is distributed to its respective database shards to complete the data update process in each database shard.

5. The method according to claim 4, characterized in that, Distributing the target data to its respective database shards includes: The target data is split into multiple target files by calling the file splitting component; The correspondence between the multiple target files and the multiple database shards is determined based on a hash algorithm; According to the correspondence, the target data in the multiple target files is written to their respective database shards.

6. The method according to claim 1, characterized in that, Extracting the data to be updated associated with the target product from multiple database shards of the persistent database, including: Based on the data content fields contained in the data processing request, extract the data to be updated from the corresponding database shard; The extracted data to be updated is aggregated and stored in the in-memory database corresponding to the target transaction node.

7. The method according to claim 1, characterized in that, The memory database is an embedded database.

8. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive data processing instructions for the target product from the scheduling module; A sending module is configured to send the data processing request to a data synchronization component, so that the data synchronization component extracts the data to be updated associated with the target product from multiple database shards of the persistent database, and aggregates and stores the data to be updated in a memory database corresponding to the target transaction node; and The processing module is used to process the data to be updated in the memory database to obtain the updated target data.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images