Data processing method and device, computer device, and storage medium

Abstract

The application discloses a data processing method, which is applied to the technical field of data processing and the field of digital medical treatment. The method provided by the application comprises the following steps: receiving a target data processing request containing a target data table in a target relational database; performing time slicing on data in the target data table to obtain a time-sliced mapping data table; splitting the target data processing request into data processing subtasks; sending the data processing subtasks to computing subnodes; accessing the time-sliced mapping data table, obtaining corresponding time-sliced mapping data segments from the time-sliced mapping data table; obtaining to-be-processed data from the target relational database according to the time-sliced mapping data segments, and completing the data processing subtasks by using the to-be-processed data to obtain data processing subresults; and performing data aggregation processing on the data processing subresults to obtain a target data processing result.

Description

Technical Field

[0001] This application relates to the field of digital medical data processing technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Existing digital healthcare platforms store massive amounts of data; for example, many platforms have user data starting in the millions. Furthermore, the continuous operation of these platforms generates a vast amount of related digital healthcare data, including personal health records, prescriptions, and examination reports. However, given the evolving needs of digital healthcare services, not all of this massive amount of related data can be stored in big data systems. For instance, using big data systems increases the complexity of the technical architecture, thus increasing the instability of the digital healthcare platform. This could lead to catastrophic technical problems such as data loss due to hardware infrastructure failures in big data storage systems. On the other hand, it also significantly increases technical costs, including human resources and hardware costs.

[0003] Therefore, based on the needs of some digital healthcare business, a large amount of related digital healthcare data is still stored in relational databases. However, as the volume of such related digital healthcare data grows, relational databases also have their own performance bottlenecks, leading to technical problems such as lag, timeouts, and even memory overflows when used for some digital healthcare data processing tasks. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, and storage medium to solve the technical problems of lag, timeout, and memory overflow in existing relational databases when processing large amounts of digital medical data.

[0005] A first aspect of this application provides a data processing method, comprising:

[0006] Receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements;

[0007] The data in the target data table is sharded according to time to obtain a time sharding mapping data table, and the time sharding mapping data table is stored in the target memory database;

[0008] The task scheduling center breaks down the target data processing request into at least one data processing subtask according to the data processing business requirements, wherein the data processing subtask contains a subset of the target data table.

[0009] The data processing subtask is sent to at least one computing subnode, wherein the computing subnode contains at least one data processing subscript;

[0010] The data processing sub-script is used to access the time-slice mapping data table in the target memory database, and the corresponding time-slice mapping data segment is obtained from the time-slice mapping data table according to the subset dataset.

[0011] Data to be processed is obtained from the target relational database according to the time-sliced ​​mapping data segment, and the data to be processed is used to complete the data processing sub-task through the data processing sub-script to obtain the data processing sub-result;

[0012] The task scheduling center aggregates and processes the data processing sub-results to obtain the target data processing result.

[0013] A second aspect of this application provides a data processing apparatus, comprising:

[0014] The request receiving module is used to receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements;

[0015] The time slicing module is used to slice the data in the target data table according to time to obtain a time slicing mapping data table, and store the time slicing mapping data table in the target memory database;

[0016] The task splitting module is used to split the target data processing request into at least one data processing subtask according to the data processing business requirements through the task scheduling center, wherein the data processing subtask contains a subset of the target data table.

[0017] A subtask allocation module is used to send the data processing subtask to at least one computing subnode, wherein the computing subnode contains at least one data processing subscript;

[0018] The data segment acquisition module is used to access the time slice mapping data table in the target memory database using the data processing sub-script, and obtain the corresponding time slice mapping data segment from the time slice mapping data table according to the subset dataset;

[0019] The subtask execution module is used to obtain data to be processed from the target relational database according to the time-sliced ​​mapping data segment, and use the data to be processed to complete the data processing subtask through the data processing sub-script to obtain the data processing sub-result;

[0020] The processing result aggregation module is used to aggregate the data sub-results through the task scheduling center to obtain the target data processing result.

[0021] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data processing method described above.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described data processing method.

[0023] The aforementioned data processing method, apparatus, computer equipment, and storage medium, through receiving a target data processing request containing a target data table in a target relational database; segmenting the data in the target data table according to time to obtain a time-segmented mapping data table; breaking down the target data processing request into data processing sub-tasks; sending the data processing sub-tasks to computing sub-nodes; accessing the time-segmented mapping data table to obtain the corresponding time-segmented mapping data segment; retrieving the data to be processed from the target relational database based on the time-segmented mapping data segment, and using the data to be processed to complete the data processing sub-tasks to obtain a data processing sub-result; and performing data aggregation processing on the data processing sub-results to obtain the target data processing result. This not only improves the execution efficiency of digital medical data processing tasks using relational databases, avoiding technical problems such as lag, timeouts, and memory overflows, but also fully utilizes the various computing sub-nodes in the digital medical platform system to improve system resource utilization. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application environment for a data processing method according to an embodiment of this application;

[0026] Figure 2 This is a flowchart of a data processing method in one embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the structure of a data processing device in one embodiment of this application;

[0028] Figure 4 This is a schematic diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The data processing method provided in this application can be applied to, for example... Figure 1 In the application environment, Figure 1 The hardware devices that carry in-memory databases, relational databases, computing sub-nodes, task scheduling centers, and configuration centers can be, but are not limited to, various personal computers and laptops. They can also be servers. Servers can be standalone servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This is understandable. Figure 1 The number of hardware devices that support the in-memory database, relational database, computing sub-nodes, task scheduling center, and configuration center is merely illustrative and can be expanded in any number according to actual needs.

[0031] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which is applied to... Figure 1 Taking the in-memory database, relational database, compute sub-nodes, task scheduling center, and configuration center as examples, the explanation includes the following steps S101 to S107:

[0032] S101. Receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements.

[0033] In a more specific embodiment, the relational database is implemented using MySQL. A relational database is a database that uses a relational model to organize data, storing data in rows and columns. The rows and columns of a relational database are called tables, and a group of tables constitutes a database. The relational model can be understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. Common relational databases include, but are not limited to, MySQL, Oracle, DB2, Microsoft SQL Server, and Microsoft Access.

[0034] Furthermore, in more complex technical implementation environments, the data sources for the data processing business requirements may be multiple, such as non-relational databases (e.g., MongoDB, Neo4j, Cassandra, HBase), big data storage (Hadoop, Elasticsearch, Spark, etc.), database clusters, data warehouses, data files, and data interfaces. The technical differences in data acquisition methods from different data sources and their respective usage processes are not the core of this embodiment and will not be elaborated upon here. Furthermore, the target data table included in the target data processing request is not unique; the data to be processed may also come from different data tables in the target relational database.

[0035] For example, in the patient management subsystem of a digital healthcare platform, a digital healthcare data processing request is received. The digital healthcare data processing business requirement contained in the request is to statistically analyze the purchase records of a certain type of drug for each patient on the digital healthcare platform last year. The target relational database contained in the request is the database of the patient management subsystem, and the target data tables contained in the request are the patient table and the drug record table in the patient management subsystem.

[0036] S102. The data in the target data table is sliced ​​according to time to obtain a time-sliced ​​mapping data table, and the time-sliced ​​mapping data table is stored in the target memory database.

[0037] Furthermore, in this embodiment, the data in the target data table is sharded according to the hour as the standard time unit. The labeled time unit can be flexibly set according to the data processing business requirements and the data to be processed. For example, in the aforementioned patient management subsystem of the digital healthcare platform, if another digital healthcare data processing request is received, and the digital healthcare data processing business requirement of this request is to statistically analyze the user activity data of each patient and doctor using the digital healthcare APP provided by the digital healthcare platform for each month of the previous year (patients use the APP for online registration, payment, uploading examination or test images, etc., and doctors use the APP for online consultation, issuing electronic prescriptions, etc.), then the data in the target data table can be sharded according to the month as the standard time unit based on this digital healthcare data request, and the resulting time-sharded mapping data table can be saved to the Redis of the car insurance subsystem. The data content in the target data table is obtained from the target relational database. It should be noted that during the stage of establishing the time-slicing mapping data table, it is not necessary to obtain all the data content in the target data table. Instead, only the corresponding data row index and the corresponding data row time field are obtained. The data row time field is the dependent data for time slicing.

[0038] Furthermore, in a more specific embodiment, the target in-memory database is implemented using Redis. An in-memory database is a database that stores all data in memory, unlike traditional databases that store data on external storage. All data access in an in-memory database is also performed in memory. Common in-memory databases include, but are not limited to, Redis, Memcached, SQLite, Oracle TimesTen, and Apache Ignite. Redis, in particular, is an in-memory database that stores key-value pairs. Its data structure storage system can be used as a database, cache, and message broker, and it supports rich data structure types, such as strings, hashes, and sorted sets. Redis also enables data persistence and backup. In this more specific embodiment, Redis can be used to store the time-sharded mapping table on external storage for data persistence and backup.

[0039] S103. The task scheduling center splits the target data processing request into at least one data processing subtask according to the data processing business requirements, wherein the data processing subtask contains a subset of the target data table.

[0040] Furthermore, in a more specific embodiment, the task scheduling center uses XXL-JOB for technical implementation. The task scheduling center can also use other technical frameworks, including but not limited to: Saturn, elastic-job, Quartz, etc. XXL-JOB is a lightweight distributed task scheduling framework that includes features such as elastic scaling, fault handling, blocking handling, and self-maintenance. In a more specific embodiment, the XXL-JOB is used to split the target data processing request into at least one data processing sub-task according to the data processing business requirements. Further, in the process of splitting the target data processing request into at least one data processing sub-task, the criteria used for data splitting can be set according to the target data table included in the target data processing request. For example, each data table's data processing task can be considered as one data processing sub-task, or a data table's data processing task can be considered as at least two data processing sub-tasks, or in a more complex implementation environment, the data processing task of one data source or channel can be considered as one data processing sub-task. If the data processing task corresponding to the data processing request has a small amount of data, it can be split into a single data processing subtask, which contains all the data in the target data table.

[0041] S104. The data processing subtask is sent to at least one computing sub-node, wherein the computing sub-node contains at least one data processing sub-script.

[0042] Furthermore, before sending the data processing subtask to at least one computing sub-node, the method further includes: First, querying the resource usage and task allocation information of all current computing sub-nodes through the configuration center, such as querying the current CPU utilization, current memory utilization, current disk read / write usage, current energy consumption, and current hardware ambient temperature of the computing sub-nodes; and for example, querying the number of tasks assigned to each computing sub-node, the percentage of tasks already completed, and the estimated time required to complete the remaining tasks. Then, obtaining at least one selected computing sub-node based on the resource usage and task allocation information, wherein the selected computing sub-node serves as the data processing task sub-node for the target data processing request. Finally, establishing a first correspondence between the target data processing request and the selected computing sub-node, and storing the first correspondence in the configuration center.

[0043] Furthermore, in a more specific embodiment, the configuration center is implemented using Apollo. The configuration center can also use other technical frameworks, including but not limited to: Nacos, Diamond, Disconf, Spring Cloud Config, etc. In a distributed service architecture, the system is split from a monolithic application into multiple service nodes on a distributed system. Simultaneously, the corresponding configuration files are also migrated and dispersed. At this point, a configuration center is needed to extract the configuration files from each application for unified management, eliminating the need for applications on multiple service nodes to manage their own configuration files. Apollo can centrally manage configurations for different application environments and clusters, and configuration modifications can be pushed to the application in real time, possessing standardized permissions, process governance, and other technical characteristics. In a more specific embodiment, Apollo is used to centrally manage the configuration of the computing sub-nodes.

[0044] Furthermore, in a more specific embodiment, the data processing sub-script is implemented using the Lua scripting language. The data processing sub-script can also use other scripting technologies, including but not limited to: R, Python, Shell, MatLab, Julia, etc. The Lua scripting language can be compiled and run on all operating systems and platforms, and provides a rich set of lightweight libraries, while possessing easily extensible interfaces and mechanisms, supporting procedural and functional programming. In a more specific embodiment, the data processing sub-script is generated using the Lua scripting language to complete the data processing sub-task by utilizing the aforementioned technical characteristics of the Lua scripting language.

[0045] Furthermore, after sending the data processing subtask to at least one computing sub-node, the process further includes: first, monitoring the priorities of other computing tasks received by the computing sub-node, wherein the computing tasks are assigned by the task scheduling center and their priorities are set by the configuration center. Then, if the computing sub-node has a higher-priority computing task than the data processing subtask, the execution of the data processing subtask is paused, the higher-priority computing task is started, and the task breakpoint index of the data processing subtask is recorded in the memory database. Finally, after the higher-priority computing task finishes running, the data processing subtask resumes execution from the task breakpoint index.

[0046] S105. Use the data processing sub-script to access the time slice mapping data table in the target memory database, and obtain the corresponding time slice mapping data segment from the time slice mapping data table according to the subset data set.

[0047] Specifically, the step of using the data processing sub-script to access the time-slice mapping data table in the target memory database and obtaining the corresponding time-slice mapping data segment from the time-slice mapping data table based on the subset dataset includes: First, using the data processing sub-script to create a preset first number of data processing sub-threads, and splitting the data processing subtasks and assigning them to each data processing sub-thread, wherein the subset dataset is split and assigned to each data processing sub-thread. Then, adding the data processing sub-threads to the data processing thread pool of the computing sub-node. Finally, starting the data processing sub-thread to access the corresponding time-slice mapping data table in the memory database based on the assigned subset dataset, obtaining the corresponding time-slice mapping data segment, and locking the time-slice mapping data segment.

[0048] Further, after starting the data processing sub-thread to access the corresponding time-slice mapping data table in the memory database according to the allocated subset dataset, the method further includes: First, monitoring the real-time memory usage percentage of the computation sub-nodes; if the real-time memory usage percentage is greater than a preset first threshold, pausing a preset second number of data processing sub-threads. Then, monitoring a third number of surviving data processing sub-threads in the data processing thread pool; if the third number is less than the core thread count of the data processing thread pool, resuming the paused data processing sub-threads, where the core thread count is the minimum number of threads maintained by the data processing thread pool. Finally, if the real-time memory usage percentage is lower than a preset second threshold, increasing the core thread count of the data processing thread pool by a preset thread count increment percentage.

[0049] S106. Obtain the data to be processed from the target relational database according to the time-sliced ​​mapping data segment, and use the data to be processed to complete the data processing sub-task through the data processing sub-script to obtain the data processing sub-result.

[0050] Furthermore, after successfully retrieving the data to be processed from the target relational database, the corresponding time-slice mapping data segment in the in-memory database is locked. Even further, an exclusive lock is placed on the time-slice data segment, meaning that other data accessors cannot perform data read or write operations on the locked time-slice mapping data segment until the data processing sub-script completes its data processing task and the exclusive lock is released.

[0051] S107. The data processing sub-results are aggregated and processed by the task scheduling center to obtain the target data processing result.

[0052] Furthermore, before the data processing sub-results are aggregated and processed by the task scheduling center, the process includes: First, verifying the completeness of all data processing sub-results through the task scheduling center. If any data processing sub-results are missing, the currently obtained data processing sub-results are merged into a first data processing result. Then, the missing data processing sub-tasks are merged into a new target data processing request, and the aforementioned steps are repeated to process the new target data processing request to obtain a second data processing result. Finally, the first data processing result and the second data processing result are merged into the target data processing result.

[0053] The data processing method provided in this embodiment involves receiving a target data processing request containing a target data table in a target relational database; slicing the data in the target data table according to time to obtain a time-sliced ​​mapping data table; breaking down the target data processing request into data processing sub-tasks; sending the data processing sub-tasks to computing sub-nodes; accessing the time-sliced ​​mapping data table to obtain the corresponding time-sliced ​​mapping data segment; retrieving the data to be processed from the target relational database based on the time-sliced ​​mapping data segment, and using the data to be processed to complete the data processing sub-tasks to obtain a data processing sub-result; and performing data aggregation processing on the data processing sub-results to obtain the target data processing result. This method not only improves the execution efficiency of data processing tasks using relational databases, avoiding technical problems such as lag, timeouts, and memory overflows, but also fully utilizes the various computing sub-nodes in the system to improve system resource utilization.

[0054] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0055] In one embodiment, a data processing apparatus 100 is provided, which corresponds one-to-one with the data processing methods described in the above embodiments. For example... Figure 4 As shown, the data processing device 100 includes a request receiving module 11, a time slicing module 12, a task splitting module 13, a subtask allocation module 14, a data segment acquisition module 15, a subtask execution module 16, and a processing result summarization module 17. Detailed descriptions of each functional module are as follows:

[0056] The request receiving module 11 is used to receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements;

[0057] The time slicing module 12 is used to slice the data in the target data table according to time to obtain a time slicing mapping data table, and store the time slicing mapping data table in the target memory database;

[0058] The task splitting module 13 is used to split the target data processing request into at least one data processing subtask according to the data processing business requirements through the task scheduling center, wherein the data processing subtask includes a subset of the target data table.

[0059] Subtask allocation module 14 is used to send the data processing subtask to at least one computing sub-node, wherein the computing sub-node contains at least one data processing sub-script.

[0060] Data segment acquisition module 15 is used to access the time slice mapping data table in the target memory database using the data processing sub-script, and obtain the corresponding time slice mapping data segment from the time slice mapping data table according to the subset data set;

[0061] Subtask execution module 16 is used to obtain data to be processed from the target relational database according to the time-sliced ​​mapping data segment, and use the data to be processed to complete the data processing subtask through the data processing sub-script to obtain the data processing sub-result;

[0062] The processing result summary module 17 is used to summarize the data sub-results through the task scheduling center to obtain the target data processing result.

[0063] Furthermore, the data segment acquisition module 15 also includes:

[0064] A sub-thread creation splitting sub-module is used to create a preset first number of data processing sub-threads using the data processing sub-script, and to split the data processing sub-tasks and assign them to each of the data processing sub-threads, wherein the sub-data set is split and assigned to the data processing sub-threads;

[0065] The thread pool submodule is used to add the data processing sub-thread to the data processing thread pool of the computing sub-node;

[0066] The sub-thread execution sub-module is used to start the data processing sub-thread to access the corresponding time-slice mapping data table in the memory database according to the allocated sub-dataset, obtain the corresponding time-slice mapping data segment, and lock the time-slice mapping data segment.

[0067] Furthermore, the subtask allocation module 14 also includes:

[0068] The first information query submodule is used to query the resource usage and task allocation information of all current computing sub-nodes through the configuration center;

[0069] The sub-node selection sub-module is used to obtain at least one selected computing sub-node based on the resource occupancy information and the task allocation information, wherein the selected computing sub-node serves as the data processing task sub-node of the target data processing request;

[0070] The first correspondence submodule is used to establish a first correspondence between the target data processing request and the selected computing sub-node, and to store the first correspondence in the configuration center.

[0071] Furthermore, the sub-thread execution submodule also includes:

[0072] The first monitoring subunit is used to monitor the real-time memory usage percentage of the computing sub-node. If the real-time memory usage percentage is greater than a preset first threshold for memory usage percentage, then a preset second number of the data processing sub-threads are paused.

[0073] The second monitoring subunit is used to monitor the third number of surviving data processing sub-threads in the data processing thread pool. When the third number is less than the core thread count of the data processing thread pool, the suspended data processing sub-threads are resumed. The core thread count is the minimum number of threads maintained by the data processing thread pool.

[0074] The thread count increment subunit is used to increase the core thread count of the data processing thread pool by a preset thread count increment percentage if the real-time memory usage percentage is lower than a preset second threshold for memory usage percentage.

[0075] Furthermore, the subtask allocation module 14 also includes:

[0076] The third monitoring submodule is used to monitor the priority of other computing tasks received by the computing subnode, wherein the computing tasks are allocated by the task scheduling center and the priority of the computing tasks is set by the configuration center.

[0077] The subtask pause submodule is used to pause the execution of the data processing subtask and start the execution of the high-priority calculation task if there is a high-priority calculation task with a higher priority than the data processing subtask in the calculation subnode, and record the task breakpoint data index of the data processing subtask in the memory database.

[0078] The subtask recovery submodule is used to restore the data processing subtask to continue execution from the task breakpoint data index after the high-priority calculation task has finished running.

[0079] Furthermore, the processing result summarization module 17 also includes:

[0080] The integrity verification submodule is used to verify the integrity of all data processing sub-results through the task scheduling center;

[0081] The first processing result merging submodule is used to merge the currently obtained data processing sub-results into the first data processing result if there are missing data processing sub-results.

[0082] The task merging submodule is missing. It is used to merge the missing data processing subtasks into a new target data processing request and repeat the above steps to process the new target data processing request to obtain a second data processing result.

[0083] The second processing result merging submodule is used to merge the first data processing result and the second data processing result into the target data processing result.

[0084] The terms "first" and "second" in the above-mentioned modules / units are only used to distinguish different modules / units and are not intended to specify which module / unit has a higher priority or any other limiting meaning. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The module divisions appearing in this application are merely logical divisions; in actual applications, different division methods may be used.

[0085] Specific limitations regarding the data processing device can be found in the limitations regarding the data processing method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0086] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data involved in data processing methods. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0087] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the data processing method described in the above embodiments, for example... Figure 2 The steps S101 to S107 shown, as well as other extensions and related steps of the method, are examples. Alternatively, when the processor executes a computer program, it implements the functions of each module / unit of the data processing device in the above embodiments, for example... Figure 3 The functions of modules 11 to 17 are shown. To avoid repetition, they will not be described again here.

[0088] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0089] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, video data, etc.).

[0090] The memory can be integrated into the processor or it can be set up separately from the processor.

[0091] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the steps of the data processing method described in the above embodiments, for example... Figure 2 The steps S101 to S107 shown, as well as other extensions and related steps of the method, are considered. Alternatively, when a computer program is executed by a processor, it implements the functions of each module / unit of the data processing apparatus in the above embodiments, for example... Figure 3 The functions of modules 11 to 17 are shown. To avoid repetition, they will not be described again here.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0094] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements; The data in the target data table is sharded according to time to obtain a time sharding mapping data table, and the time sharding mapping data table is stored in the target memory database. The data content in the target data table is obtained from the target relational database. During the stage of establishing the time sharding mapping data table, not all the data content in the target data table is obtained, but only the corresponding data row index and the corresponding data row time field are obtained. The data row time field is the dependent data for time sharding. The task scheduling center breaks down the target data processing request into at least one data processing subtask according to the data processing business requirements, wherein the data processing subtask contains a subset of the target data table. The data processing subtask is sent to at least one computing subnode, wherein the computing subnode contains at least one data processing subscript; The data processing sub-script is used to access the time slice mapping data table in the target memory database, and the corresponding time slice mapping data segment is obtained from the time slice mapping data table according to the sub-data set; Data to be processed is obtained from the target relational database according to the time-sliced ​​mapping data segment, and the data processing sub-script is used to complete the data processing sub-task to obtain the data processing sub-result. After successfully obtaining the data to be processed from the target relational database, an exclusive lock is placed on the corresponding time-sliced ​​mapping data segment in the memory database until the exclusive lock is released after the data processing sub-script completes the data processing task. The task scheduling center performs data aggregation and processing on the data processing sub-results to obtain the target data processing result. Before sending the data processing subtask to at least one computing sub-node, the method further includes: querying the resource occupancy information and task allocation information of all current computing sub-nodes through the configuration center; obtaining at least one selected computing sub-node based on the resource occupancy information and the task allocation information, wherein the selected computing sub-node serves as the data processing task sub-node of the target data processing request; establishing a first correspondence between the target data processing request and the selected computing sub-node, and storing the first correspondence in the configuration center; The process further includes, after sending the data processing subtask to at least one computing sub-node, monitoring the priorities of other computing tasks received by the computing sub-node, wherein the computing tasks are allocated by the task scheduling center and their priorities are set by the configuration center; if the computing sub-node has a higher priority computing task than the data processing subtask, the execution of the data processing subtask is paused, the execution of the higher priority computing task begins, and the task breakpoint data index of the data processing subtask is recorded in the memory database; after the higher priority computing task finishes running, the data processing subtask resumes execution from the task breakpoint data index. Before the data processing sub-results are aggregated and processed by the task scheduling center, the process further includes: verifying the completeness of all data processing sub-results through the task scheduling center; if any data processing sub-results are missing, merging the currently obtained data processing sub-results into a first data processing result; merging the missing data processing sub-tasks into a new target data processing request, and repeating the aforementioned steps to process the new target data processing request to obtain a second data processing result; and merging the first data processing result and the second data processing result into the target data processing result.

2. The data processing method according to claim 1, characterized in that, The step of accessing the time-slice mapping data table in the target memory database using the data processing sub-script, and obtaining the corresponding time-slice mapping data segment from the time-slice mapping data table according to the subset dataset includes: The data processing sub-script is used to create a preset first number of data processing sub-threads, and the data processing sub-tasks are split and assigned to each of the data processing sub-threads, wherein the sub-data set is split and assigned to the data processing sub-threads; Add the data processing sub-thread to the data processing thread pool of the computing sub-node; The data processing sub-thread is started to access the corresponding time-slice mapping data table in the memory database according to the allocated subset of data, obtain the corresponding time-slice mapping data segment, and lock the time-slice mapping data segment.

3. The data processing method according to claim 2, characterized in that, After starting the data processing sub-thread to access the corresponding time-slice mapping data table in the memory database according to the allocated subset of data, the method further includes: Monitor the real-time memory usage percentage of the computing sub-nodes. If the real-time memory usage percentage is greater than a preset first threshold for memory usage percentage, then pause a preset second number of the data processing sub-threads. Monitor the third number of surviving data processing sub-threads in the data processing thread pool. When the third number is less than the core thread count of the data processing thread pool, resume the suspended data processing sub-threads. The core thread count is the minimum number of threads maintained by the data processing thread pool. If the real-time memory usage percentage is lower than the second preset memory usage percentage threshold, the number of core threads in the data processing thread pool will be increased by a preset thread count increment percentage.

4. A data processing apparatus, characterized in that, include: The request receiving module is used to receive a target data processing request, wherein the target data processing request includes a target data table in a target relational database and data processing business requirements; The time sharding module is used to shard the data in the target data table according to time to obtain a time sharding mapping data table, and store the time sharding mapping data table in the target memory database. The data content in the target data table is obtained from the target relational database. During the stage of establishing the time sharding mapping data table, not all the data content in the target data table is obtained, but only the corresponding data row index and the corresponding data row time field are obtained. The data row time field is the dependent data for time sharding. The task splitting module is used to split the target data processing request into at least one data processing subtask according to the data processing business requirements through the task scheduling center, wherein the data processing subtask contains a subset of the target data table. A subtask allocation module is used to send the data processing subtask to at least one computing subnode, wherein the computing subnode contains at least one data processing subscript; The data segment acquisition module is used to access the time slice mapping data table in the target memory database using the data processing sub-script, and to obtain the corresponding time slice mapping data segment from the time slice mapping data table according to the subset dataset; The subtask execution module is used to obtain data to be processed from the target relational database according to the time-sliced ​​mapping data segment, and use the data to complete the data processing subtask through the data processing sub-script to obtain the data processing sub-result. After successfully obtaining the data to be processed from the target relational database, an exclusive lock is placed on the corresponding time-sliced ​​mapping data segment in the memory database until the exclusive lock is released after the data processing sub-script completes the data processing task. The processing result aggregation module is used to aggregate the data sub-results through the task scheduling center to obtain the target data processing result. The subtask allocation module further includes: a first information query submodule, used to query the resource usage information and task allocation information of all current computing sub-nodes through the configuration center; a sub-node selection submodule, used to obtain at least one selected computing sub-node based on the resource usage information and the task allocation information, wherein the selected computing sub-node serves as the data processing task sub-node of the target data processing request; and a first correspondence submodule, used to establish a first correspondence between the target data processing request and the selected computing sub-node, and store the first correspondence in the configuration center. The subtask allocation module further includes: a third monitoring submodule, used to monitor the priority of other computing tasks received by the computing subnode, wherein the computing tasks are allocated by the task scheduling center and the priority of the computing tasks is set by the configuration center; a subtask pause submodule, used to pause the execution of the data processing subtask and start the execution of the high-priority computing task if there is a high-priority computing task with a higher priority than the data processing subtask on the computing subnode, and record the task breakpoint data index of the data processing subtask in the memory database; and a subtask recovery submodule, used to resume the execution of the data processing subtask from the task breakpoint data index after the high-priority computing task has finished running. The processing result aggregation module further includes: an integrity verification submodule, used to verify the integrity of all data processing sub-results through the task scheduling center; a first processing result merging submodule, used to merge the currently obtained data processing sub-results into a first data processing result if any data processing sub-results are missing; a missing task merging submodule, used to merge the missing data processing sub-tasks into a new target data processing request, and repeat the aforementioned steps to process the new target data processing request to obtain a second data processing result; and a second processing result merging submodule, used to merge the first data processing result and the second data processing result into the target data processing result.

5. The data processing apparatus according to claim 4, characterized in that, The data segment acquisition module further includes: A sub-thread creation splitting sub-module is used to create a preset first number of data processing sub-threads using the data processing sub-script, and to split and allocate the data processing sub-tasks to each of the data processing sub-threads, wherein the sub-data set is split and allocated to the data processing sub-threads; The thread pool submodule is used to add the data processing sub-thread to the data processing thread pool of the computing sub-node; The sub-thread execution sub-module is used to start the data processing sub-thread to access the corresponding time-slice mapping data table in the memory database according to the allocated sub-dataset, obtain the corresponding time-slice mapping data segment, and lock the time-slice mapping data segment.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data processing method as described in any one of claims 1 to 3.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 3.

Images