A data processing method, device, apparatus, and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment, and a storage medium. The method comprises: determining a configuration item corresponding to a current data processing task, determining a plurality of time intervals corresponding to a current execution time of the current data processing task based on the configuration item, and determining target database operation statements corresponding to each time interval, wherein the time intervals are obtained by dividing the current execution time; executing the target database operation statements in parallel to obtain current business data corresponding to each time interval from a business database, and calculating current index data according to the current business data; and aggregating the current business data and the current index data to obtain a data processing result. The above technical solution realizes unified management and configuration of the current data processing task, divides the current data processing task into a plurality of parallel processes, improves data processing speed, reduces management and operation and maintenance costs, and further improves user experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to data processing technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the development of logistics, the transportation status of large quantities of items needs to be monitored to determine the business volume within a preset time period. Logistics systems include multiple business systems, such as the outbound order system, sorting system, carrier system, scheduling system, and warehousing system. The business data contained in these business systems is used to monitor the transportation status of items.

[0003] The business data contained in each business system is also used to determine indicator data related to order volume. In actual applications, it is necessary to calculate multiple indicator data based on the business system, and the calculation of indicator data often involves business data from multiple business systems.

[0004] In the process of implementing the present invention, it was found that the prior art has at least the following technical problems:

[0005] The amount of business data is in the tens of millions, and the calculation of indicator data requires a lot of complex business logic, and the calculation speed is slow. Summary of the Invention

[0006] The present invention provides a data processing method, apparatus, device and storage medium to improve the calculation speed of indicator data.

[0007] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0008] After determining the configuration item corresponding to the current data processing task, determining the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time;

[0009] Executing the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating the current indicator data based on each of the current business data;

[0010] Summarize the current business data and the current indicator data to obtain a data processing result.

[0011] Furthermore, before determining the target database operation statements corresponding to the respective time intervals based on the multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, the method further includes:

[0012] Configure the current execution time and time partition step for the configuration item;

[0013] The current execution time is divided into a plurality of time intervals based on the time partition step.

[0014] Furthermore, based on a plurality of time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, determining a target database operation statement corresponding to each of the time intervals includes:

[0015] Determining an original database operation statement for obtaining original business data within the current execution time and calculating original indicator data based on the original business data;

[0016] The original database operation statements are configured according to the start time and end time corresponding to each time interval to obtain the target database operation statement.

[0017] Furthermore, after executing the target database operation statements in parallel to obtain the current business data corresponding to the time intervals from the business database and calculating the current indicator data based on the current business data, the method further includes:

[0018] The previous indicator data in each time interval within the previous execution time of the current data processing task is updated based on the current indicator data.

[0019] Furthermore, when the configuration item corresponds to multiple data processing tasks, each of the data processing tasks is configured with a switch control valve, and the method further includes:

[0020] The status information of each data processing task is determined according to the switch control valve corresponding to each data processing task.

[0021] Furthermore, executing the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating the current indicator data based on each of the current business data, includes:

[0022] Execute the target database operation statements in parallel to obtain from the business database multiple groups of business data corresponding to each of the time intervals required for multiple data processing tasks with status information of the switch control valve being opened, and calculate the indicator data corresponding to the multiple data processing tasks based on each group of business data.

[0023] Furthermore, the current business data and the current indicator data are aggregated to obtain a data processing result, including:

[0024] Determine data calculation dimensions based on business needs;

[0025] The current business data and the current indicator data of each time interval are aggregated according to the data calculation dimension to obtain the data processing result.

[0026] Furthermore, after aggregating the current business data and the current indicator data to obtain the data processing result, the method further includes:

[0027] Determining an output path for the data processing result;

[0028] The data processing result is output in a message form based on the output path.

[0029] In a second aspect, an embodiment of the present invention further provides a data processing device, including:

[0030] a determination module configured to, after determining a configuration item corresponding to a current data processing task, determine a target database operation statement corresponding to each of a plurality of time intervals corresponding to a current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time;

[0031] An execution module, configured to execute the target database operation statements in parallel, so as to obtain the current business data corresponding to each of the time intervals from the business database, and calculate the current indicator data based on each of the current business data;

[0032] The summarizing module is used to summarize the current business data and the current indicator data to obtain data processing results.

[0033] In a third aspect, an embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in any one of the first aspects is implemented.

[0034] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to execute the data processing method as described in any one of the first aspects.

[0035] The embodiments of the above invention have the following advantages or beneficial effects:

[0036] An embodiment of the present invention provides a data processing method, comprising: after determining a configuration item corresponding to a current data processing task, determining a target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time; executing each of the target database operation statements in parallel to obtain current business data corresponding to each of the time intervals from a business database, and calculating current indicator data based on each of the current business data; and summarizing each of the current business data and the current indicator data to obtain a data processing result. The above technical solution can first determine the configuration item corresponding to the current data processing task, and determine the target database operation statement according to the current execution time of the current data processing task corresponding to the configuration item corresponding to multiple time intervals. Since the target database operation statement is configured with time information, different target database operation statements can be executed in each time interval, and each target database operation statement can be executed in parallel. In the process of executing the target database operation statement, the current business data in each time interval is obtained from the business system database, and the current indicator data is determined based on the current business data. When obtaining the current business data, the business data of the entire execution time is queried once instead of multiple queries for the current business data of each time interval. The time interval is used to reduce the pressure of database queries, and the current business data and current indicator data are summarized to obtain the data processing results, thereby realizing unified management and configuration of the current data processing task. The current data processing task is divided into multiple parallel execution processes according to the time interval, reducing the data processing amount of each process, improving the data processing speed, reducing the management and operation and maintenance costs, and further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a logistics system provided by an embodiment of the present invention;

[0038] Figure 2 A flowchart of a data processing method provided by an embodiment of the present invention;

[0039] Figure 3 A flowchart of another data processing method provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a configuration center provided in an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of time partitioning provided by an embodiment of the present invention;

[0042] Figure 6a A schematic diagram of a compensation mechanism provided by an embodiment of the present invention; Figure 6b A schematic diagram of another compensation mechanism provided by an embodiment of the present invention; Figure 6c A schematic diagram of another compensation mechanism provided by an embodiment of the present invention;

[0043] Figure 7 A schematic structural diagram of a data processing device provided by an embodiment of the present invention;

[0044] Figure 8 A schematic structural diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0046] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. When its operation is completed, the process can be terminated, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the features in the embodiments of the present invention and the embodiments can be combined with each other without conflict.

[0047] Figure 1 A schematic diagram of a logistics system provided by an embodiment of the present invention, such as Figure 1 As shown, the logistics system may include multiple business systems, such as an external order system, a sorting system, a carrier system, a scheduling system, and a warehousing system. The database for storing business data contained in the business system may be MySQL, and each system provides an API interface for querying business data, so that when the SQL statement corresponding to the index data calculation is run, the business data can be obtained from the database corresponding to each business system based on the API interface. The process of obtaining business data from the database based on the SQL statement can be understood as the process of querying the database based on the SQL statement. The SQL statement directly queries the database, which may be slow. The execution of multiple or even hundreds of complex logical SQL statements puts a lot of pressure on the database, which may result in slow SQL, poor query experience and performance. After obtaining the business data, the corresponding business processing logic is calculated based on the pre-set index data, and the index data can be obtained by processing the business data. When calculating multiple index data, the R&D cost of calculating each index data is high and cannot be managed uniformly, which cannot meet the needs of unified management.

[0048] The data processing method provided by the embodiments of the present invention will be described in detail below in conjunction with various embodiments.

[0049] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to situations where the data processing speed needs to be improved. The method can be executed by a data processing device, which can be implemented by software and / or hardware. Figure 2 Said method specifically comprises the following steps:

[0050] Step 210: After determining the configuration item corresponding to the current data processing task, determine the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item.

[0051] The current data processing task can be to obtain current business data and calculate current indicator data based on the current business data. The current business data can be obtained from various business systems included in the logistics system, and the current indicator data can be indicator data related to order volume. The time interval is divided according to the execution time of the configuration item.

[0052] Specifically, a target database operation statement for executing the current data processing task may be determined, specifically a target database operation statement for obtaining current business data from the business system and calculating current indicator data based on the current business data may be determined.

[0053] In practical applications, the configuration center can be used to configure corresponding configuration items for the current data processing task and assign unique codes to the configuration items, allowing each configuration item to be executed on a thread-by-thread basis. The configuration center can also be used to configure corresponding execution times and time partition steps for the configuration items, allowing the execution time to be divided into multiple time intervals based on the time partition steps. Of course, the configuration center can also be used to configure multiple time intervals corresponding to the current execution time for target database operation statements, thereby obtaining the target database operation statement for the current data processing task in embodiments of the present invention.

[0054] The configuration center configures time information for the target database operation statement, that is, the target database operation statement configured with the time interval can be executed in multiple time intervals corresponding to the current execution time.

[0055] In an embodiment of the present invention, time information is configured for the target database operation statement so that different target database operation statements are executed in each time interval. By running each target database operation statement in multiple threads, the data processing results of each time interval can be determined separately, thereby reducing the computational amount of data processing in each time interval and improving data processing efficiency.

[0056] Step 220: Execute the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculate the current indicator data based on each of the current business data.

[0057] Specifically, multiple thread executors can be executed in parallel based on the parallel statistician to execute in parallel the aforementioned target database operation statements configured with time information, that is, the target database operation statements corresponding to each time interval, to obtain the current business data in each time interval from the business system, and calculate the current indicator data corresponding to each time interval based on the current business data in each time interval.

[0058] Of course, the parallel counter can also cache the current indicator data corresponding to each time interval.

[0059] In an embodiment of the present invention, the thread executor can realize the parallel execution of the target database operation statements corresponding to each time interval to obtain the current business data within each time interval, and calculate the current indicator data corresponding to each time interval based on the current business data, thereby reducing the amount of data processing while improving the speed of data processing.

[0060] Step 230: Summarize the current business data and the current indicator data to obtain a data processing result.

[0061] Specifically, based on the parallel statistics device, the current business data obtained in each time interval included in the execution time of the current data processing task and the current indicator data obtained by calculating the current business data can be summarized, and the summary result can be determined as the data processing result.

[0062] In an embodiment of the present invention, the current data processing task within the execution time is divided into current data processing tasks within multiple time intervals, and the current data processing task is executed within each time interval to obtain the current indicator data within each time interval. The current business data obtained by executing the current data processing task within each time interval and the current indicator data obtained by calculating the current business data are summarized to obtain the data processing results, which reduces the data processing amount within each time interval, improves the data processing speed, reduces the management and operation and maintenance costs, and improves the user experience.

[0063] A data processing method provided by an embodiment of the present invention includes: after determining the configuration item corresponding to the current data processing task, based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, determining the target database operation statement corresponding to each time interval, wherein the time interval is obtained by dividing the current execution time; executing each of the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating current indicator data based on each of the current business data; and summarizing each of the current business data and the current indicator data to obtain a data processing result. The above technical solution can first determine the configuration item corresponding to the current data processing task, and determine the target database operation statement according to the current execution time of the current data processing task corresponding to the configuration item corresponding to multiple time intervals. Since the target database operation statement is configured with time information, different target database operation statements can be executed in each time interval, and each target database operation statement can be executed in parallel. In the process of executing the target database operation statement, the current business data in each time interval is obtained from the business system database, and the current indicator data is determined based on the current business data. When obtaining the current business data, the business data of the entire execution time is queried once instead of multiple queries for the current business data of each time interval. The time interval is used to reduce the pressure of database queries, and the current business data and current indicator data are summarized to obtain the data processing results, thereby realizing unified management and configuration of the current data processing task. The current data processing task is divided into multiple parallel execution processes according to the time interval, reducing the data processing amount of each process, improving the data processing speed, reducing the management and operation and maintenance costs, and further improving the user experience.

[0064] Figure 3A flowchart of another data processing method provided by an embodiment of the present invention, which can be applied to situations where the data processing speed needs to be improved. Based on the above embodiment, the embodiment of the present invention adds "configuring the current execution time and time partition step for the configuration item; dividing the current execution time into multiple time intervals based on the time partition step" before determining the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item. After executing each target database operation statement in parallel to obtain the current business data corresponding to each time interval from the business database and calculating the current indicator data based on each current business data, "update the previous indicator data in each time interval within the previous execution time of the current data processing task based on the current indicator data" is added. After summarizing each current business data and the current indicator data to obtain the data processing result, "determine the output path of the data processing result, and output the data processing result in the form of a message based on the output path." The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. See Figure 3 , the data processing method provided by the embodiment of the present invention includes:

[0065] Step 310: Determine the configuration items of the current data processing task, and configure the switch control valve for the current data processing task.

[0066] The current data processing task corresponds to a unique configuration item, and a configuration item can correspond to multiple data processing tasks. That is, when the target database operation statement corresponding to the configuration item is executed, multiple data processing tasks can be executed simultaneously.

[0067] Figure 4 A schematic diagram of a configuration center provided in an embodiment of the present invention, such as Figure 4 As shown, the configuration center may include a basic configuration module, a query configuration module and a processing configuration module. The basic configuration module may include a configuration coding unit, in which corresponding configuration items may be configured for data processing, and a unique code may be determined for the configuration item to execute each configuration item based on a thread.

[0068] The switch control valve can control whether the current data processing task can be executed. When the switch control valve is open, it indicates that the current data processing task corresponding to the switch control valve can be executed when the target database operation statement is executed; when the switch control valve is closed, it indicates that the current data processing task corresponding to the switch control valve cannot be executed when the target database operation statement is executed.

[0069] In an embodiment of the present invention, the switch control valve can be used to control whether to execute the data processing tasks corresponding to the configuration items when executing the target database operation statement, so as to execute the actually required data processing tasks based on the configuration items according to actual needs and avoid wasting processes.

[0070] Step 320: Configure an execution time and a time partition step for the configuration item; and divide the execution time into multiple time intervals based on the time partition step.

[0071] Specifically, Figure 4 The query configuration module of the configuration center shown in the figure can include a time generation unit and a time partition step configuration unit. Therefore, the execution time corresponding to the configuration item can be configured in the time generation unit, and the time partition step can be configured in the time partition step configuration unit. After configuring the execution time and time partition step for the configuration item, the execution time is divided into multiple time intervals based on the time partition step using the time partition strategy. For example, if the execution time is ["2022-01-01 00:00:00", "2022-01-01 23:59:59"] and the time partition step is 60 minutes, ["2022-01-01 00:00:00", "2022-01-01 23:59:59"] will be partitioned into ["2022-01-01 00:00:00", "2022-01-01 00:59:59"], ["2022-01-01 01:00:00", "2022-01-01 01:59:59"], ["2022-01-01 02:00:00", "2022-01-01 2:59:59”]…[“2022-01-01 23:00:00”, “2022-01-01 23:59:59”].

[0072] Furthermore, a common interface can be dynamically called based on JSF to obtain the execution time and time partition step based on the common interface. The execution time and time partition step are then sent to the time segmenter, which then divides the execution time into intervals according to the 60-minute time partition step. Of course, after completing the interval division of the execution time, the time segmenter can return the obtained time intervals to the configuration center.

[0073] It should be noted that when the amount of data in the business system surges, that is, when the amount of business data in any business system within the preset time is greater than the preset value, the time partition step can be updated to divide the execution time into smaller time intervals, further reducing the amount of data processed each time.

[0074] In actual applications, the execution time can be divided into the following two situations: ① When calculating indicator data from 0:00 to the current time, for example, calculating the order volume based on the order acceptance time, the execution time is ["2020-06-18 00:00", "2020-06-18 10:10"], which can be divided into multiple time intervals such as ["2020-06-18 00:00", "2020-06-18 00:15"], ["2020-06-18 15:00", "2020-06-18 30:00"]....["2020-06-18 10:00", "2020-06-18 10:10"]. ② When calculating the indicator data of the all-day time dimension from 00:00 to 23:59, a total of 96 time partitions are divided into 15-minute units. Figure 5 A schematic diagram of time partitioning provided by an embodiment of the present invention, such as Figure 5 As shown, the execution time can be [00:00, 23:59], and the time partition step is 15 minutes. Therefore, the time period corresponding to the execution time is divided into time partitions according to the unit of 15 minutes, and the last partition is <= 15 minutes.

[0075] In the embodiment of the present invention, the execution time is divided in the configuration center in conjunction with the time segmenter, and the pressure of database query is reduced by using time partitioning.

[0076] Step 330: Based on a plurality of time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, determine a target database operation statement corresponding to each of the time intervals.

[0077] Configuration items correspond to multiple data processing tasks, and time intervals are divided according to the execution time of the configuration items. If the business system database is MySQL, the target database operation statement can be an SQL statement.

[0078] In one implementation, step 330 may specifically include:

[0079] According to the switch control valve corresponding to each of the data processing tasks, the status information of each of the data processing tasks is determined; the original database operation statement corresponding to the data processing task whose status information is that the switch control valve is open, which obtains business data and calculates indicator data based on the business data, is determined; the original database operation statement is configured according to the start time and end time corresponding to each of the time intervals to obtain the target database operation statement.

[0080] Specifically, Figure 4The basic configuration module of the configuration center shown also includes a switch configuration unit, which can configure the status information of the switch control valve for each of the multiple data processing tasks corresponding to the configuration items. The switch control valve can be configured to open or close for each data processing task. If the switch control valve is configured to open for a data processing task, it indicates that the data processing task corresponding to the switch control valve can be executed when the SQL statement corresponding to the configuration item is executed; if the switch control valve is configured to close for a data processing task, it indicates that the data processing task corresponding to the switch control valve cannot be executed when the SQL statement corresponding to the configuration item is executed.

[0081] Therefore, the original SQL statement corresponding to the executable data processing task can be determined, that is, the original SQL statement corresponding to the data processing task for determining that the state information of the switch control valve is that the switch control valve is open. Specifically, the original SQL statement corresponding to each data processing task for determining that the state information of the switch control valve is that the switch control valve is open can be determined to obtain business data and calculate indicator data based on the business data.

[0082] Figure 4 The query configuration module of the configuration center shown also includes an SQL statement unit, which can determine the state information of the on-off control valve as the original SQL statement corresponding to each data processing task for obtaining business data and calculating indicator data based on the business data. The SQL statement unit can also be configured with time information for each original SQL statement. Specifically, the start time and end time corresponding to each time interval can be used as time information to configure the original SQL statement to obtain a target SQL statement. The target SQL statement is executed only within the time interval formed by the start and end times included in the time information, thereby obtaining business data within the time interval and calculating indicator data based on the business data.

[0083] In an embodiment of the present invention, time information is configured for the target database operation statement so that different target database operation statements can be executed in each time interval. Each target database operation statement can be run in multiple threads to execute the current data processing task in each time interval, thereby reducing the computational amount of data processing in each time interval and improving data processing efficiency.

[0084] Step 340: Execute the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculate the current indicator data based on the current business data.

[0085] Specifically, the target database operation statement can be delivered to the parallel enumerator for parallel computing. The parallel enumerator can execute multiple thread executors in parallel and use the thread executors to execute the target database operation statement to obtain current business data from the database of the business system.

[0086] Figure 4 The data query configuration model of the configuration center shown also includes a data source unit, which can be used to configure the enumeration value of business data in the business system connected to the data source unit, that is, the current business data can be obtained from the database corresponding to the business system based on the data source unit.

[0087] In actual applications, parallel counters may include DB parallel counters, JSF parallel counters and custom parallel counters, and thread executors may include JimDB executor, JSF executor, DB executor and business system special executor.

[0088] When the current business data required to execute the current data processing task cannot be obtained based on the target database operation statement, that is, when the target database operation statement obtains the current business data from the database of the preset business system, the API interface of the preset business system can be called based on the business system special executor or JSF executor to obtain the current business data from the database of the preset business system based on the API interface.

[0089] Of course, after obtaining the current business data corresponding to each time interval from the business system database according to the time interval, the current business data for each time partition can be cached. Then, the target database operation statement can be continued to be executed, and the current indicator data for each time interval can be calculated based on the current business data of each time partition.

[0090] After executing step 340, the method further includes: updating the previous indicator data in each time interval within the previous execution time of the current data processing task based on the current indicator data.

[0091] The compensation mechanism can be used to compensate the previous data processing result of the previous time partition included in the previous execution time based on the current data processing result of the current time partition included in the current execution time, thereby ensuring the real-time performance of the data in the cache. Specifically, the 24 hours in a day, namely ["00:00:00", "23:59:59"], can be divided into 24 time intervals ["00:00:00", "00:59:59"], ["2020-06-18 The time intervals are divided into four time units. For example, ["00:00:00", "00:59:59"] is divided into ["00:00:00", "00:14:59"], ["15:00:00", "29:59:59"], ["30:00:00", "44:59:59"], ["45:00:00", "59:59:59"], and so on. Other time intervals can also be divided into four time units.

[0092] Figure 6a A schematic diagram of a compensation mechanism provided by an embodiment of the present invention is shown in FIG. Figure 6aAs shown, if the start time of the time partition corresponding to the configuration item's execution time is not within one hour of the current time and is not at 23:00, the minutes of the current time are used as the start time of each time partition. For example, if the current time is 02:15, the start time of the previous time partition may be 01:15. Therefore, based on the target database operation statement configured with this time partition, the current business data can be obtained from the database and the current time partition cache can be updated. Based on the current business data obtained at ["02:15", "02:30"] and the current indicator data calculated, the previous business data obtained at ["01:15", "01:30"] and the previous indicator data calculated are updated; similarly, the current business data obtained at ["02:30", "02:45"] and the current indicator data calculated are updated to update the previous business data obtained at ["01:30", "01:45"] and the previous indicator data calculated; the current business data obtained at ["02:45", "03:00"] and the current indicator data calculated are updated to update the previous business data obtained at ["01:45", "02:00"] and the previous indicator data calculated, so as to update the previous business data and the previous indicator data in the previous time interval according to the current business data and the current indicator data in the current time interval. Of course, you can call the scheduled task execution frequency, starting at 02:15, and perform three updates to update the previous business data and indicator data for the previous time partition corresponding to the current execution time. In actual applications, you can use the target database operation statement + date + hour + minute as the key to lock, allowing the current time partition to update the previous business data and indicator data in the previous time partition corresponding to the previous execution time, preventing multiple executions from placing excessive pressure on the system.

[0093] Figure 6b A schematic diagram of another compensation mechanism provided by an embodiment of the present invention is shown in FIG. Figure 6b As shown, if the start time of the time partition corresponding to the configuration item's execution time is within one hour of the current time, to ensure that the most recently calculated current indicator data is the latest indicator data, you can use the current time as the start time of the time partition and execute the target database operation statement to obtain the current business data from the database and calculate the current indicator data based on the current business data. For example, if the current time is 03:00, you can obtain the current business data from the database in the range ["03:00", "03:15"], ["03:15", "03:30"], ["03:30", "03:45"], ["03:45", "04:00"] in sequence and calculate the current indicator data based on the current business data.

[0094] Figure 6c A schematic diagram of another compensation mechanism provided by an embodiment of the present invention is shown as follows: Figure 6c As shown in the figure, if the start time of the time partition corresponding to the execution time of the configuration item is 23:00, since the four time units at 23:00 are the last four units of the last day, and the hour after 23:00 is the next day, there is no need to update the business data obtained in the 23 hours of that day. Therefore, when executing SQL statements within 23 hours, the current business data can be obtained from the database in ["23:00", "23:15"], ["23:15", "23:30"], ["23:30", "23:45"], ["23:45", "00:00"] in sequence, and the current indicator data can be calculated based on the current business data.

[0095] In addition, when a configuration item corresponds to multiple data processing tasks, the target database operation statements are executed in parallel. Multiple sets of business data corresponding to each time interval required for multiple data processing tasks with status information of opening the switch control valve can be obtained from the business database, and the indicator data corresponding to the multiple data processing tasks can be calculated based on each set of business data.

[0096] A configuration item corresponds to multiple data processing tasks. When executing the target database operation statement corresponding to the configuration item, the status information of each data processing task can be used to determine whether to execute each data processing task. Specifically, when executing the target database operation statement corresponding to the configuration item, only the data processing task whose status information indicates that the switch control valve is open will be executed. During the parallel execution of the target database operation statements, each data processing task requires access to different business data. Therefore, multiple sets of business data must be acquired during each time interval. Each data processing task whose status information indicates that the switch control valve is open can correspond to a set of business data. Furthermore, the indicator data corresponding to the data processing task can be calculated based on each set of business data.

[0097] In this embodiment of the present invention, the thread executor can execute the target database operation statements corresponding to each time interval in parallel, thereby obtaining current business data within each time interval and calculating the current indicator data corresponding to each time interval based on the current business data, thereby reducing the amount of data processed while increasing the speed of data processing. Of course, a compensation mechanism can be used to update the current business data and calculated current indicator data obtained in the previous time interval corresponding to the current time interval to ensure the real-time performance of the data in the cache.

[0098] Step 350: Summarize the current business data and the current indicator data to obtain a data processing result.

[0099] Specifically, using the summary calculation method, multiple threads execute and traverse each thread result set; multiple results summarize all the current business data obtained and the current indicator data calculated into the final result set according to the unique identifier to obtain the data processing result.

[0100] In one implementation, step 350 may specifically include:

[0101] Determine the data calculation dimension according to business needs; summarize the current business data and the current indicator data in each time interval according to the data calculation dimension to obtain the data processing result.

[0102] like Figure 4 The processing configuration module report of the configuration center shown includes a result summary unit and a data conversion unit. Therefore, in the data conversion unit, the data conversion strategy can be configured according to business needs to determine the data calculation dimension.

[0103] In the "Distribution Center Dimension," you can aggregate current business data and indicator data within a preset city to obtain data processing results for that city. In the "Region Dimension," you can aggregate current business data and indicator data within a preset region to obtain data processing results for that region. If the business requirement is to obtain current indicator data within a region, the data conversion strategy is to convert the "Distribution Center Dimension" to the "Region Dimension." Therefore, the data calculation dimension can be determined to be the region dimension, and the current business data and indicator data for the "Region Dimension" can be aggregated to obtain data processing results for the "Region Dimension."

[0104] In an embodiment of the present invention, the current business data obtained by executing the current data processing task in each time interval and the current indicator data obtained by calculation are summarized to obtain data processing results of various dimensions, thereby improving data processing speed, reducing management and operation and maintenance costs, and improving user experience.

[0105] Step 360: Determine an output path for the data processing result, and output the data processing result in a message form based on the output path.

[0106] like Figure 4 The processing configuration module of the configuration center shown in the figure also includes an output unit, such as Figure 4As shown, the output path corresponding to the data processing result can be configured based on the output unit. Specifically, the JsonPath plug-in can be used to fill the data processing result into the json format message configuration path. For example, when the indicator data is the amount of tasks to be collected, the output path corresponding to the business data and the amount of tasks to be collected can be configured as "$.pickUp.total.shouldPickUpTaskCount" based on the output unit. Then, you can first find the pickUp node in the first level, and then find the node with the total in the second level under the pickup node, and then find the third-level node under the node. If it does not exist, create a third-level node named "shouldPickUpTaskCount" and store the data in the node. If it exists, directly store the data in the node, and output the data processing result in the form of a message.

[0107] In practical applications, the data processing results can be output to a display page for display, so as to facilitate real-time monitoring of the data processing results.

[0108] In the embodiment of the present invention, the output of data processing results is achieved based on the message.

[0109] The data processing method provided by an embodiment of the present invention includes: determining a configuration item of a current data processing task; configuring an execution time and a time partition step for the configuration item; dividing the execution time into multiple time intervals based on the time partition step; determining a target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item; executing each target database operation statement in parallel to obtain current business data corresponding to each time interval from a business database, and calculating current indicator data based on each current business data; summarizing each current business data and the current indicator data to obtain a data processing result; determining an output path for the data processing result, and outputting the data processing result in the form of a message based on the output path. The above technical solution can first determine the configuration item corresponding to the data processing and configure the corresponding execution time and time partition step for the configuration item. Then, based on the time partition step, the execution time can be divided into multiple time intervals. Each time interval is used as time information to configure the original database operation statement used to obtain current business data and calculate current indicator data based on the current business data to obtain the target database operation statement. The time information is configured for the target database operation statement so that different target database operation statements are executed in each time interval respectively. Each target database operation statement is executed in parallel. During the execution of the target database operation statement, the current business data in each time interval is obtained from the business system database and the current indicator data is determined based on the current business data. When obtaining the current business data, the business data of the entire execution time is changed from being queried once to the business data of each time interval multiple times. The time interval is used to reduce the pressure of database queries, and the current business data and current indicator data are summarized to obtain the data processing results, thereby realizing unified management and configuration of the current data processing task. The current data processing task is divided into multiple parallel execution processes according to the time interval, reducing the data processing volume of each process, improving the data processing speed, reducing management and operation and maintenance costs, and further improving the user experience.

[0110] Furthermore, when a configuration item corresponds to multiple data processing tasks, the target database operation statements are executed in parallel. Multiple sets of business data corresponding to the time intervals required for multiple data processing tasks, including the status information for the on / off control valve opening, are retrieved from the business database. Based on these sets of business data, the corresponding indicator data for the multiple data processing tasks is calculated. This ensures that the required data processing tasks are executed according to actual needs, avoiding wasted processing. Furthermore, a compensation mechanism enables updates to both business and indicator data, improving data processing accuracy.

[0111] Figure 7This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. This device and the data processing methods of the above embodiments belong to the same inventive concept. For details not fully described in the embodiments of the data processing device, reference can be made to the embodiments of the above data processing methods.

[0112] The specific structure of the data processing device is as follows: Figure 7 Shown, including:

[0113] Determination module 710 is configured to, after determining the configuration item corresponding to the current data processing task, determine a target database operation statement corresponding to each of multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time;

[0114] An execution module 720 is configured to execute the target database operation statements in parallel to obtain current business data corresponding to each time interval from the business database, and calculate current indicator data based on the current business data;

[0115] The summarizing module 730 is used to summarize the current business data and the current indicator data to obtain a data processing result.

[0116] Based on the above embodiment, the device further includes:

[0117] The partitioning module is used to configure the execution time and the time partition step for the configuration item; and divide the execution time into multiple time intervals based on the time partition step.

[0118] Based on the above embodiment, when the configuration item corresponds to multiple data processing tasks, each of the data processing tasks is configured with a switch control valve. Accordingly, the determination module 710 is specifically configured to:

[0119] According to the switch control valve corresponding to each of the data processing tasks, the status information of each of the data processing tasks is determined; the original database operation statement corresponding to the data processing task whose status information is that the switch control valve is open, which obtains business data and calculates indicator data based on the business data, is determined; the original database operation statement is configured according to the start time and end time corresponding to each of the time intervals to obtain the target database operation statement.

[0120] Based on the above embodiment, the device further includes:

[0121] An updating module is used to update the previous indicator data in each time interval within the previous execution time of the current data processing task based on the current indicator data.

[0122] Based on the above embodiment, the execution module 720 is specifically configured to:

[0123] Execute the target database operation statements in parallel to obtain from the business database multiple groups of business data corresponding to each of the time intervals required for multiple data processing tasks with status information of the switch control valve being opened, and calculate the indicator data corresponding to the multiple data processing tasks based on each group of business data.

[0124] Based on the above embodiment, the summarizing module 730 is specifically configured to:

[0125] Determine the data calculation dimension according to business needs; summarize the current business data and the current indicator data in each time interval according to the data calculation dimension to obtain the data processing result.

[0126] Based on the above embodiment, the device further includes:

[0127] The output module is used to determine an output path for the data processing result and output the data processing result in a message form based on the output path.

[0128] The data processing device provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.

[0129] It is worth noting that in the embodiment of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0130] Figure 8 A schematic structural diagram of a computer device provided in an embodiment of the present invention. Figure 8 A block diagram of an exemplary computer device 8 suitable for use in implementing embodiments of the present invention is shown. Figure 8 The computer device 8 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Figure 8 As shown, computer device 8 is a general-purpose computing computer device. Components of computer device 8 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components (including system memory 28 and processing unit 16).

[0132] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0133] Computer device 8 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 8, including volatile and non-volatile media, removable and non-removable media.

[0134] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 8 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0135] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0136] The computer device 8 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the computer device 8, and / or any device that enables the computer device 8 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 8 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. Figure 8 As shown, the network adapter 20 communicates with other modules of the computer device 8 via the bus 18. Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with computer device 8, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, for example, implementing the data processing method provided in the embodiment of the present invention, which includes:

[0138] After determining the configuration item corresponding to the current data processing task, determining the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time;

[0139] Executing the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating the current indicator data based on each of the current business data;

[0140] Summarize the current business data and the current indicator data to obtain a data processing result.

[0141] Of course, those skilled in the art will understand that the processor may also implement the technical solution of the data processing method provided by any embodiment of the present invention.

[0142] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the data processing method provided in the embodiment of the present invention is implemented, for example, and the method includes:

[0143] After determining the configuration item corresponding to the current data processing task, determining the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time;

[0144] Executing the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating the current indicator data based on each of the current business data;

[0145] Summarize the current business data and the current indicator data to obtain a data processing result.

[0146] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0147] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0149] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0150] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0151] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention comply with the relevant provisions of national laws and regulations.

[0152] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: After determining the configuration item corresponding to the current data processing task, determining the target database operation statement corresponding to each time interval based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time; Executing the target database operation statements in parallel to obtain the current business data corresponding to each of the time intervals from the business database, and calculating the current indicator data based on each of the current business data; Summarize the current business data and the current indicator data to obtain a data processing result; Wherein, based on multiple time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, determining the target database operation statement corresponding to each time interval includes: Determining an original database operation statement for obtaining original business data within the current execution time and calculating original indicator data based on the original business data; According to the start time and end time corresponding to each time interval, the original database operation statement is respectively configured to obtain the target database operation statement; When the configuration item corresponds to multiple data processing tasks, each of the data processing tasks is configured with a switch control valve, and the method further includes: Determining status information of each data processing task according to the switch control valve corresponding to each data processing task; Executing the target database operation statements in parallel to obtain the current business data corresponding to each time interval from the business database, and calculating the current indicator data based on each current business data, including: Execute the target database operation statements in parallel to obtain from the business database multiple groups of business data corresponding to each of the time intervals required for multiple data processing tasks with status information of the switch control valve being opened, and calculate the indicator data corresponding to the multiple data processing tasks based on each group of business data.

2. The data processing method according to claim 1, wherein: Before determining target database operation statements corresponding to respective time intervals based on a plurality of time intervals corresponding to the current execution time of the current data processing task corresponding to the configuration item, the method further includes: Configure the current execution time and time partition step for the configuration item; The current execution time is divided into a plurality of time intervals based on the time partition step.

3. The data processing method according to claim 1, wherein: After executing the target database operation statements in parallel to obtain the current business data corresponding to the time intervals from the business database and calculating the current indicator data based on the current business data, the method further includes: The previous indicator data in each time interval within the previous execution time of the current data processing task is updated based on the current indicator data.

4. The data processing method according to claim 1, wherein: Summarize the current business data and the current indicator data to obtain data processing results, including: Determine data calculation dimensions based on business needs; The current business data and the current indicator data of each time interval are aggregated according to the data calculation dimension to obtain the data processing result.

5. The data processing method according to claim 1, wherein: After aggregating the current business data and the current indicator data to obtain the data processing result, the method further includes: Determining an output path for the data processing result; The data processing result is output in a message form based on the output path.

6. A data processing device, characterized in that: include: a determination module configured to, after determining a configuration item corresponding to a current data processing task, determine a target database operation statement corresponding to each of a plurality of time intervals corresponding to a current execution time of the current data processing task corresponding to the configuration item, wherein the time intervals are obtained by dividing the current execution time; An execution module, configured to execute the target database operation statements in parallel, so as to obtain the current business data corresponding to each of the time intervals from the business database, and calculate the current indicator data based on each of the current business data; A summary module, used to summarize the current business data and the current indicator data to obtain data processing results; The determining module is specifically configured to: Determining an original database operation statement for obtaining original business data within the current execution time and calculating original indicator data based on the original business data; According to the start time and end time corresponding to each time interval, the original database operation statement is respectively configured to obtain the target database operation statement; When the configuration item corresponds to multiple data processing tasks, each of the data processing tasks is configured with a switch control valve. Accordingly, the determination module is specifically configured to: Determining status information of each data processing task according to the switch control valve corresponding to each data processing task; The execution module is specifically used to: Execute the target database operation statements in parallel to obtain from the business database multiple groups of business data corresponding to each of the time intervals required for multiple data processing tasks with status information of the switch control valve being opened, and calculate the indicator data corresponding to the multiple data processing tasks based on each group of business data.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method according to any one of claims 1 to 5 is implemented.

8. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions are used to perform the data processing method according to any one of claims 1 to 5 when executed by a computer processor.

Images