Business data processing method and device, electronic equipment and storage medium

By splitting business data into shards and generating sub-task processing, the problem of excessive processing time in the logistics settlement system is solved, more efficient data processing is achieved, and hardware costs and resource waste are reduced.

CN118939406APending Publication Date: 2024-11-12BEIJING JINGDONG QIANSHITECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310519523.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, when the logistics settlement system generates logistics bills, the processing efficiency is inefficient due to the long processing time of massive business data.

Method used

By splitting the service data into at least two pieces of shard data, generating subtasks corresponding to each piece of shard data, processing them separately, and finally synthesizing the processing results of the business data based on the results of the subtask, thereby shortening the processing time.

Benefits of technology

Through sharding processing, the processing time of business data is significantly shortened, the long wait required for centralized processing is avoided, and the problem of increasing hardware to reduce costs and resource waste is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939406A_ABST
    Figure CN118939406A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a business data processing method and device, electronic equipment and a storage medium. The method comprises the steps of determining to-be-processed business data in response to a business data processing instruction, and generating a processing task for processing the business data; splitting the business data into at least two pieces of fragmented data by executing the processing task, and generating sub-tasks respectively corresponding to each piece of fragmented data in the at least two pieces of fragmented data; for each sub-task in the at least two generated sub-tasks, processing the fragmented data corresponding to the sub-task by executing the sub-task to obtain a processing result of the fragmented data; and under the condition that the at least two sub-tasks are executed, obtaining a processing result of the business data according to the processing result corresponding to each piece of fragmented data. According to the technical scheme provided by the embodiment of the invention, the processing time of the business data can be shortened by splitting the business data and generating the subtasks respectively corresponding to the obtained fragmented data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a business data processing method, device, electronic device and storage medium. Background Art

[0002] In recent years, with the opening of logistics business, related business data has grown explosively, which requires the logistics settlement system to process massive business data when generating logistics bills at the beginning of the month.

[0003] In the process of implementing the present invention, the inventors discovered that the prior art has the following technical problems: the processing time of business data is too long. Summary of the invention

[0004] Embodiments of the present invention provide a business data processing method, device, electronic device and storage medium to shorten the processing time of business data.

[0005] According to one aspect of the present invention, a service data processing method is provided, which may include:

[0006] In response to the business data processing instruction, determining the business data to be processed and generating a processing task for processing the business data;

[0007] By executing the processing task, the business data is split into at least two pieces of shard data, and a subtask corresponding to each piece of shard data in the at least two pieces of shard data is generated;

[0008] For each of the at least two generated subtasks, by executing the subtask, the shard data corresponding to the subtask is processed to obtain a processing result of the shard data;

[0009] When at least two subtasks are completed, the processing result of the business data is obtained according to the processing results corresponding to each shard data.

[0010] According to another aspect of the present invention, a service data processing device is provided, which may include:

[0011] A processing task generation module, used to respond to the business data processing instruction, determine the business data to be processed, and generate a processing task for processing the business data;

[0012] A subtask generation module is used to split the business data into at least two pieces of shard data by executing the processing task, and generate a subtask corresponding to each piece of shard data in the at least two pieces of shard data;

[0013] A shard data processing module is used to process the shard data corresponding to each of the at least two generated subtasks by executing the subtask to obtain a processing result of the shard data;

[0014] The processing result obtaining module is used to obtain the processing result of the business data according to the processing results corresponding to each shard data when at least two subtasks are executed.

[0015] According to another aspect of the present invention, there is provided an electronic device, which may include:

[0016] at least one processor; and

[0017] a memory communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor implements the business data processing method provided by any embodiment of the present invention when executing the computer program.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer instructions are stored. The computer instructions are used to enable a processor to implement the business data processing method provided by any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention determines the business data to be processed by responding to the business data processing instruction, and generates a processing task for processing the business data; by executing the processing task, the business data is split into at least two shard data, and a subtask corresponding to each of the at least two shard data is generated; for each of the at least two subtasks that have been generated, the shard data corresponding to the subtask is processed by executing the subtask to obtain the processing result of the shard data; when at least two subtasks are executed, the processing result of the business data is obtained according to the processing results corresponding to each shard data. The above technical solution, by splitting the business data and generating subtasks corresponding to each of the obtained shard data, since the amount of business data contained in each shard data is less than the amount of all business data, the processing time spent on completing the processing of the business data by executing the subtask to process the shard data is less than the processing time spent on centrally processing the business data, thereby achieving the effect of shortening the processing time of the business data.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a business data processing method provided according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of business data splitting in a business data processing method provided according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of a master-sub-operator task model in a business data processing method provided according to an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of a main-subtask notification mechanism in a business data processing method provided according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of an anti-duplicate example in a business data processing method provided according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of an example of data integrity in a business data processing method provided according to an embodiment of the present invention;

[0029] Figure 7 is a flowchart of another business data processing method provided by an embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of an operator push-down mode in another business data processing method provided according to an embodiment of the present invention;

[0031] Fig. 9 is a flowchart of another business data processing method provided by an embodiment of the present invention;

[0032] Fig.10 is a schematic diagram of big data scoring processing logic in another business data processing method provided according to an embodiment of the present invention;

[0033] Fig.11 is a flowchart of an optional example of another business data processing method provided by an embodiment of the present invention;

[0034] Fig.12is a structural block diagram of a business data processing device provided according to an embodiment of the present invention;

[0035] Fig.13 It is a structural schematic diagram of an electronic device for implementing the business data processing method of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Figure 1 1 is a flow chart of a business data processing method provided in an embodiment of the present invention. This embodiment is applicable to business data processing, and is particularly applicable to processing massive amounts of business data. The method can be executed by a business data processing device provided in an embodiment of the present invention, which can be implemented in software and / or hardware, and can be integrated in an electronic device, which can be various user terminals or servers.

[0039] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:

[0040] S110 . In response to a business data processing instruction, determine business data to be processed, and generate a processing task for processing the business data.

[0041] Among them, the business data processing instruction can be understood as an instruction for instructing the processing of business data. Business data can be understood as data that has processing needs at the business level. Combined with the application scenarios that may be involved in the embodiments of the present invention, it can be the sales amount of a merchant, the operating cost of a store, or the logistics cost of a certain area, etc., which are not specifically limited here. In actual applications, optionally, the amount of business data can be one, two or more, and in particular, it can be massive, which is related to the actual situation and is not specifically limited here. A processing task can be understood as a task for processing business data. In response to the business data processing instruction, the business data is determined and a processing task is generated.

[0042] S120. Split the business data into at least two shard data by executing the processing task, and generate subtasks corresponding to each of the at least two shard data.

[0043] Among them, when the amount of business data is multiple, especially in the case of massive data, if these business data are processed centrally, that is, these business data are processed as a whole, it is easy to have a long processing time. On this basis, in order to shorten the processing time of business data, the above-generated processing tasks can be specifically used for sharding business data.

[0044] By executing the processing task, the business data is split into at least two shard data. On this basis, for each of the at least two shard data, the amount of business data contained in the shard data can be one, two or more pieces. This is related to the specific splitting situation and is not specifically limited here. Exemplarily, the business data is split into at least two logical shards, and the business data on each of the at least two logical shards is used as one shard data, so that at least two shard data can be obtained. In combination with the application scenarios that may be involved in the embodiments of the present invention, for example, see Figure 2 , process the business data to obtain the data to be settled, and then send the data to be settled to the settlement pool through MQ. On this basis, the settlement pool evenly splits the data to be settled into different logical shards, for example, storing it in 2 according to preset rules. N logical shards, and stores the resulting shard data in a MySQL database cluster, where N is a positive integer.

[0045] For each of the at least two shard data, a subtask corresponding to the shard data is generated, that is, a subtask for processing the shard data is generated, thereby obtaining at least two subtasks, and the number of subtasks is consistent with the number of shard data. In other words, the number of subtasks can be expanded by expanding the number of shard data, thereby combining with subsequent steps to further shorten the processing time of business data. Compared with the term subtask, the above-mentioned processing task can also be called a main task.

[0046] S130. For each of the at least two generated subtasks, the slice data corresponding to the subtask is processed by executing the subtask to obtain a processing result of the slice data.

[0047] Among them, for each of the at least two subtasks, by executing the subtask, the shard data corresponding to the subtask is processed, thereby obtaining the processing result of the shard data corresponding to the subtask. In practical applications, optionally, in order to shorten the processing time of business data, these subtasks can be executed in parallel; more optionally, the processing tasks and subtasks can be executed through task middleware.

[0048] S140. When at least two subtasks are executed, the processing result of the business data is obtained according to the processing results corresponding to each shard data.

[0049] Among them, when at least two subtasks are executed and completed, the processing results of the shard data corresponding to the at least two subtasks can be obtained, and then the processing results of the business data can be obtained based on the processing results of these shard data, thereby completing the processing process of the business data.

[0050] On this basis, for each of the at least two subtasks, in actual applications, the subtask may be executed successfully or unsuccessfully. Therefore, the processing result of the shard data corresponding to the subtask can be used to characterize whether the subtask is executed successfully, and can also be further used to characterize the result data obtained after processing the shard data if the execution is successful. Correspondingly, the processing result of the business data can be used to characterize whether the processing task is executed successfully, and can also be further used to characterize the result data of the business data if the execution is successful.

[0051] It should be noted that, compared with the technical solution in which the generated processing tasks are specifically used to centrally process business data, and thus the business data is centrally processed by executing processing tasks, the embodiment of the present invention generates sub-tasks corresponding to each piece of shard data based on the processing tasks. Since the amount of business data contained in each piece of shard data is less than the amount of all business data, the processing time spent on processing the shard data by executing sub-tasks to complete the processing of the business data is less than the processing time spent on centrally processing the business data, thereby achieving the effect of shortening the processing time of the business data.

[0052] Or, to explain from another perspective, the embodiments of the present invention push down operators, that is, push down the calculation process completed by the processing task to the completion of the subtask, and change the computing power from 1 (i.e., the computing power represented by the processing task) to at least two (i.e., the computing power represented by at least two subtasks), thereby shortening the processing time.

[0053] In addition, the embodiments of the present invention can shorten the processing time without adding hardware such as application servers and / or database servers, thereby solving the problem of high hardware cost investment and waste of resources when the amount of business data is not high due to shortening the processing time by adding hardware.

[0054] The technical solution of the embodiment of the present invention determines the business data to be processed by responding to the business data processing instruction, and generates a processing task for processing the business data; by executing the processing task, the business data is split into at least two shard data, and a subtask corresponding to each of the at least two shard data is generated; for each of the at least two subtasks that have been generated, the shard data corresponding to the subtask is processed by executing the subtask to obtain the processing result of the shard data; when at least two subtasks are executed, the processing result of the business data is obtained according to the processing results corresponding to each shard data. The above technical solution, by splitting the business data and generating subtasks corresponding to each of the obtained shard data, since the amount of business data contained in each shard data is less than the amount of all business data, the processing time spent on completing the processing of the business data by executing the subtask to process the shard data is less than the processing time spent on centrally processing the business data, thereby achieving the effect of shortening the processing time of the business data.

[0055] An optional technical solution, the above-mentioned business data processing method further includes:

[0056] Scan the task status of the processing task;

[0057] Splitting the business data into at least two pieces of shard data, and generating subtasks corresponding to each piece of shard data in the at least two pieces of shard data, including:

[0058] When the task state is task initialization, the business data is split into at least two shard data, and a subtask corresponding to each of the at least two shard data is generated;

[0059] After generating the subtasks corresponding to the respective pieces of shard data in the at least two pieces of shard data, the business data processing method further includes:

[0060] Update the task status from task initialization to task execution;

[0061] When at least two subtasks are completed, the processing results of the business data are obtained according to the processing results corresponding to each shard data, including:

[0062] When at least two subtasks are completed, the task status is updated from task execution in progress to subtask execution completed;

[0063] When the task status is that the subtask is completed, the processing result of the business data is obtained according to the processing results corresponding to each shard data;

[0064] After obtaining the processing result of the business data, the business data processing method further includes:

[0065] Update the task status from subtask execution completed to task completed.

[0066] In other words, during the execution of the processing task, the specific execution content can be determined by scanning the task status of the processing task, thereby ensuring the integrity of the execution of each subtask. In practical applications, the scanning task status described above can be real-time or scheduled scanning, which is not specifically limited here.

[0067] On this basis, in order to better understand the above technical solution, the following is an exemplary explanation of the specific example of the master operator task model (Try-Confirm-Cancel). Figure 3 :

[0068] 1. The task middleware scans the main task in the "task initialization" state, splits it into subtasks by processing the business data, and updates the task status of the main task to the "task executing" state.

[0069] 2. After all subtasks are executed, the task status will be updated to "Subtask Execution Completed" status.

[0070] 3. The task middleware scans the "subtask execution completed" status, and the main task proceeds to the next step, such as asynchronously notifying other systems. If successful, the task status is updated to the "task completed" status; if failed, the task status is updated to the "task failed" status, and the corresponding interface is called back to notify the business data processing failure, and the alarm notification is sent to the corresponding R&D client.

[0071] Another optional technical solution is that after generating the subtasks corresponding to each of the at least two pieces of shard data, the business data processing method further includes:

[0072] Record the total number of at least two subtasks;

[0073] After processing each of the at least two subtasks that have been generated, the business data processing method further includes:

[0074] When the subtasks are completed, counting is performed by a counter to obtain a counting result, wherein the counting result is used to represent the number of subtasks that have been completed among the at least two subtasks;

[0075] When the counting result is equal to the total number, it is determined that at least two subtasks are completed.

[0076] The above technical solution records the total number and the count of the counter, and then compares the total number with the calculation result of the calculator, thereby accurately and timely determining the completion status of at least two subtasks.

[0077] On this basis, in order to better understand the above technical solution, the following is an exemplary explanation of the specific example of the master-subtask notification mechanism. Figure 4 When the main task generates subtasks, the total number of generated subtasks is recorded in the main task. On this basis, every time a subtask is completed, the Redis server is provided with a counting service (i.e., a counter) +1, that is, the number of subtasks that have been completed is recorded in Redis and compared with the "total number" in the main task. If the two are equal, it means that all subtasks have been completed. At this time, the task status of the main task can be updated to "subtask execution completed", thereby ensuring the task status consistency and data consistency of the main and subtasks.

[0078] Another optional technical solution is to generate a subtask corresponding to each of the at least two pieces of shard data, including:

[0079] Add a target lock for the processing task, and if the addition is successful, generate a subtask corresponding to each of the at least two shard data;

[0080] The above-mentioned business data processing method further includes:

[0081] When at least two subtasks are generated successfully, the target lock added to the processing task is released.

[0082] The above technical solution can ensure that the processing task does not repeatedly generate subtasks.

[0083] On this basis, in order to better understand the above technical solution, the following is an exemplary explanation in conjunction with the specific example of Prevent-Repeat. Figure 5 When generating subtasks based on the main task, a Redis lock is added to the main task, and the Redis lock is released after the subtask is successfully generated, thereby ensuring that the main task does not repeatedly generate subtasks. Of course, the above-mentioned Redis lock is only a specific example of a target lock, and other locks that can play the same role can also be used in actual applications, which are not specifically limited here.

[0084] Another optional technical solution, the above business data processing method further includes:

[0085] In the event that a subtask fails to execute, the number of times the subtask failed to execute is recorded;

[0086] When the number of execution failures reaches a preset threshold, an alarm notification indicating that the subtask execution failed is sent to the client through the alarm service.

[0087] The above technical solution can ensure the complete execution of subtasks, thereby ensuring data integrity, and will not affect the processing efficiency of business data due to the continuous failure of subtask execution.

[0088] On this basis, in order to better understand the above technical solution, the following is an exemplary explanation in combination with the specific example of data integrity. Figure 6 When the number of subtask execution failures reaches the preset threshold, the corresponding R&D department will be notified through the alarm service, so that the subtask can be completely executed through manual processing, thereby ensuring data integrity and preventing the processing efficiency of business data from being affected by the continuous failure of subtask execution.

[0089] Figure 7It is a flowchart of another business data processing method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, by executing a processing task, the business data is split into at least two pieces of data, including: by executing a processing task, the business information of the business data is obtained, and the business data is classified according to the business information of the business data to obtain at least one group of category data; for each group of category data in at least one group of category data, the category data is split into at least two pieces of data. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0090] See also Figure 7 The method of this embodiment may specifically include the following steps:

[0091] S210 . In response to a business data processing instruction, determine business data to be processed, and generate a processing task for processing the business data.

[0092] S220: Obtain business information of the business data by executing the processing task, and classify the business data according to the business information of the business data to obtain at least one set of category data.

[0093] Among them, the business information can be used to characterize the information of the business data at the business level. In combination with the application scenarios that may be involved in the embodiments of the present invention, it can be, for example, the place where the business occurs or the account settlement entity, etc., which is not specifically limited here. It should be noted that, as mentioned above, the amount of business data can be one, two or more pieces, and the business information of different business data in these business data may be the same or different. For example, taking the place where the business occurs as an example, the place where some of the business data occurs in these business data may be in South China and the place where some of the business data occurs may be in North China. In addition, at least one of the above-mentioned business data can be stored in a data table, that is, these business data are presented through a data table. At this time, the business information can be represented as one or more fields in the data table.

[0094] The business data is classified according to the business information of the business data, for example, business data with the same or similar business information are classified into one category, thereby obtaining at least one group of category data. On this basis, for each group of category data in at least one group of category data, the amount of business data contained in the group of category data may be one, two or more pieces, which is related to the specific classification situation and is not specifically limited here.

[0095] S230. For each group of category data in at least one group of category data, split the category data into at least two pieces of shard data, and generate subtasks corresponding to each piece of shard data in the at least two pieces of shard data.

[0096] Among them, the shard data splitting operation and the sub-task generation operation are performed on the basis of category data, that is, for each group of category data in at least one group of category data, the category data is split into at least two shard data, and sub-tasks corresponding to each shard data in the at least two shard data are generated.

[0097] S240: For each of the at least two generated subtasks, the slice data corresponding to the subtask is processed by executing the subtask to obtain a processing result of the slice data.

[0098] S250: When at least two subtasks are completed, the processing result of the business data is obtained according to the processing results corresponding to each shard data.

[0099] The technical solution of the embodiment of the present invention classifies business data through business information, and then performs the splitting operation of the shard data and the generation operation of subtasks based on each group of category data in at least one group of category data obtained. Compared with directly splitting the business data, it realizes further detailed splitting of the business data by applying the business information, thereby reducing the amount of business data that needs to be processed by each subtask, thereby further shortening the processing time of the business data.

[0100] An optional technical solution is to split the category data into at least two shard data, including:

[0101] Process the category data to obtain the sharding information of the category data;

[0102] Based on the sharding information of the category data, the category data is split into at least two sharding data;

[0103] Process the shard data corresponding to the subtask, including:

[0104] Generate query conditions based on the business information and shard information of the shard data corresponding to the subtask;

[0105] For a database used to store business data, a query is performed in the database based on query conditions to obtain shard data, and the shard data is processed in the database.

[0106] Among them, the category data is processed to obtain the sharding information of the category data, and the sharding information can be used to characterize the information of the category data in terms of sharding. Exemplarily, as described above, the amount of business data contained in the category data can be one, two or more, then the sharding information can be used to characterize the logical sharding to which each of these business data belongs. In practical applications, optionally, the category data can be hashed to obtain the hash value of each business data, and then these hash values ​​can be used as the sharding information of the category data; optionally, after obtaining the sharding information, for the data table storing the business data, the sharding information can be stored under the corresponding field in the data table, so that the sharding information of the category data can be obtained directly through the data table later.

[0107] The category data is split based on the shard information, for example, the business data with the same or similar shard information in one or more business data contained in the category data are classified into one category, thereby obtaining at least two pieces of shard data. Furthermore, for each of the at least two pieces of shard data, in the process of processing the shard data based on the subtask corresponding to the shard data, a query condition can be generated based on the business information of the shard data and the shard information, and then a query is performed in a database storing the business data based on the query condition, thereby querying the shard data from the database, and then processing the shard data in the database, thereby completing the processing of the shard data.

[0108] The above technical solution searches the database together with the business information and the shard information, and the query result (i.e., the shard data) obtained thereby contains a relatively limited amount of business data, which reduces the query pressure on the database compared to directly searching for all business data in the database; moreover, the subsequent processing of such shard data in the database reduces the processing pressure on the database compared to directly processing all business data in the database. The above two aspects cooperate with each other, thereby reducing the pressure on the database during the execution of the subtask.

[0109] On this basis, in order to better understand the above technical solution, the following is an exemplary explanation in combination with the operator push-down mode (Task-Split-Calculate). Figure 8 The data to be settled in the pool to be settled is obtained by processing the business data. The summary condition configuration can be used to characterize how to classify the data to be settled. Therefore, the classification process of the data to be settled can be performed based on the summary condition configuration and the business information of the data to be settled, thereby obtaining N groups of category data, where N is a positive integer.

[0110] For each of the N groups of category data, 64 subtasks are generated based on the category data, thus obtaining 64*N subtasks, and then the database is updated based on the 64*N subtasks. In this way, the business data processing process can be realized based on 64*N subtasks (i.e. 64*N computing power), thereby achieving the effect of shortening the processing time of business data.

[0111] Fig. 9 It is a flowchart of another business data processing method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, generating a processing task for processing business data may include: determining at least one shard object and the business object to which the business data belongs, wherein the object data of each shard object in at least one shard object needs to be processed in shards; matching the business object with at least one shard object, and generating a processing task for processing the business data based on the matching result; splitting the business data into at least two shard data by executing the processing task, which may include: when the processing task is used for shard processing of business data, splitting the business data into at least two shard data by executing the processing task. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0112] See also Fig. 9 The method of this embodiment may specifically include the following steps:

[0113] S310. In response to a business data processing instruction, determine at least one shard object, business data to be processed, and a business object to which the business data belongs, wherein object data of each shard object in the at least one shard object needs to be processed in shards.

[0114] Among them, the shard object can be understood as an object whose object data needs to be sharded, and the number of shard objects can be one, two or more, which is not specifically limited here. Under normal circumstances, the amount of object data of the shard object is usually massive. The business object can be understood as the object to which the business data belongs, which may be a shard object in at least one shard object, or it may not belong to any shard object, which depends on the actual situation. In actual applications, the essence of the above-mentioned shard objects and business objects can be merchants, brands or regions, etc., which can be set according to actual needs and are not specifically limited here.

[0115] S320: Match the business object with at least one shard object, and generate a processing task for processing the business data according to the matching result.

[0116] The business object is matched with at least one shard object to determine whether the business object belongs to at least one shard object, and then a processing task for processing the business data can be generated according to the obtained matching result. Exemplarily, if the business object belongs to at least one shard object, it means that the business data needs to be processed in shards, so a processing task for processing the business data in shards can be generated; otherwise, it means that the business data does not need to be processed in shards, so a processing task for centrally processing the business data can be generated.

[0117] S330: When the processing task is used for sharding business data, the business data is split into at least two shards by executing the processing task, and subtasks corresponding to each of the at least two shards are generated.

[0118] In which, when the processing task is specifically used for processing business data in slices, the business data can be split and subtasks can be generated by executing the processing task, thereby realizing the processing process of the business data.

[0119] On this basis, optionally, when the processing task is used to centrally process business data, the business data is processed by executing the processing task to obtain the processing result of the business data. In other words, at this time, there is no need to split the business data and generate subtasks, and all business data can be directly processed.

[0120] S340: For each of the at least two generated subtasks, the slice data corresponding to the subtask is processed by executing the subtask to obtain a processing result of the slice data.

[0121] S350: When at least two subtasks are completed, the processing result of the business data is obtained according to the processing results corresponding to each shard data.

[0122] The technical solution of the embodiment of the present invention determines a processing method that is adapted to the business data, specifically, the data volume of the business data, by matching the business object with at least one shard object, thereby generating a processing task that is adapted to the processing method to process the business data, thereby ensuring the processing effect of the business data.

[0123] An optional technical solution, the above-mentioned business data processing method further includes:

[0124] For each application object of at least one application object, obtaining a first processing indicator of first historical data of the application object, wherein the at least one application object includes a business object;

[0125] Inputting the first processing indicator into the adjusted shard object determination model to obtain an actual output result of the shard object determination model;

[0126] At least one shard object in at least one application object is determined according to actual output results corresponding to each application object.

[0127] The application object can be understood as an object that may be involved in the actual application, and its essence is the same as the business object, such as a merchant, a brand or a region, etc., which is not specifically limited here. The number of application objects can be one, two or more, and the business object comes from these application objects.

[0128] On this basis, for each application object in at least one application object, the first historical data can be understood as the business-related data generated by the application object at a third historical time. Usually, the first historical data and the business data are of the same nature, for example, both can be sales quotas, transportation costs, or logistics costs, etc.; of course, the natures of the two can also be different, and no specific limitation is made here. The first processing index can be understood as an index obtained by processing the first historical data. In practical applications, it can optionally include at least one of an index before operator calculation, an index during operator calculation, and an index after operator calculation, wherein the index before operator calculation may include data volume, the index during operator calculation may include database pressure and / or index hit rate, and the index after operator calculation may include at least one of task execution time (i.e., calculation duration), task failure reason (i.e., calculation failure reason), and calculation mode (e.g., centralized or sharded).

[0129] The shard object determination model can be understood as a model for determining the shard object, which can be implemented by an algorithm or a machine learning model, etc., and is not specifically limited here. The first processing indicator is input into the shard object determination model to obtain the actual output result of the shard object determination model, and the actual output result can be represented in a variety of ways, such as 0 to indicate that the application object corresponding to the first processing indicator is not a shard object, and 1 to indicate that the application object is a shard object; for example, the possibility of the application object being a shard object is represented by a score, and the shard object determination model at this time can also be called a scoring model; and so on.

[0130] Furthermore, according to the actual output results respectively corresponding to each application object, at least one slice object is determined from at least one application object, thereby achieving accurate determination of at least one slice object.

[0131] In order to better understand the above technical solution, the following is an illustrative example of the big data scoring processing logic. Fig.10, here, taking the above-mentioned object as a merchant as an example, the first processing index of at least one application merchant is synchronized to the data factory for processing, and the obtained processing results are input into the scoring model for scoring, and then at least one shard merchant obtained according to the actual output results of the scoring model is input into the database. Business personnel can use the page to create a bill summary task for summarizing the bills of business merchants to write business merchants into the database. On this basis, the task middleware matches the business merchant and at least one shard merchant, and generates a master operator task (i.e., a processing task for sharding business data) if the match is successful; otherwise, a normal operator task (i.e., a processing task for centrally processing business data) is generated. In the above example, the scoring model is obtained by analyzing big data, and the shard merchants that need to be pushed down by the operator are obtained through the scoring model.

[0132] On this basis, optionally, the shard object determination model is adjusted through the following steps:

[0133] For each application object of the at least one application object, obtaining an expected output result of the application object corresponding to a first historical time and a second processing indicator of second historical data at a second historical time, wherein the first historical time is later than the second historical time;

[0134] The expected output result and the second processing index are used as a group of adjustment samples, and the model parameters of the original determination model are adjusted according to at least one group of adjustment samples corresponding to each application object to obtain the slice object determination model.

[0135] Among them, the second historical data can be understood as the business-related data generated by the application object at the second historical time. Usually, the second historical time is earlier than the third historical time. It should be noted that the second historical data is essentially the same as the first historical data, except that the former is the historical data involved in the model adjustment process, and the latter is the historical data involved in the model application process. The second processing indicator is similar to the first processing indicator and will not be repeated here. On this basis, assuming that the historical data generated by the application object at the first historical time is the third historical data, then the expected output result can be used to characterize the expected processing method for the third historical data, which can be determined based on the data volume of the third historical data. The first historical time is later than the second historical time, and the order of the third historical time and the first historical time is uncertain.

[0136] The expected output result and the second processing index are used as a group of adjustment samples, and the model parameters of the original determination model are adjusted according to at least one group of adjustment samples corresponding to each application object, so as to obtain a shard object determination model. In practical applications, the above model parameters can be parameters in a machine learning model, or weights or scoring items in an algorithm, etc., which are not specifically limited here. The above shard object determination model can be considered to be obtained based on big data adjustment, so it can also be called a big data model.

[0137] In order to understand the above-mentioned model adjustment process and model application process more vividly, the following is an exemplary explanation of it with reference to specific examples. For example, the second historical time is the whole year of 2021, and the first historical time is the second half of 2022, that is, the model adjustment process is realized through the second processing index for the whole year of 2021 and the expected output result for the second half of 2022; further, after the model adjustment is completed, assuming that the current time is January 2023 and the third historical time is the whole year of 2022, then the first processing index for the whole year of 2022 can be input into the shard object determination model, thereby predicting the application object that belongs to the shard object in the second half of 2023 among at least one application object.

[0138] In order to better understand the above-mentioned technical solutions as a whole, the following is an exemplary description of the above-mentioned technical solutions in combination with specific examples. Fig.10 Based on the reference Fig.11 , the calculation process on the left is realized by executing ordinary operator tasks, and the calculation process on the right is realized by executing main operator tasks.

[0139] Specifically, for ordinary operator tasks, determine whether it is necessary to classify and summarize the data to be settled in the settlement pool based on the task data, and if classification and summary are required, regenerate the settlement order number, and then update the data to be settled in batches, and generate a callback function based on the regenerated settlement order number summary; if classification and summary are not required, directly update the data to be settled and create a callback function.

[0140] For the main and sub-operator tasks, the data to be settled is classified into N parts according to the classification conditions, and N settlement orders are generated according to the N types obtained, and each group of category data is split into 64 pieces of shard data, thereby generating 64*N subtasks, and each subtask processes the corresponding piece of shard data. After all subtasks are completed, the task status is updated to "subtask execution completed", and after the scanning task status is "subtask execution completed", the settlement order number is summarized according to the subtask information, and the reverse function is generated, and the task status is updated to "task completed" after completion.

[0141] Exemplarily, the business data takes financial data as an example, the various objects mentioned above take merchants as an example, and the processing mentioned above takes aggregation (i.e., aggregation processing) as an example. The specific implementation of the solution includes the following contents:

[0142] Step 1: For each application merchant among at least one application merchant, obtain a first summary indicator of the first historical data of the application merchant; input the first summary indicator into the adjusted shard merchant determination model to obtain the actual output result of the shard merchant determination model; and determine at least one shard merchant among at least one application merchant based on the actual output results corresponding to each application merchant.

[0143] Exemplarily, the above-mentioned application merchants may be merchants that may be involved in the process of generating settlement orders by the logistics settlement system. The above-mentioned first historical data may be the sales amount generated by the application merchant at a third historical time. The first summary indicator may include at least one of the pre-operator calculation indicator, the operator calculation indicator, and the post-operator calculation indicator obtained after summarizing the first historical data. Here, taking the shard merchant determination model as a scoring model as an example, the first summary indicator is input into the scoring model for scoring, thereby obtaining at least one shard merchant among the application merchants that needs to be sharded and summarized, similarly. Fig.10 shown.

[0144] Step 2: In response to the financial data aggregation instruction, determine the financial data to be aggregated and the business merchants to which the financial data belongs, wherein the business merchants belong to at least one application merchant.

[0145] Exemplarily, the above-mentioned financial data may be the sales amount of a business merchant.

[0146] Step 3: Match the business merchant with at least one shard merchant, and generate a summary task for summarizing financial data based on the matching result.

[0147] Exemplarily, the task middleware matches the business merchant with at least one shard merchant, that is, determines whether at least one shard merchant includes the business merchant, thereby generating a corresponding aggregation task, that is, an aggregation task for centrally aggregating financial data (i.e., a common operator task) or an aggregation task for sharding financial data (i.e., a master operator task), similar to Fig.10 On this basis, optionally, after the summary task is generated, the task status of the summary task can be set to task initialization.

[0148] Step 4: Scan the task status of the summary task.

[0149] Exemplarily, this step can be performed in real time or periodically.

[0150] Step 5: When the summary task is used to centrally summarize financial data and the task status is task initialization, the financial data is summarized by executing the summary task to obtain a summary result of the financial data.

[0151] Step 6: When the summary task is used to summarize financial data in segments and the task status is task initialization, execute the following steps 6-1 to 6-8 to obtain the summary results of the financial data:

[0152] Step 6-1: Obtain business information of financial data by executing the aggregation task, and classify the financial data according to the business information of the financial data to obtain at least one set of category data.

[0153] For example, the above business information may be the place where the business takes place or the subject of account settlement, etc., which is not specifically limited here. Taking the place where the business takes place as an example, for a large amount of financial data, usually, the place where the business of some financial data takes place may be in South China and the place where the business of some financial data takes place may be in North China. Therefore, these financial data can be classified according to the place where the business takes place, thereby obtaining at least one group of category data. Here, the number of groups of category data is represented as N. The financial data under a group of category data can be used to summarize a settlement statement, similar to Figure 8 shown.

[0154] Step 6-2: for each group of category data in at least one group of category data, process the category data, and based on the obtained sharding information of the category data, split the category data into at least two sharding data, and generate subtasks corresponding to each sharding data in the at least two sharding data.

[0155] Exemplarily, the shard information can be used to characterize the logical shard to which each piece of financial data in the category data belongs. Therefore, according to the shard information, the category data can be split into at least two pieces of shard data, and each piece of financial data in each of the at least two pieces of shard data belongs to the same logical shard. For the sake of visualization, 64 logical shards can be preset here, so that 64 pieces of shard data can be obtained.

[0156] After splitting each group of category data, 64*N pieces of shard data can be obtained. Further, corresponding subtasks are generated for each piece of shard data, thereby obtaining 64*N subtasks.

[0157] Step 6-3: After processing each set of category data, record the total number of all subtasks that have been generated.

[0158] Exemplarily, the total number may be 64*N in the above example.

[0159] Step 6-4: Update the task status from task initialization to task execution.

[0160] Step 6-5: For each subtask among all subtasks, by executing the subtask, based on the business information and shard information of the shard data corresponding to the subtask, generate query conditions, and based on the query conditions, query in the database used to store financial data to obtain the shard data, and summarize the shard data in the database to obtain the summary results of the shard data.

[0161] In other words, 64*N subtasks aggregate the shard data on their respective logical shards.

[0162] Step 6-6: When any subtask among all subtasks is completed, a counter is used to count to obtain a counting result, and the counting result is used to indicate the number of subtasks among all subtasks that have been completed.

[0163] For example, each time a subtask is completed, the counter is added with 1, so that it can be determined whether all subtasks are completed according to the calculated result and the total number, similar to Figure 4 shown.

[0164] Step 6-7: When the count result is equal to the total number, that is, when all subtasks are completed, the task status is updated from task executing to subtask execution completed.

[0165] Step 6-8: When the task status is that the subtask is completed, the summary result of the financial data is obtained according to the summary results corresponding to each shard data in each group of category data.

[0166] Exemplarily, based on the summary results corresponding to each shard data in the same category of data, a settlement statement corresponding to the category of data is obtained, and the summary results of the financial data can be understood as N settlement statements.

[0167] Step 7: When the summary result of the financial data is obtained, the task status is updated from subtask execution completed to task completed.

[0168] In the above example, through the cooperation of main and subtasks (i.e., summary tasks and subtasks) and operator pushdown (i.e., pushing the calculation process originally completed by the summary task down to the subtask to complete), the massive financial data is split into shards with limited data volume, thereby performing "divide and conquer" processing on the massive financial data, dispersing the summary tasks and database pressure by reducing them to small ones. This not only meets the timeliness requirements of the summary business, but also ensures the high availability and stability of the entire summary process, and controls the hardware cost.

[0169] Fig.12This is a structural block diagram of a business data processing device provided in an embodiment of the present invention. The device is used to execute the business data processing method provided in any of the above embodiments. The device and the business data processing method of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the business data processing device, please refer to the embodiments of the above business data processing method. Fig.12 The device may specifically include: a processing task generating module 410, a subtask generating module 420, a shard data processing module 430 and a processing result obtaining module 440.

[0170] The processing task generating module 410 is used to determine the business data to be processed in response to the business data processing instruction, and to generate a processing task for processing the business data;

[0171] The subtask generating module 420 is used to split the business data into at least two pieces of shard data by executing the processing task, and generate a subtask corresponding to each piece of shard data in the at least two pieces of shard data;

[0172] The shard data processing module 430 is used to process the shard data corresponding to each of the at least two generated subtasks by executing the subtask to obtain a processing result of the shard data;

[0173] The processing result obtaining module 440 is used to obtain the processing result of the business data according to the processing results corresponding to each piece of shard data when at least two subtasks are executed.

[0174] Optionally, the subtask generating module 420 may include:

[0175] A business data classification submodule is used to obtain business information of business data by executing a processing task, and classify the business data according to the business information of the business data to obtain at least one set of category data;

[0176] The category data splitting submodule is used to split the category data into at least two pieces of fragmented data for each group of category data in at least one group of category data.

[0177] On this basis, optional, category data splitting submodules may include:

[0178] A fragmentation information obtaining unit is used to process the category data to obtain fragmentation information of the category data;

[0179] A category data splitting unit, used for splitting the category data into at least two pieces of shard data based on shard information of the category data;

[0180] The fragment data processing module 430 may include:

[0181] A query condition generating unit, used to generate query conditions based on the business information and shard information of the shard data corresponding to the subtask;

[0182] The shard data processing unit is used to query the database for storing business data based on the query conditions, obtain the shard data, and process the shard data in the database.

[0183] Optionally, the processing task generating module 410 may include:

[0184] A business object determination submodule, used to determine at least one shard object and the business object to which the business data belongs, wherein the object data of each shard object in the at least one shard object needs to be processed in shards;

[0185] A processing task generation submodule, used to match a business object with at least one shard object, and generate a processing task for processing business data according to the matching result;

[0186] The subtask generation module 420 may include:

[0187] The business data splitting submodule is used to split the business data into at least two shard data by executing the processing task when the processing task is used to process the business data in shards.

[0188] On this basis, optionally, the above-mentioned business data processing device further includes:

[0189] A first processing indicator acquisition module, configured to acquire, for each application object of at least one application object, a first processing indicator of first historical data of the application object, wherein the at least one application object includes a business object;

[0190] An actual output result obtaining module, used for inputting the first processing index into the adjusted shard object determination model to obtain an actual output result of the shard object determination model;

[0191] The slicing object determining module is used to determine at least one slicing object in at least one application object according to the actual output results corresponding to each application object.

[0192] On this basis, optionally, the shard object determination model is obtained by adjusting the following modules:

[0193] A second processing indicator acquisition module, used to acquire, for each application object of at least one application object, a second processing indicator of an expected output result corresponding to a first historical time and second historical data of the application object at a second historical time, wherein the first historical time is later than the second historical time;

[0194] The shard object determination model obtaining module is used to use the expected output result and the second processing indicator as a group of adjustment samples to adjust the model parameters of the original determination model according to at least one group of adjustment samples corresponding to each application object to obtain the shard object determination model.

[0195] Alternatively, the above-mentioned business data processing device further includes:

[0196] The business data processing module is used to process the business data by executing the processing task to obtain the processing result of the business data.

[0197] Optionally, the above-mentioned business data processing device further includes:

[0198] A task status scanning module is used to scan the task status of the processing task;

[0199] Subtask generation module, specifically used for:

[0200] When the task state is task initialization, the business data is split into at least two shard data, and a subtask corresponding to each of the at least two shard data is generated;

[0201] The above-mentioned business data processing device further includes:

[0202] A first task status update module, configured to update the task status from task initialization to task execution after generating subtasks corresponding to each of the at least two shard data;

[0203] The processing result is obtained by the module, which is specifically used for:

[0204] When at least two subtasks are completed, the task status is updated from task execution in progress to subtask execution completed;

[0205] When the task status is that the subtask is completed, the processing result of the business data is obtained according to the processing results corresponding to each shard data;

[0206] The above-mentioned business data processing device further includes:

[0207] The second task status updating module is used to update the task status from subtask execution completion to task completion after obtaining the processing result of the business data.

[0208] Optionally, the above-mentioned business data processing device further includes:

[0209] A total quantity recording module, used for recording the total quantity of at least two subtasks after generating subtasks corresponding to each of the at least two pieces of shard data;

[0210] A counting result obtaining module is used to, after each of the at least two subtasks that have been generated, count the number of subtasks that have been executed by a counter to obtain a counting result when the subtask is executed, wherein the counting result is used to indicate the number of subtasks that have been executed in the at least two subtasks;

[0211] The subtask execution completion determination module is used to determine that at least two subtasks are executed completely when the counting result is equal to the total number.

[0212] Optionally, the subtask generating module 420 includes:

[0213] A subtask generating unit, used for adding a target lock for the processing task, and generating a subtask corresponding to each of the at least two pieces of shard data respectively if the adding is successful;

[0214] The above-mentioned business data processing device further includes:

[0215] The target lock releasing module is used to release the target lock added to the processing task when at least two subtasks are generated successfully.

[0216] Optionally, the above-mentioned business data processing device further includes:

[0217] The execution failure count recording module is used to record the execution failure count of the subtask when the subtask fails to execute;

[0218] The alarm notification sending module is used to send an alarm notification indicating the failure of subtask execution to the client through the alarm service when the number of execution failures reaches a preset threshold.

[0219] The business data processing device provided by the embodiment of the present invention determines the business data to be processed and generates a processing task for processing the business data by a processing task generation module in response to a business data processing instruction; executes the processing task by a subtask generation module to split the business data into at least two shard data and generates subtasks corresponding to each of the at least two shard data; processes the shard data corresponding to the subtask by executing the subtask for each of the at least two subtasks generated, and obtains the processing result of the shard data; obtains the processing result of the business data according to the processing results corresponding to each of the shard data by a processing result obtaining module when at least two subtasks are executed. The above device splits the business data and generates subtasks corresponding to each of the obtained shard data. Since the amount of business data contained in each shard data is less than the amount of all business data, the processing time spent on processing the shard data by executing the subtasks to complete the processing process of the business data is less than the processing time spent on centrally processing the business data, thereby achieving the effect of shortening the processing time of the business data.

[0220] The business data processing device provided in the embodiment of the present invention can execute the business data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0221] It is worth noting that in the embodiment of the above-mentioned business data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0222] Fig.13 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0223] like Fig.13As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0224] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a business data processing method.

[0226] In some embodiments, the business data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the business data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the business data processing method in any other appropriate manner (e.g., by means of firmware).

[0227] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0228] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0229] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0230] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0231] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0232] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0233] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0234] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A business data processing method, It is characterized in that include: In response to the business data processing instruction, determining the business data to be processed, and generating a processing task for processing the business data; By executing the processing task, the business data is split into at least two pieces of shard data, and a subtask corresponding to each piece of shard data in the at least two pieces of shard data is generated; For each of the at least two generated subtasks, by executing the subtask, the slice data corresponding to the subtask is processed to obtain a processing result of the slice data; When the at least two subtasks are executed completely, the processing result of the business data is obtained according to the processing results corresponding to each of the shard data.

2. The method according to claim 1, It is characterized in that The step of splitting the business data into at least two pieces of data by executing the processing task includes: By executing the processing task, the business information of the business data is obtained, and the business data is classified according to the business information of the business data to obtain at least one set of category data; For each group of category data in the at least one group of category data, the category data is split into at least two pieces of fragment data.

3. The method according to claim 2, It is characterized in that The step of splitting the category data into at least two pieces of shard data includes: Processing the category data to obtain segmentation information of the category data; Based on the sharding information of the category data, split the category data into at least two sharding data; The processing of the slice data corresponding to the subtask includes: Generate query conditions based on the business information and shard information of the shard data corresponding to the subtask; For a database for storing the business data, a query is performed in the database based on the query condition to obtain the shard data, and the shard data is processed in the database.

4. The method according to claim 1, It is characterized in that The generating of a processing task for processing the business data comprises: Determine at least one shard object and a business object to which the business data belongs, wherein object data of each shard object in the at least one shard object needs to be processed in shards; Matching the business object with the at least one shard object, and generating a processing task for processing the business data according to the matching result; The step of splitting the business data into at least two pieces of data by executing the processing task includes: In the case where the processing task is used for sharding the business data, the business data is split into at least two shards of data by executing the processing task.

5. The method according to claim 4, It is characterized in that Also includes: For each application object of at least one application object, obtaining a first processing indicator of first historical data of the application object, wherein the at least one application object includes the business object; Inputting the first processing indicator into the adjusted slice object determination model to obtain an actual output result of the slice object determination model; The at least one slice object in the at least one application object is determined according to the actual output results respectively corresponding to each application object.

6. The method according to claim 5, It is characterized in that The shard object determination model is adjusted by the following steps: For each application object of the at least one application object, obtaining an expected output result corresponding to a first historical time and a second processing indicator of second historical data of the application object at a second historical time, wherein the first historical time is later than the second historical time; The expected output result and the second processing index are used as a group of adjustment samples, and the model parameters of the original determination model are adjusted according to at least one group of adjustment samples corresponding to each application object to obtain the slice object determination model.

7. The method according to claim 4, It is characterized in that Also includes: In the case where the processing task is used to centrally process the business data, the business data is processed by executing the processing task to obtain a processing result of the business data.

8. The method according to claim 1, It is characterized in that Also includes: Scanning the task status of the processing task; The step of splitting the business data into at least two pieces of shard data and generating subtasks corresponding to each piece of shard data in the at least two pieces of shard data includes: When the task state is task initialization, split the business data into at least two pieces of shard data, and generate subtasks corresponding to each piece of shard data in the at least two pieces of shard data; After generating the subtasks corresponding to the respective pieces of shard data in the at least two pieces of shard data, the method further includes: Updating the task status from the task initialization to the task execution; When the at least two subtasks are executed and completed, obtaining the processing result of the business data according to the processing results corresponding to each of the shard data, includes: When the at least two subtasks are completed, updating the task status from task being executed to subtask execution completed; When the task status is that the subtask is completed, the processing result of the business data is obtained according to the processing results corresponding to each of the shard data; After obtaining the processing result of the business data, the method further includes: The task status is updated from the subtask execution completed to the task completed.

9. The method according to claim 1, It is characterized in that After the subtasks corresponding to the respective pieces of shard data of the at least two pieces of shard data are generated, the method further includes: Recording the total number of the at least two subtasks; After each of the at least two subtasks that have been generated, the method further includes: When the subtasks are completed, counting is performed by a counter to obtain a counting result, wherein the counting result is used to represent the number of subtasks that are completed among the at least two subtasks; When the counting result is equal to the total number, it is determined that the at least two subtasks are all completed.

10. The method according to claim 1, It is characterized in that The generating of subtasks corresponding to each of the at least two pieces of shard data includes: Adding a target lock for the processing task, and if the adding is successful, generating subtasks corresponding to each of the at least two pieces of shard data; The method further comprises: When the at least two subtasks are both generated successfully, the target lock added to the processing task is released.

11. The method according to claim 1, It is characterized in that Also includes: In the event that the subtask fails to be executed, the number of times the subtask fails to be executed is recorded; When the number of execution failures reaches a preset number threshold, an alarm notification indicating that the subtask execution has failed is sent to the client through an alarm service.

12. A business data processing device, It is characterized in that include: A processing task generation module, used to respond to the business data processing instruction, determine the business data to be processed, and generate a processing task for processing the business data; A subtask generation module, configured to split the business data into at least two pieces of shard data by executing the processing task, and generate a subtask corresponding to each piece of shard data in the at least two pieces of shard data; A shard data processing module is used to process the shard data corresponding to each of the at least two generated subtasks by executing the subtask to obtain a processing result of the shard data; The processing result obtaining module is used to obtain the processing result of the business data according to the processing results corresponding to each of the shard data when the at least two subtasks are executed.

13. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the business data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the business data processing method as described in any one of claims 1-11 when executed.