Task data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a task data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps of obtaining candidate business data matched with a target task in response to a processing instruction of the target task; under the condition that the data volume of the candidate service data is determined to be greater than a set data volume threshold value, grouping the candidate service data based on the target attribute information of the candidate service data to obtain at least two candidate service data groups; determining a target business analysis model matched with the target task in the big data calculation service, and performing analysis processing on each candidate business data group based on the target business analysis model; and respectively storing the processing result of each candidate service data group in a target data table. According to the scheme, the business data of different tasks can be quickly and accurately processed, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, apparatus, electronic device, and storage medium for processing task data. Background Art

[0002] In the enterprise resource planning (ERP) industry, a large amount of business data is usually involved. For example, sales data of various merchants or distributors, order data of different products, logistics data, or log data, etc.

[0003] At present, the processing of each business data is mainly carried out in the form of a single task, that is, a single thread. This processing method is slow and time-consuming.

[0004] How to process the business data of different tasks quickly and accurately and improve the user experience is a key issue studied in the industry. Summary of the Invention

[0005] The present invention provides a method, apparatus, electronic device, and storage medium for processing task data to process the business data of different tasks quickly and accurately and improve the user experience.

[0006] According to an aspect of the present invention, there is provided a method for processing task data, which is executed by a target client. The target client is communicatively connected to a big data computing service. The method includes:

[0007] Responding to a processing instruction of a target task, and obtaining candidate business data matching the target task;

[0008] In the case where it is determined that the data volume of the candidate business data is greater than a set data volume threshold, grouping the candidate business data based on the target attribute information of each candidate business data to obtain at least two candidate business data groups;

[0009] Determining a target business analysis model in the big data computing service that matches the target task, and respectively analyzing and processing each candidate business data group based on the target business analysis model;

[0010] Storing the processing results of each candidate business data group in a target data table respectively.

[0011] According to another aspect of the present invention, there is provided a device for processing task data, which is deployed on a target client for execution. The target client is communicatively connected to a big data computing service. The device includes:

[0012] A candidate service data acquisition module, configured to acquire candidate service data matching the target task in response to a processing instruction of the target task;

[0013] A grouping module, configured to, when determining that the data volume of the candidate service data is greater than a set data volume threshold, group the candidate service data based on the target attribute information of each piece of candidate service data to obtain at least two candidate service data groups;

[0014] A processing module, configured to determine a target service analysis model in the big data computing service that matches the target task, and respectively perform analysis and processing on each candidate service data group based on the target service analysis model;

[0015] A merging module, configured to respectively store the processing results of each candidate service data group in a target data table.

[0016] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the processing method of task data according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions, and when the computer instructions are executed by a processor, the processing method of task data according to any embodiment of the present invention is implemented.

[0021] According to another aspect of the present invention, there is provided a computer program product including a computer program, and when the computer program is executed by a processor, the processing method of task data according to any embodiment of the present invention is implemented.

[0022] In the technical solution of the embodiment of the present invention, the target client responds to the processing instruction of the target task to obtain candidate service data matching the target task; in the case where it is determined that the data volume of the candidate service data is greater than the set data volume threshold, the candidate service data is grouped based on the target attribute information of each candidate service data to obtain at least two candidate service data groups; the target service analysis model matching the target task in the big data computing service is determined, and each candidate service data group is analyzed and processed based on the target service analysis model; the processing results of each candidate service data group are respectively stored in the target data table, so that the service data of different tasks can be processed quickly and accurately, improving the user experience.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0025] Figure 1 is a flowchart of a method for processing task data according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flowchart of a method for processing task data according to Embodiment 2 of the present invention;

[0027] Figure 3 is a flowchart of a method for processing task data according to Embodiment 3 of the present invention;

[0028] Figure 4 is a schematic diagram of a task data processing flow according to Embodiment 3 of the present invention;

[0029] Figure 5 is a schematic structural diagram of a task data processing device according to Embodiment 4 of the present invention;

[0030] Figure 6 is a schematic structural diagram of an electronic device for implementing the task data processing method of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 is a flowchart of a method for processing task data according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of processing a large amount of task data. This method can be executed by a task data processing device, and the task data processing device can be implemented in the form of hardware and / or software. The task data processing device can be configured in an electronic device such as a computer, a server, or a tablet computer where a target client is deployed.

[0035] Before specifically introducing the specific steps of the method for processing task data, it should also be noted that the target client involved in this embodiment can be a cloud development platform, a desktop application, or a web service, etc., and this embodiment does not limit it.

[0036] Specifically, referring to Figure 1 , the method includes:

[0037] Step 110, in response to a processing instruction of a target task, obtain candidate service data matching the target task.

[0038] Among them, the target task can be a profit analysis task, a procurement task, an order tracking task, or an inventory surplus statistics task of a target commodity (such as clothes, magazines, or household items, etc., and this embodiment does not limit it), etc.

[0039] In this embodiment, if the target task is a target commodity profit analysis task, the candidate business data matching the target task may be: the purchase amount, purchase quantity, sales amount, sales quantity, logistics cost, labor cost, etc. of different merchants for the target commodity; if the target task is an order tracking task, the candidate business data matching the target task may be: the order placement time, delivery time, receipt time, whether to return or exchange goods, etc. of each order of different merchants or distributors. This embodiment does not limit it either.

[0040] Optionally, in this embodiment, after receiving the processing instruction of the target task, the target client can further obtain the candidate business data matching the target task; in this embodiment, obtaining the candidate business data matching the target task may include: determining the target database associated with the target task; determining the first business data corresponding to the task identifier of the target task from the target database; and filtering the first business data based on the preset time information to obtain the candidate business data.

[0041] Among them, the target database may be a relational database or a distributed database, etc., and the number may be one or more. This embodiment does not limit it; in this embodiment, different business data can be stored in different databases or in different regions of the same database; for example, the sales database can be stored in the first database, the logistics data can be stored in the first region of the second database, and the inventory data can be stored in the second region of the second database. This embodiment does not limit it either.

[0042] In this embodiment, the task identifier of the target task can uniquely identify the target task, and through the task identifier of the target task, the processing objective of the target task, the target tasks required to process the target task, etc. can be determined.

[0043] In this embodiment, the preset time information may be a time period such as the past three days, the past week, the past half month, or the past month. This embodiment does not limit it.

[0044] In an optional implementation manner of this embodiment, after receiving the processing instruction of the target task, the relevant business data of the target task can be further determined, and the target database in which each business data is stored can be determined; further, all the business data associated with the target task can be obtained from the target database. In this embodiment, it is referred to as the first business data; in this embodiment, the business identifiers of the business data in the target database can be compared with the task identifier of the target task to determine the first business data.

[0045] Further, the first service data can be filtered based on preset time information. For example, the first service data outside the preset time information can be filtered out to obtain candidate service data.

[0046] Step 120: When it is determined that the data volume of the candidate service data is greater than the set data volume threshold, group the candidate service data based on the target attribute information of each candidate service data to obtain at least two candidate service data groups.

[0047] Optionally, in this embodiment, after obtaining the candidate service data, the data volume of the candidate service data can be further determined. If it is determined that the data volume of the candidate service data is greater than the set data volume threshold, then the candidate service data can be further grouped based on the target attribute information of the candidate service data to obtain multiple candidate service data groups, such as 2, 4, 8, or 16, etc. This embodiment does not limit it.

[0048] Among them, the set data volume threshold can be the number threshold of the data volume, such as 100,000, 50,000, or 20,000, etc.; it can also be the file size threshold, such as 100 kB, 200 kB, or 10 mB, etc. It can also be other parameters that can be used to characterize the data volume size. This embodiment does not limit it.

[0049] Optionally, in this embodiment, determining the data volume of the candidate service data may include: obtaining the number of rows and / or columns of the candidate data table storing the candidate service data, and determining the data volume of the candidate service data based on the number of rows and / or columns; or, determining the file size of the candidate data table, and determining the data volume of the candidate service data based on the file size; or, inputting the candidate service data into a data volume calculation large model that has been pre-fine-tuned to obtain the data volume of the candidate service data.

[0050] In an optional implementation manner of this embodiment, after obtaining the candidate service data, the number of rows, columns, or the number of rows and columns of the data table (which can be one or multiple, and this embodiment does not limit it) storing the candidate service data can be further determined, and the data volume of the candidate service data can be further determined based on the number of rows, columns, or the number of rows and columns; further, if it is determined that the number of rows is greater than the set number of rows value, the number of columns is greater than the set number of columns value, or both the number of rows and columns are greater than the set value, then it can be determined that the data volume of the candidate service data is large, and processing the candidate service data based on a single thread will take a long time.

[0051] In another alternative implementation of this embodiment, after obtaining the candidate service data, the file size of the data table storing the candidate service data can be further determined. Further, the data volume of the candidate service data can be determined based on the file size. Further, if it is determined that the file size is greater than the set file size threshold, it can be determined that the data volume of the candidate service data is large, and processing the candidate service data based on a single thread will take a long time.

[0052] In another alternative implementation of this embodiment, after obtaining the candidate service data, the candidate service data can be further input into a data volume calculation large model that has been pre-fine-tuned to determine the data volume of the candidate service data, such as the number of candidate service data items. Further, if it is determined that the number of candidate service data items is greater than the set number threshold, it can be determined that the data volume of the candidate service data is large, and processing the candidate service data based on a single thread will take a long time. In this way, the data volume of the candidate service data can be quickly determined, providing a basis for subsequent processing of the candidate service data.

[0053] It should be noted that the data volume calculation large model involved in this embodiment can be obtained by fine-tuning the original large data model based on different service data and is a large model deployed in the data calculation service.

[0054] In an alternative implementation of this embodiment, when it is determined that the data volume of the candidate service data is greater than the set data volume threshold, the candidate service data can be further grouped (classified) based on the target attribute information of the candidate service data, thereby obtaining multiple candidate service data groups.

[0055] Among them, the target attribute information may include at least one of the following: name, relationship attribute, identification information, time attribute, status attribute, location attribute, and label attribute.

[0056] In an alternative implementation of this embodiment, the candidate service data can be grouped according to different merchants, thereby obtaining candidate service data groups based on different merchants.

[0057] It should be noted that in another alternative implementation of this embodiment, the number of groups of the candidate service data can also be comprehensively determined based on the data volume of the candidate service data and the resource volume (such as computing resources, storage resources, network resources, and memory resources) of the candidate service data to be processed. Exemplarily, if the data volume of the candidate service data is large but the corresponding computing resources are also large, the number of groups of the candidate service data can be appropriately increased; if the data volume of the candidate service data is large but the corresponding computing resources are small, the number of groups of the candidate service data can be appropriately decreased.

[0058] Step 130: Determine the target business analysis model in the big data computing service that matches the target task, and respectively analyze and process each of the candidate business data groups based on the target business analysis model.

[0059] Among them, business analysis models matching all tasks can be deployed in the data computing service communicating with the target client. For example, a profit analysis model, a purchase quantity calculation model, an order query model, or a logistics information statistics model, etc. are not limited in this embodiment.

[0060] Optionally, in this embodiment, after obtaining multiple candidate business data groups, the target business analysis model matching the target task can be further determined in the data computing service communicating with the target client. Further, the candidate services of each candidate business data group can be processed simultaneously through the target business analysis model, so as to obtain the task processing results matching each candidate business data group; Exemplarily, if the target task is a profit analysis task, then the profit analysis results for the first candidate business data group, the profit analysis results for the second candidate business data group, and the profit analysis results for the third candidate business data group can be obtained respectively.

[0061] Step 140: Store the processing results of each of the candidate business data groups in a target data table respectively.

[0062] Optionally, in this embodiment, after the target business analysis model finishes processing all candidate business data groups, the processing results can be stored in the target data table respectively.

[0063] Exemplarily, in the above example, the data tables of the profit analysis results of the first candidate business data group, the data tables of the profit analysis results of the second candidate business data group, and the profit analysis results of the third candidate business data group can be stored in the target data table respectively for subsequent consumption of the data in the target data table.

[0064] The technical solution of this embodiment is as follows: The target client responds to the processing instruction of the target task to obtain candidate business data matching the target task; when it is determined that the data volume of the candidate business data is greater than the set data volume threshold, each of the candidate business data is grouped based on the target attribute information of each of the candidate business data to obtain at least two candidate business data groups; determine the target business analysis model in the big data computing service that matches the target task, and respectively analyze and process each of the candidate business data groups based on the target business analysis model; store the processing results of each of the candidate business data groups in a target data table respectively, which can quickly and accurately process the business data of different tasks and improve the user experience.

[0065] Embodiment Two

[0066] Figure 2 It is a flowchart of a method for processing task data provided in Embodiment 2 of the present invention. This embodiment further refines the above technical solution, and the technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. As Figure 2 shown, the method includes:

[0067] Step 210: In response to a processing instruction of a target task, obtain candidate service data that matches the target task.

[0068] Step 220: Determine that the data volume of the candidate service data is greater than a set data volume threshold.

[0069] Step 230: Determine the processing objective of the target task, and determine the attribute information of the service data associated with the processing objective; determine the attribute information of the service data associated with the processing objective as the target attribute information; perform clustering on each candidate service data based on the target attribute information to obtain each candidate service data group.

[0070] Among them, the target attribute information includes at least one of the following: name, relationship attribute, identification information, time attribute, status attribute, location attribute, and label attribute.

[0071] Optionally, in this embodiment, when it is determined that the data volume of the candidate service data that matches the target task is greater than the set data volume threshold, the processing objective of the target task can be further determined. In this embodiment, the processing objective of the target task can be determined based on the task identifier of the target task. Exemplarily, the processing objective of the target task can be profit analysis, logistics analysis, inventory analysis, etc.

[0072] Furthermore, the attribute information of the service data associated with the processing objective can be determined; Exemplarily, if the target task is the profit analysis of different merchants regarding the target commodity, then the attribute information of the service data associated with the processing objective can be the information of different merchants; if the target task is the profit analysis of the same merchant regarding different commodities, then the attribute information of the service data associated with the processing objective can be the information of different commodities.

[0073] Furthermore, the attribute information of the service data associated with the processing objective can be determined as the target attribute information. Furthermore, clustering can be performed on each candidate service data based on the target attribute information, so as to obtain multiple candidate service data groups; Exemplarily, in the above example, different merchants can be grouped into different groups (for example, each group can contain a set number of merchants), so as to obtain multiple candidate service data groups.

[0074] Step 240: Determine the target business analysis model in the big data computing service that matches the target task, and respectively analyze and process each of the candidate business data groups based on the target business analysis model.

[0075] Optionally, in this embodiment, determining the target business analysis model in the big data computing service that matches the target task, and respectively analyzing and processing each of the candidate business data groups based on the target business analysis model may include: screening the business analysis models deployed in the big data service based on the processing objective of the target task and the attribute information of each business data to obtain the target business analysis module; creating a first virtual node; in response to the trigger instruction of the first virtual node, simultaneously processing each of the candidate business data groups based on the target business analysis module.

[0076] Optionally, in this embodiment, after dividing the candidate business data into multiple candidate business data groups, the target business analysis model may be further screened from the business analysis models deployed in the big data service based on the processing objective of the target task determined in the above steps and the attribute information of each business data; exemplarily, the processing objective of the target task and the attribute information of each business data may be compared one by one with the processing objective and the attribute information of each business analysis model to determine the target business analysis model.

[0077] Further, a first virtual node may be created in the target client. In this embodiment, the first virtual node is the upstream task of the processing tasks of different candidate business data groups. Correspondingly, the first virtual node may be used to control the simultaneous start of the processing tasks of different candidate business data groups.

[0078] Optionally, in this embodiment, after receiving the trigger instruction of the first virtual node, each of the candidate business data groups may be simultaneously processed based on the target business analysis module.

[0079] It should be noted that in this embodiment, the first virtual node may also be created before the target task is processed, and it may also be used to process other tasks, and it is not limited in this embodiment.

[0080] Step 250: Store the processing results of each of the candidate business data groups in the target data table.

[0081] Step 260: Create a second virtual node; automatically trigger the second virtual node after obtaining the processing result of a candidate business data group; when the number of trigger times of the second virtual node is consistent with the number of candidate business data groups, determine that the second virtual node runs successfully.

[0082] Optionally, in this embodiment, a second virtual node may be created in the target client. In this embodiment, the second virtual node is a downstream task for the processing tasks of different candidate service data groups. It should be noted that in this embodiment, the second virtual node may also be created before the target task is processed, and it may also be used to process other tasks, which is not limited in this embodiment.

[0083] In an optional manner of this embodiment, after the target service analysis model finishes processing the data of a candidate service data group, the second virtual node will be automatically triggered once. When the number of times the second virtual node is triggered is consistent with the number of each candidate service data group, it can be determined that the second virtual node runs successfully. At this time, the downstream task can sense and start consuming the result data stored in the target data table.

[0084] The solution of this embodiment can screen the service analysis models deployed in the big data service based on the processing objectives of the target task and the attribute information of each service data to obtain the target service analysis module; create a first virtual node; in response to the trigger instruction of the first virtual node, and at the same time process each candidate service data group based on the target service analysis module; further, a second virtual node can be created; after obtaining the processing result of a candidate service data group, the second virtual node is automatically triggered; when the number of times the second virtual node is triggered is consistent with the number of candidate service data groups, it can be determined that the second virtual node runs successfully. At this time, the downstream task can sense and start consuming the result data stored in the target data table, which can realize the parallel processing of the data of different candidate service data groups, and can uniformly consume the downstream of the processing results, improving the efficiency of task processing, saving task processing time, and improving the user experience.

[0085] Embodiment III

[0086] Figure 3 It is a flowchart of a method for processing task data provided in Embodiment II of the present invention. This embodiment further refines the above technical solution, and the technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. As Figure 3 shown, the method includes:

[0087] Step 310: In response to a processing instruction of a target task, obtain candidate service data matching the target task.

[0088] Step 320: When it is determined that the data volume of the candidate service data is greater than a set data volume threshold, group each candidate service data based on the target attribute information of each candidate service data to obtain at least two candidate service data groups.

[0089] Step 330: determine a target business analysis model in the big data computing service that matches the target task, and analyze and process each of the candidate business data groups based on the target business analysis model.

[0090] Step 340, in response to the target candidate business data group's exception processing instruction, perform anomaly detection on each target business data of the target candidate business data group to determine the abnormal business data, and eliminate the abnormal business data; re-process the target candidate business data group from which the abnormal business data has been eliminated based on the target business analysis model.

[0091] The target candidate service data group may be any candidate service data group, which is not limited in this embodiment.

[0092] Optionally, in the present embodiment, in the process of processing each candidate business data group based on the target business analysis model, if an instruction to process an exception for the target candidate business data group is received, anomaly detection can be directly performed on each target business data of the target candidate business data group to determine the abnormal business data and eliminate the abnormal business data; further, the target candidate business data group that has eliminated the abnormal business data can be processed based on the target business analysis model.

[0093] The solution of this embodiment divides the candidate business data group into different candidate business data groups for processing. When a candidate business data group fails during the processing, only the candidate business data group needs to be reprocessed, and there is no need to process all the candidate business data. This improves the fault handling efficiency and does not extend the result output time due to the processing of all the business data.

[0094] In order to better understand the processing method of task data involved in this embodiment, Figure 4 is a schematic diagram of a task data processing flow provided according to Embodiment 3 of the present invention, Figure 4 In the figure, concurrent task 1, concurrent task 2, concurrent task 3 and concurrent task 4 are tasks for processing different candidate task data groups in this embodiment.

[0095] The solution of the embodiment of the present invention can solve the problem of extending the overall result output time due to reprocessing after a single task exception, because the data of all customers are calculated in one task, so all customers will be affected; it can also solve the problem of improving the certainty of the output timeliness of single-task large data volume (TB level) calculation, and can take into account the timeliness and stability of task processing at the same time.

[0096] Embodiment 4

[0097] Figure 5It is a schematic structural diagram of a processing device for task data provided in Embodiment 3 of the present invention. The device is deployed on a target client for execution, and the target client is communicatively connected to a big data computing service, such as Figure 5 As shown, the device includes: a candidate service data acquisition module 510, a grouping module 520, a processing module 53, and a merging module 540.

[0098] Among them, the candidate service data acquisition module 510 is configured to acquire candidate service data matching the target task in response to a processing instruction of the target task;

[0099] The grouping module 520 is configured to, when determining that the data volume of the candidate service data is greater than a set data volume threshold, group the candidate service data based on the target attribute information of each candidate service data to obtain at least two candidate service data groups;

[0100] The processing module 530 is configured to determine a target service analysis model in the big data computing service that matches the target task, and perform analysis and processing on each candidate service data group based on the target service analysis model;

[0101] The merging module 540 is configured to store the processing results of each candidate service data group in a target data table respectively.

[0102] In the solution of this embodiment, the candidate service data acquisition module acquires candidate service data matching the target task in response to a processing instruction of the target task; the grouping module groups the candidate service data based on the target attribute information of each candidate service data to obtain at least two candidate service data groups when determining that the data volume of the candidate service data is greater than a set data volume threshold; the processing module determines a target service analysis model in the big data computing service that matches the target task, and performs analysis and processing on each candidate service data group based on the target service analysis model; the merging module stores the processing results of each candidate service data group in a target data table respectively, so as to quickly and accurately process the service data of different tasks and improve the user experience.

[0103] In an optional implementation manner of this embodiment, the candidate service data acquisition module 810 is specifically configured to determine a target database associated with the target task;

[0104] Determine first service data corresponding to the task identifier of the target task from the target database;

[0105] Screen the first service data based on preset time information to obtain the candidate service data.

[0106] In an alternative implementation of this embodiment, the candidate service data acquisition module 810 is further specifically configured to obtain the number of rows and / or columns of the candidate data table storing the candidate service data, and determine the data volume of the candidate service data based on the number of rows and / or columns;

[0107] Or,

[0108] Determine the file size of the candidate data table, and determine the data volume of the candidate service data based on the file size;

[0109] Or,

[0110] Input the candidate service data into a data volume calculation large model that has been pre-fine-tuned to obtain the data volume of the candidate service data.

[0111] In an alternative implementation of this embodiment, the grouping module 820 is specifically configured to determine the processing objective of the target task, and determine the attribute information of the service data associated with the processing objective;

[0112] Determine the attribute information of the service data associated with the processing objective as the target attribute information;

[0113] Cluster each of the candidate service data based on the target attribute information to obtain each candidate service data group;

[0114] Wherein, the target attribute information includes at least one of the following:

[0115] Name, relationship attribute, identification information, time attribute, status attribute, location attribute, and label attribute.

[0116] In an alternative implementation of this embodiment, the processing module 830 is specifically configured to screen the service analysis models deployed in the big data service based on the processing objective of the target task and the attribute information of each service data to obtain the target service analysis module;

[0117] Create a first virtual node;

[0118] In response to the trigger instruction of the first virtual node, simultaneously process each candidate service data group based on the target service analysis module.

[0119] In an alternative implementation of this embodiment, the processing device for task data further includes: a second virtual node creation module, configured to create a second virtual node;

[0120] When the processing result of a candidate service data group is obtained, automatically trigger the second virtual node;

[0121] When the number of trigger times of the second virtual node is consistent with the number of the candidate service data groups, it is determined that the second virtual node runs successfully.

[0122] In an optional implementation manner of this embodiment, the processing device for task data further includes: an exception handling module, configured to perform exception detection on each target service data of the target candidate service data group in response to a processing exception instruction of the target candidate service data group, so as to determine exception service data, and eliminate the exception service data;

[0123] Re - process the target candidate service data group with the exception service data eliminated based on the target service analysis model.

[0124] The processing device for task data provided by the embodiments of the present invention can execute the processing method for task data provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0125] In the technical solution of the embodiments of the present invention, the collection, storage, use, processing, transmission, provision, and disclosure of the involved service data all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0126] Embodiment Five

[0127] Figure 6 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0128] Such as Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0129] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above. For example, for a method of processing task data, which is executed by a target client communicatively connected to a big data computing service, the method is characterized in that it includes: in response to a processing instruction of a target task, obtaining candidate service data matching the target task; in the case where it is determined that the data volume of the candidate service data is greater than a set data volume threshold, grouping the candidate service data based on the target attribute information of each candidate service data to obtain at least two candidate service data groups; determining a target service analysis model in the big data computing service that matches the target task, and respectively analyzing and processing each candidate service data group based on the target service analysis model; and respectively storing the processing results of each candidate service data group in a target data table.

[0131] In some embodiments, the method for processing task data may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for processing task data described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method by any other suitable means (e.g., by means of firmware).

[0132] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0134] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0138] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0139] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0140] The embodiment of the present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the detection method of the database provided in any embodiment of the present application.

[0141] In the process of implementing the computer program product, the computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0142] It should be noted that in the embodiments of the present application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be considered exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solution of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0143] Note that the above are only the preferred embodiments of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments here, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments only. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for processing task data, executed by a target client, wherein the target client is connected to a big data computing service in communication, characterized in that: The method comprises: In response to a processing instruction of a target task, acquiring candidate business data matching the target task; When it is determined that the data volume of the candidate business data is greater than a set data volume threshold, grouping each of the candidate business data based on target attribute information of each of the candidate business data to obtain at least two candidate business data groups; Determine a target business analysis model in the big data computing service that matches the target task, and analyze and process each of the candidate business data groups based on the target business analysis model; The processing results of each candidate business data group are stored in the target data table respectively.

2. The method for processing task data according to claim 1, characterized in that: The acquiring of candidate business data matching the target task includes: Determining a target database associated with the target task; Determine, from the target database, first business data corresponding to the task identifier of the target task; The first service data is screened based on preset time information to obtain the candidate service data.

3. The method for processing task data according to claim 1, characterized in that: The determining the data volume of the candidate service data includes: Acquire the number of rows and / or columns of a candidate data table storing the candidate business data, and determine the data volume of the candidate business data based on the number of rows and / or columns; or, Determine a file size of the candidate data table, and determine a data volume of the candidate service data based on the file size; or, The candidate business data is input into a pre-fine-tuned large data volume calculation model to obtain the data volume of the candidate business data.

4. The method for processing task data according to claim 1, characterized in that: The step of grouping the candidate business data based on the target attribute information of the candidate business data to obtain at least two candidate business data groups includes: Determining a processing target of the target task, and determining attribute information of business data associated with the processing target; Determining attribute information of business data associated with the processing target as target attribute information; Clustering each of the candidate business data based on the target attribute information to obtain each of the candidate business data groups; The target attribute information includes at least one of the following: Name, relationship attributes, identification information, time attributes, state attributes, location attributes, and tag attributes.

5. The method for processing task data according to claim 4, characterized in that: The determining a target business analysis model matching the target task in the big data computing service, and analyzing and processing each of the candidate business data groups based on the target business analysis model, includes: Based on the processing target of the target task and the attribute information of each of the business data, each business analysis model deployed in the big data service is screened to obtain the target business analysis module; Creating a first virtual node; In response to the trigger instruction of the first virtual node, each of the candidate business data groups is processed based on the target business analysis module.

6. The method for processing task data according to claim 1, characterized in that: The method further comprises: Creating a second virtual node; After obtaining a processing result of a candidate service data group, automatically triggering the second virtual node; When the triggering times of the second virtual node are consistent with the number of the candidate service data groups, it is determined that the second virtual node runs successfully.

7. The method for processing task data according to claim 1, characterized in that: The method further comprises: In response to the abnormality processing instruction of the target candidate service data group, performing abnormality detection on each target service data of the target candidate service data group to determine abnormal service data, and eliminating the abnormal service data; The target candidate business data group from which abnormal business data is eliminated is reprocessed based on the target business analysis model.

8. A task data processing device, deployed on a target client for execution, wherein the target client is connected to a big data computing service for communication, characterized in that: The device comprises: A candidate business data acquisition module, used to respond to a processing instruction of a target task and acquire candidate business data matching the target task; A grouping module, configured to group each candidate business data based on target attribute information of each candidate business data to obtain at least two candidate business data groups when it is determined that the data volume of the candidate business data is greater than a set data volume threshold; A processing module, used to determine a target business analysis model matching the target task in the big data computing service, and analyze and process each of the candidate business data groups based on the target business analysis model; The merging module is used to store the processing results of each candidate business data group in the target data table respectively.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the task data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the task data processing method according to any one of claims 1 to 7 when executed.

Images