Business data processing method, device, computer readable medium and electronic device

By aggregating and merging business data of multiple business types, and generating merged business aggregate data, the problem of insufficient accuracy of business prediction data in the prior art is solved, and the accuracy of business prediction is improved.

CN113689028BActive Publication Date: 2024-11-15SHANSHU TECH (BEIJING) CO LTD +4
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110821870.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-19
Publication Date
2024-11-15
Estimated Expiration
2041-07-19

AI Technical Summary

Technical Problem

The prior art is difficult to effectively process multiple types of service data in the service data processing scenario, resulting in insufficient accuracy of service prediction data.

Method used

By acquiring a set of business data of multiple business types, the business data with the same identification information is aggregated, and the data is merged based on a distributed cluster to generate merged business aggregate data for calculating business predictions.

Benefits of technology

By comprehensively considering business data of multiple business types, aggregation of consolidated business data that can fully characterize business characteristics can be generated, thereby improving the accuracy of business prediction data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113689028B_ABST
    Figure CN113689028B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a business data processing method, device, computer-readable medium and electronic device. The method includes: obtaining a business data set of at least two business types, the business data set includes multiple business data, and the business data includes identification information; for each target business data set, aggregating the business data with the same identification information in the target business data set to obtain the business aggregation data corresponding to each identification information in the target business data set, the target business data set is any one of the business data sets of at least two business types; based on the business aggregation data corresponding to each business data set, merging the business aggregation data with the same identification information to obtain the merged business aggregation data corresponding to each identification information for calculating the business forecast data. The technical solution of the embodiment of the present application can improve the accuracy of calculating the business forecast data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer and data processing technology, and in particular, to a business data processing method, device, computer-readable medium and electronic device. Background Art

[0002] In business data processing scenarios, data can generally be processed through designed custom functions. However, this cannot support the processing of multiple types of business data. When business forecast data needs to be calculated through business data, the accuracy of the business forecast data cannot be guaranteed. Based on this, how to improve the accuracy of calculating business forecast data is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a business data processing method, device, computer program product or computer program, computer-readable medium and electronic device, which can improve the accuracy of calculating business forecast data at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a business data processing method is provided, the method comprising: obtaining a business data set of at least two business types, the business data set comprising multiple business data, the business data comprising identification information; for each target business data set, aggregating the business data with the same identification information in the target business data set to obtain business aggregated data corresponding to each identification information in the target business data set, the target business data set being any one of the business data sets of the at least two business types; based on the business aggregated data corresponding to each business data set, merging the business aggregated data with the same identification information to obtain merged business aggregated data corresponding to each identification information, the merged business aggregated data being used to calculate business forecast data.

[0006] According to one aspect of an embodiment of the present application, a business data processing device is provided, the device comprising: a first acquisition unit, used to acquire a business data set of at least two business types, the business data set comprising multiple business data, the business data comprising identification information; an aggregation unit, used to aggregate the business data with the same identification information in the target business data set for each target business data set, to obtain business aggregated data corresponding to each identification information in the target business data set, the target business data set being any one of the business data sets of the at least two business types; a merging unit, used to merge the business aggregated data with the same identification information based on the business aggregated data corresponding to each business data set, to obtain merged business aggregated data corresponding to each identification information, the merged business aggregated data being used to calculate business forecast data.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the aggregation unit is configured to: send the business data in the target business data set to a distributed cluster, and aggregate the business data with the same identification information in the target business data set through the distributed cluster.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the aggregation unit is further configured to: group the business data in the target business data set based on the identification information of the business data to obtain business data groups corresponding to each identification information; and concurrently aggregate the business data in each obtained business data group through each cluster node in the distributed cluster.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the business data also includes business information, and the aggregation unit is further configured to: distribute the business data groups corresponding to the various identification information to the various cluster nodes according to the load of the various cluster nodes in the distributed cluster; for each cluster node in the distributed cluster, serialize and aggregate the business information of each business data in the business data group through the cluster node.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the merging unit is configured to: determine the business aggregation data with the same identification information in each cluster node of the distributed cluster; respectively splice the business aggregation data corresponding to each identification information to obtain the merged business aggregation data corresponding to each identification information.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the device also includes a second acquisition unit, which is used to acquire the merged business aggregation data corresponding to each identification information; a distribution unit, which is used to distribute the merged business aggregation data corresponding to each identification information to each cluster node in the distributed cluster according to the load of each cluster node in the distributed cluster; and a calculation unit, which is used to calculate the business prediction data corresponding to the target identification information through the corresponding cluster node based on the merged business aggregation data corresponding to each target identification information, and the target identification information is any one of the identification information.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the computing unit is configured as follows: the corresponding cluster node restores the merged business aggregation data corresponding to the target identification information to obtain the business data corresponding to the target identification information under each business type; the corresponding cluster node calls the operation code matching the business requirements; the business data corresponding to the target identification information under each business type is used as input data, and the business prediction data corresponding to the target identification information is calculated by running the operation code.

[0013] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the business data processing method described in the above embodiment.

[0014] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the business data processing method as described in the above embodiment is implemented.

[0015] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the business data processing method as described in the above embodiments.

[0016] In the technical solutions provided in some embodiments of the present application, business aggregate data corresponding to each identification information is obtained by aggregating business data with the same identification information in a business data set, and then, based on the business aggregate data corresponding to each business data set, the business aggregate data with the same identification information is merged to obtain merged business aggregate data corresponding to each identification information for calculating business forecast data. Since merged business aggregate data that can fully characterize business characteristics is obtained by comprehensively considering business data sets of multiple business types, more accurate business forecast data can be calculated by merging business aggregate data.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0020] Figure 2 A flowchart of a method for processing business data according to an embodiment of the present application is shown;

[0021] Figure 3 A detailed flow chart of aggregating the business data having the same identification information in the target business data set through the distributed cluster according to an embodiment of the present application is shown;

[0022] Figure 4 A detailed flow chart of concurrently aggregating the service data in each service data group obtained according to an embodiment of the present application is shown;

[0023] Figure 5 A detailed flow chart of merging service aggregation data with the same identification information to obtain merged service aggregation data corresponding to each identification information according to an embodiment of the present application is shown;

[0024] Figure 6 A further flow chart of a method for processing business data according to an embodiment of the present application is shown;

[0025] Figure 7A detailed flow chart showing calculation of the business prediction data corresponding to the target identification information by the corresponding cluster node according to an embodiment of the present application is shown;

[0026] Figure 8 A schematic diagram of a service data flow according to an embodiment of the present application is shown;

[0027] Fig. 9 A block diagram of a service data processing device according to an embodiment of the present application is shown;

[0028] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0030] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0033] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0035] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0036] like Figure 1 As shown, the system architecture may include a server 101 (which may also include one or more of a smart phone, a tablet computer, and a portable computer), a distributed cluster 102, and a network 103. The network 103 is a medium for providing a communication link between the server 101 and the distributed cluster 102. The network 103 may include various connection types, such as a wired communication link, a wireless communication link, and the like.

[0037] In one embodiment of the present application, the business data processing method may be executed by the server 101 , and further, may be executed by the distributed cluster 102 in cooperation with the server 101 .

[0038] Specifically, the server 101 may obtain a business data set of at least two business types, wherein the business data set includes multiple business data, and the business data includes identification information. For each target business data set, the business data with the same identification information in the target business data set may be aggregated through the distributed cluster 102 to obtain business aggregate data corresponding to each identification information in the target business data set. The target business data set is any one of the business data sets of the at least two business types. Then, the server 101 may merge the business aggregate data with the same identification information based on the business aggregate data corresponding to each business data set to obtain merged business aggregate data corresponding to each identification information. The merged business aggregate data is used to calculate business prediction data.

[0039] The present application obtains merged business aggregation data that can fully characterize business characteristics by comprehensively considering business data sets of multiple business types, thereby making it possible to calculate more accurate business prediction data by merging the business aggregation data.

[0040] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0041] Figure 2 A flowchart of a business data processing method according to an embodiment of the present application is shown. The business data processing method can be executed by a device having a computing and processing function, such as Figure 1 The servers and distributed clusters shown in the figure are used for execution. Figure 2 As shown, the business data processing method at least includes steps 210 to 230, which are described in detail as follows:

[0042] In step 210, a business data set of at least two business types is acquired, wherein the business data set includes a plurality of business data, and the business data includes identification information.

[0043] In the present application, the business data processing method can be applied in the scenario of processing cargo management data, wherein the cargo management data includes cargo sales data, cargo inventory data, cargo price data and other business types of data.

[0044] Furthermore, in the scenario of processing goods management data, obtaining business data sets of at least two business types may refer to obtaining a goods sales data set, a goods inventory data set, and a goods price data set.

[0045] The business data set of each business type includes multiple business data, and each business data includes an identification information. For example, the goods sales data set includes multiple goods sales data, wherein each goods sales data includes an identification information, and the identification information can be the type of goods, the model of the goods, the name of the goods or the number of the goods.

[0046] It should be noted that, in the present application, the business data processing method can be applied not only to the scenario of processing cargo management data, but also to other scenarios, such as the scenario of processing financial data.

[0047] It should also be noted that the business data in the business data set is historical business data.

[0048] Continue to refer to Figure 2In step 230, for each target business data set, the business data with the same identification information in the target business data set are aggregated to obtain business aggregate data corresponding to each identification information in the target business data set, and the target business data set is any one of the business data sets of the at least two business types.

[0049] In one embodiment of the present application, the business data with the same identification information in the target business data set is aggregated by sending the business data in the target business data set to a distributed cluster, and aggregating the business data with the same identification information in the target business data set through the distributed cluster.

[0050] It should be understood that in the present application, it is necessary to perform aggregation processing on the business data in each business data set.

[0051] In this embodiment, the distributed cluster may be a Spark cluster, wherein Spark is a big data parallel computing framework based on memory computing, and can process large-scale data in the form of a cluster through a master+slave management mode.

[0052] In this implementation, the distributed cluster aggregates the service data with the same identification information in the target service data set, and the following method can be used: Figure 3 Follow the steps shown.

[0053] See also Figure 3 , shows a detailed flow chart of aggregating the business data with the same identification information in the target business data set through the distributed cluster according to an embodiment of the present application. Specifically including steps 231 to 232:

[0054] In step 231, based on the identification information of the business data, the business data in the target business data set are grouped to obtain business data groups corresponding to each piece of identification information.

[0055] Specifically, since each piece of business data in the business data set corresponds to an identification information, the business data with the same identification information can be grouped together. For example, in the scenario of processing goods management data, the goods sales data set includes 6 sales data as shown in Table 1 below. In this goods sales data set, "(identification information, sales information 1, sales information 2, sales information 3) = (a, a1, a2, a3)", "(identification information, sales information 1, sales information 2, sales information 3) = (a, a4, a5, a6)", "(identification information, sales information 1, sales information 2, sales information 3) = (a, a7, a8, a9)" can be grouped together; (identification information, sales information 1, sales information 2, sales information 3) = (b, b1, b2, b3)", "(identification information, sales information 1, sales information 2, sales information 3) = (b, b4, b5, b6)" can be grouped together; (identification information, sales information 1, sales information 2, sales information 3) = (c, c1, c2, c3)" can be grouped together.

[0056] Identification Information Sales Information 1 Sales Information 2 Sales Information 3 Model a a1 a2 a3 Model c c1 c2 c3 Model b b1 b2 b3 Model a a4 a5 a6 Model c c4 c5 c6 Model a a7 a8 a9

[0057] Table 1

[0058] In step 232, the business data in each obtained business data group is aggregated concurrently through each cluster node in the distributed cluster.

[0059] In the present application, the business data also includes business information. For example, in Table 1, “sales information 1, sales information 2, sales information 3” are the business information of the business data.

[0060] In one embodiment of the present application, the business data in each business data group obtained is aggregated concurrently by each cluster node in the distributed cluster, and the following can be performed: Figure 4 Follow the steps shown.

[0061] See also Figure 4 , shows a detailed flowchart of concurrently aggregating the business data in each business data group obtained according to an embodiment of the present application. Specifically including steps 2321 to 2322:

[0062] In step 2321, the business data groups corresponding to the respective identification information are distributed to the respective cluster nodes according to the load of the respective cluster nodes in the distributed cluster.

[0063] In step 2322, for each cluster node in the distributed cluster, the business information of each piece of business data in the business data group is serialized and aggregated through the cluster node.

[0064] Specifically, for example, in Table 1, the business data group with identification information "model a" includes three sales data: "(identification information, sales information 1, sales information 2, sales information 3) = (a, a1, a2, a3)", "(identification information, sales information 1, sales information 2, sales information 3) = (a, a4, a5, a6)", and "(identification information, sales information 1, sales information 2, sales information 3) = (a, a7, a8, a9)". The business information in the three sales data is serialized and aggregated through the cluster node to obtain the business aggregation data corresponding to the identification information "model a" "(sales information 1 sales information 2 sales information 3) = (a1 a2 a3, a4a5 ​​a6, a7 a8 a9)".

[0065] In the present application, the business data in each obtained business data group is aggregated concurrently through each cluster node in the distributed cluster. This has the advantage of increasing the speed of aggregating the business data in the business data set and improving the aggregation efficiency.

[0066] Continue to refer to Figure 2 In step 250, based on the business aggregation data corresponding to each business data set, the business aggregation data with the same identification information is merged to obtain the merged business aggregation data corresponding to each identification information, and the merged business aggregation data is used to calculate the business prediction data.

[0067] By aggregating the business data in each business data set in step 230, each business data set corresponds to business aggregated data including each identification information. Since the business data of each identification information in a business data set is aggregated by different cluster nodes, it can be understood that the business aggregated data of each identification information in each business data set is distributed in different cluster nodes.

[0068] Based on this, in one embodiment of the present application, based on the business aggregation data corresponding to each business data set, the business aggregation data with the same identification information is merged to obtain the merged business aggregation data corresponding to each identification information. Figure 5 Follow the steps shown.

[0069] Reference Figure 5 , shows a detailed flow chart of merging service aggregation data with the same identification information to obtain the merged service aggregation data corresponding to each identification information according to an embodiment of the present application. Specifically including steps 251 to 252:

[0070] In step 251, business aggregation data having the same identification information is determined in each cluster node of the distributed cluster.

[0071] In step 252, the service aggregation data corresponding to each piece of identification information are concatenated to obtain the merged service aggregation data corresponding to each piece of identification information.

[0072] Specifically, for example, in the scenario of processing goods management data, in the sales data set, the business aggregation data corresponding to the identification information "model a" is "(sales information 1 sales information 2 sales information 3) = (a1 a2 a3, a4 a5 a6, a7a8 a9)"; in the inventory data set, the business aggregation data corresponding to the identification information "model a" is "(inventory information 1 inventory information 2 inventory information 3) = (aa1 aa2 aa3, aa4 aa5 aa6, aa7 aa8 aa9)"; in the price data set, the business aggregation data corresponding to the identification information "model a" is "(price information 1 price information 2 price information 3) = (aaa1 aaa2aaa3, aaa4 aaa5 aaa6, aaa7 aaa8 aaa9)".

[0073] By merging the various business aggregation data corresponding to the identification information "model a", the merged business aggregation data corresponding to the identification information "model a" is obtained: "(sales information 1 sales information 2 sales information 3) = (a1 a2 a3, a4 a5 a6, a7 a8 a9); (inventory information 1 inventory information 2 inventory information 3) = (aa1 aa2 aa3, aa4 aa5 aa6, aa7 aa8aa9); (price information 1 price information 2 price information 3) = (aaa1 aaa2 aaa3, aaa4 aaa5 aaa6, aaa7 aaa8aa9)".

[0074] In the present application, the business data corresponding to the identification information is first aggregated into business aggregate data, and then the business aggregate data corresponding to the identification information is merged into merged business aggregate data. The scattered business data can be combined into one piece of data that can comprehensively characterize the business characteristics, thereby enabling more accurate business prediction data to be calculated by merging the business aggregate data.

[0075] In the present application, after obtaining the combined service aggregation data corresponding to each identification information, the following operations may also be performed: Figure 6 Steps shown.

[0076] See also Figure 6 , shows a further flow chart of a business data processing method according to an embodiment of the present application. Specifically comprising steps 260 to 280:

[0077] In step 260, the combined service aggregation data corresponding to each piece of identification information is obtained.

[0078] In step 270, the merged service aggregation data corresponding to each piece of identification information is distributed to each cluster node of the distributed cluster according to the load of each cluster node in the distributed cluster.

[0079] In step 280, based on the merged business aggregation data corresponding to each target identification information, the business prediction data corresponding to the target identification information is calculated by the corresponding cluster node, and the target identification information is any one of the respective identification information.

[0080] In the present application, each merged business aggregation data is distributed to each cluster node, and each cluster node calculates the business prediction data of the corresponding identification information based on the distributed merged business aggregation data. The advantage is that the speed of calculating the business prediction data can be enhanced and the calculation efficiency can be improved.

[0081] Distributing the merged business aggregation data to each cluster node according to the load of each cluster node can balance the computing pressure of each cluster node, while making full use of the computing resources of each cluster node, further improving the efficiency of computing business prediction data.

[0082] In one embodiment of step 280, the business prediction data corresponding to the target identification information is calculated by the corresponding cluster node, which can be as follows: Figure 7 Follow the steps shown.

[0083] See also Figure 7 , shows a detailed flow chart of calculating the business prediction data corresponding to the target identification information by the corresponding cluster node according to an embodiment of the present application. Specifically including steps 281 to 283:

[0084] In step 281, the corresponding cluster node performs restoration processing on the merged service aggregation data corresponding to the target identification information to obtain service data corresponding to the target identification information under each service type.

[0085] Specifically, as shown in Table 2, in the scenario of processing cargo management data, the business data corresponding to the identification information "model a" under each business type.

[0086]

[0087] Table 2

[0088] In step 282, the corresponding cluster node calls the operation code that matches the business requirement.

[0089] In step 283, the business data corresponding to the target identification information under each business type is used as input data, and the business prediction data corresponding to the target identification information is calculated by running the operation code.

[0090] Specifically, for example, in a scenario of processing cargo management data, business forecast data may include sales forecast data, inventory forecast data, and price forecast data.

[0091] In order to make those skilled in the art better understand the present application, Figure 8 The business data processing method proposed in this application is explained.

[0092] See also Figure 8 , showing a schematic diagram of a business data flow according to an embodiment of the present application.

[0093] like Figure 8 As shown, in step 801, a service data set of a first service type and a service data set of a second service type are obtained.

[0094] In step 802, in a service data set of a first service type and a service data set of a second service type, service data of identification information 1 and service data of identification information 2 are aggregated respectively to obtain service aggregate data.

[0095] In step 803, the service aggregation data corresponding to the first service type and the service aggregation data corresponding to the second service type are merged according to identification information 1, and the service aggregation data corresponding to the first service type and the service aggregation data corresponding to the second service type are merged according to identification information 2.

[0096] In step 804 , the business prediction data corresponding to identification information 1 and identification information 2 are calculated based on the data obtained in step 803 .

[0097] In one embodiment of the present application, the proposed business data processing method is applicable to the programming method based on Pandas UDF in Py Spark (i.e., the Python programming interface in the Spark suite). Specifically, Spark can read multiple algorithm input sources (i.e., multiple types of business data sets) into Dataframes, and group, fold and serialize (i.e., aggregate) them according to the minimum parallel unit Key (i.e., identification information), and then join all Dataframes by Key to obtain a full data Dataframe, and finally execute Pandas UDF on the full data Dataframe in a Group by key apply manner to obtain the calculation result.

[0098] Specifically, in this embodiment, the following 6 steps are included:

[0099] Step 1: Determine the minimum parallel unit Key (keyword). The minimum parallel unit is defined as the minimum data unit required to complete an algorithm process in the algorithm design. It is also the keyword for Pandas UDF to do Group by. For example, if a prediction algorithm is modeled by a single product, the minimum parallel unit is the product ID or product name. If a prediction algorithm is modeled by category, the minimum parallel unit is the category ID or category name.

[0100] Step 2: Use Spark to read multiple input data from files or databases into multiple Dataframes.

[0101] Step 3: Fold the Dataframe read in step 2 through the designed Pandas UDF function. In this function, the data will be grouped according to the minimum parallel unit and the grouped data will be serialized. Each folded Dataframe will be transformed into a new Dataframe with 2 columns and n rows after the folding function. One column is the parallel unit key, and the other column is the json string serialized according to the parallel unit. This json string contains all the information about this parallel unit in this Dataframe; n is the number of parallel unit keys contained in this Dataframe.

[0102] Step 4: All Dataframes are folded in step 3 and then merged into a large table by joining with the minimum parallel unit Key. This step will get a full data Dataframe with m+l columns and n rows, where m is the number of Dataframes read in step 2, the other column is the parallel unit key, and n is the number of parallel unit keys.

[0103] Step 5. After the above operations, a unit in the full data Dataframe contains all the data of a certain type of input under a minimum parallel unit. Next, you only need to write a full process. In the data acquisition part, use pd.read_json. The function reads the serialized json string in the previous unit to restore the data of this minimum parallel unit. Its format is Pandas Dataframe. All the main processes of the algorithm can be written into this Pandas UDF, which is completely implemented by Python.

[0104] Step 6: Group by the key on the entire data frame and apply the pandas UDF in step 5 to get the final algorithm result in the format of Spark Dataframe.

[0105] Through this scheduling method, Spark only undertakes the data reading, processing and distribution process. The specific algorithm process is in Pandas UDF. With only a small development cost, it can be achieved by using various algorithm packages in Python and taking advantage of Spark's parallel mode to accelerate the algorithm and improve computing efficiency.

[0106] In the technical solutions provided in some embodiments of the present application, business aggregate data corresponding to each identification information is obtained by aggregating business data with the same identification information in a business data set, and then, based on the business aggregate data corresponding to each business data set, the business aggregate data with the same identification information is merged to obtain merged business aggregate data corresponding to each identification information for calculating business forecast data. Since merged business aggregate data that can fully characterize business characteristics is obtained by comprehensively considering business data sets of multiple business types, more accurate business forecast data can be calculated by merging business aggregate data.

[0107] The following describes an apparatus embodiment of the present application, which can be used to execute the business data processing method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the business data processing method of the present application.

[0108] Fig. 9 A block diagram of a business data processing device according to an embodiment of the present application is shown.

[0109] Reference Fig. 9 As shown, a business data processing device 900 according to an embodiment of the present application includes: a first acquisition unit 901, an aggregation unit 902 and a merging unit 903.

[0110] Among them, the first acquisition unit 901 is used to obtain a business data set of at least two business types, wherein the business data set includes multiple business data, and the business data includes identification information; the aggregation unit 902 is used to aggregate the business data with the same identification information in the target business data set for each target business data set, and obtain the business aggregation data corresponding to each identification information in the target business data set, and the target business data set is any one of the business data sets of the at least two business types; the merging unit 903 is used to merge the business aggregation data with the same identification information based on the business aggregation data corresponding to each business data set, and obtain the merged business aggregation data corresponding to each identification information, and the merged business aggregation data is used to calculate the business forecast data.

[0111] In some embodiments of the present application, based on the aforementioned scheme, the aggregation unit 902 is configured to: send the business data in the target business data set to a distributed cluster, and aggregate the business data with the same identification information in the target business data set through the distributed cluster.

[0112] In some embodiments of the present application, based on the aforementioned scheme, the aggregation unit 902 is further configured to: group the business data in the target business data set based on the identification information of the business data to obtain business data groups corresponding to each identification information; and concurrently aggregate the business data in each obtained business data group through each cluster node in the distributed cluster.

[0113] In some embodiments of the present application, based on the aforementioned scheme, the business data also includes business information, and the aggregation unit 902 is further configured to: distribute the business data groups corresponding to the various identification information to the various cluster nodes according to the load of the various cluster nodes in the distributed cluster; for each cluster node in the distributed cluster, serialize and aggregate the business information of each business data in the business data group through the cluster node.

[0114] In some embodiments of the present application, based on the aforementioned scheme, the merging unit 903 is configured to: determine the business aggregation data with the same identification information in each cluster node of the distributed cluster; respectively splice the business aggregation data corresponding to each identification information to obtain the merged business aggregation data corresponding to each identification information.

[0115] In some embodiments of the present application, based on the aforementioned scheme, the device also includes a second acquisition unit, which is used to acquire the merged business aggregation data corresponding to each identification information; a distribution unit, which is used to distribute the merged business aggregation data corresponding to each identification information to each cluster node in the distributed cluster according to the load of each cluster node in the distributed cluster; and a calculation unit, which is used to calculate the business prediction data corresponding to the target identification information through the corresponding cluster node based on the merged business aggregation data corresponding to each target identification information, and the target identification information is any one of the identification information.

[0116] In some embodiments of the present application, based on the aforementioned scheme, the computing unit is configured as follows: the corresponding cluster node restores the merged business aggregation data corresponding to the target identification information to obtain the business data corresponding to the target identification information under each business type; the corresponding cluster node calls the operation code matching the business requirements; the business data corresponding to the target identification information under each business type is used as input data, and the business prediction data corresponding to the target identification information is calculated by running the operation code.

[0117] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0118] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method described in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0120] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0121] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0122] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0123] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0125] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the business data processing method described in the above embodiment.

[0126] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the business data processing method described in the above embodiment.

[0127] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0128] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0129] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0130] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A business data processing method, It is characterized in that The method comprises: Acquire business data sets of at least two business types, the business data sets including at least one of a goods sales data set, a goods inventory data set, and a goods price data set, the business data set including a plurality of business data, and the business data including identification information; For each target business data set, aggregating business data with the same identification information in the target business data set to obtain business aggregate data corresponding to each identification information in the target business data set, wherein the target business data set is any one of the business data sets of the at least two business types; Also includes: Obtain the combined business aggregation data corresponding to each identification information; Distribute the merged business aggregation data corresponding to each identification information to each cluster node of the distributed cluster according to the load of each cluster node in the distributed cluster; and calculate the business prediction data corresponding to the identification information based on the distributed merged business aggregation data by each cluster node; Based on the merged business aggregation data corresponding to each target identification information, the business prediction data corresponding to the target identification information is calculated by the corresponding cluster node, and the target identification information is any one of the respective identification information; Based on the business aggregation data corresponding to each business data set, the business aggregation data with the same identification information are spliced ​​to obtain the merged business aggregation data corresponding to each identification information, and the merged business aggregation data is used to calculate the business prediction data.

2. The method according to claim 1, It is characterized in that The aggregating the business data with the same identification information in the target business data set includes: The business data in the target business data set is sent to a distributed cluster, and the business data with the same identification information in the target business data set is aggregated through the distributed cluster.

3. The method according to claim 2, It is characterized in that The aggregating the business data having the same identification information in the target business data set by using the distributed cluster includes: Based on the identification information of the business data, the business data in the target business data set are grouped to obtain business data groups corresponding to each piece of identification information; The business data in each obtained business data group is aggregated concurrently through each cluster node in the distributed cluster.

4. The method according to claim 3, It is characterized in that The business data also includes business information, and the business data in each business data group obtained by each cluster node in the distributed cluster is concurrently aggregated, including: Distributing the service data groups corresponding to the respective identification information to the respective cluster nodes according to the load of the respective cluster nodes in the distributed cluster; For each cluster node in the distributed cluster, the business information of each piece of business data in the business data group is serialized and aggregated through the cluster node.

5. The method according to claim 3, It is characterized in that The business aggregation data corresponding to each business data set is spliced ​​with the business aggregation data having the same identification information to obtain the merged business aggregation data corresponding to each identification information, including: Determining service aggregation data having the same identification information in each cluster node of the distributed cluster; The business aggregation data corresponding to each piece of identification information are spliced ​​respectively to obtain the merged business aggregation data corresponding to each piece of identification information.

6. The method according to claim 1, It is characterized in that The calculating, by the corresponding cluster node, the business prediction data corresponding to the target identification information includes: The corresponding cluster node performs restoration processing on the merged service aggregation data corresponding to the target identification information to obtain service data corresponding to the target identification information under each service type; The corresponding cluster node calls the operation code that matches the business requirement; The business data corresponding to the target identification information under each business type is used as input data, and the business prediction data corresponding to the target identification information is calculated by running the operation code.

7. A business data processing device, It is characterized in that The device comprises: A first acquisition unit is used to acquire business data sets of at least two business types, wherein the business data sets include at least one of a goods sales data set, a goods inventory data set, and a goods price data set, wherein the business data set includes a plurality of business data, and the business data includes identification information; an aggregation unit, configured to aggregate, for each target business data set, business data having the same identification information in the target business data set to obtain business aggregate data corresponding to each piece of identification information in the target business data set, wherein the target business data set is any one of the business data sets of the at least two business types; It also includes a second acquisition unit, which is used to acquire the merged business aggregation data corresponding to each identification information; a distribution unit, which is used to distribute the merged business aggregation data corresponding to each identification information to each cluster node in the distributed cluster according to the load of each cluster node in the distributed cluster; each cluster node calculates the business prediction data corresponding to the identification information based on the distributed merged business aggregation data; a calculation unit, which is used to calculate the business prediction data corresponding to the target identification information through the corresponding cluster node based on the merged business aggregation data corresponding to each target identification information, and the target identification information is any one of the identification information The splicing unit is used to splice the business aggregation data with the same identification information based on the business aggregation data corresponding to each business data set, so as to obtain the merged business aggregation data corresponding to each identification information, and the merged business aggregation data is used to calculate the business prediction data.

8. An electronic device, It is characterized in that The electronic device includes one or more processors and one or more memories, and at least one program code is stored in the one or more memories. The at least one program code is loaded and executed by the one or more processors to implement the operations performed by the business data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the business data processing method according to any one of claims 1 to 6.