Method and device for carrying out distributed metering on data set, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of distributed metering, and provides a method and device for carrying out distributed metering on a data set, computer equipment and a readable storage medium. The method comprises the following steps: packaging each metering element to obtain a metering operator corresponding to each metering element; selecting corresponding metering operators for combination to obtain each metering tool and deploying the metering tools on a plurality of computing nodes; dividing the data set into a plurality of data subsets; and sending the plurality of data subsets to different computing nodes, so that each computing node meters the data subsets according to the corresponding metering tool. By adopting the method, the coupling degree between the metering elements can be reduced, and the metering efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed metering technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for performing distributed metering on a data set. Background Art

[0002] Due to the complexity and dynamic changes of computing environment, distributed metering technology has emerged. Distributed metering is a method of collecting and processing data in a distributed system, which can collect and process data efficiently and in real time.

[0003] When performing distributed metering on a data set, multiple computing nodes can be used to complete the task of metering the data set. However, traditional processing methods make it difficult to balance the scalability and efficiency of metering. Summary of the invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for distributed metering of data sets in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for performing distributed metering on a data set, comprising:

[0006] Encapsulating each metering element to obtain a metering operator corresponding to each metering element;

[0007] Select corresponding metering operators and combine them to obtain each metering tool and deploy it on multiple computing nodes;

[0008] Divide the data set into multiple data subsets evenly;

[0009] The multiple data subsets are sent to different computing nodes, so that each computing node measures the data subset according to a corresponding measurement tool.

[0010] In one embodiment, dividing the data set into multiple data subsets evenly includes:

[0011] Obtaining the total amount of the data set;

[0012] According to the computing capacity of the computing node, the amount of data that can be processed by the computing node is obtained;

[0013] According to the total amount and the processable data amount, the data set is evenly divided into a plurality of data subsets.

[0014] In one embodiment, the data set is evenly divided into a plurality of data subsets according to the total amount and the processable data amount, including:

[0015] Obtaining a data average according to the total amount and the number of computing nodes;

[0016] When the data average amount is smaller than the processable data amount, dividing the data set into a plurality of data subsets on average according to the data average amount;

[0017] When the data average component is greater than the processable data volume, the data set is divided into multiple batches of data sets according to the processable data volume and the number of computing nodes, and each batch of data sets is divided into corresponding data subsets according to the processable data volume and the number of computing nodes.

[0018] In one embodiment, the step of evenly dividing the data set into a plurality of data subsets according to the data average component includes:

[0019] Obtaining the processing time level of each data in the data set;

[0020] According to the processing time level of each data in the data set, the data set is divided to obtain a first type of data set corresponding to different processing time levels;

[0021] According to the number of computing nodes, the first type of data set corresponding to each processing time level is evenly divided to obtain a corresponding plurality of second type of data sets;

[0022] A plurality of data subsets are obtained according to the plurality of second-category data sets corresponding to different processing time levels.

[0023] In one embodiment, obtaining the processing time level of each data in the data set includes:

[0024] According to the measurement tools required for measuring each piece of data, the estimated processing time corresponding to each piece of data is obtained;

[0025] According to the corresponding relationship between the estimated processing time and the processing time level, the processing time level corresponding to each piece of data is determined.

[0026] In one embodiment, the plurality of second-type data sets corresponding to different processing time levels are used to obtain a plurality of data subsets, including:

[0027] Selecting one second-category data set from a plurality of second-category data sets corresponding to different processing time levels;

[0028] A single data subset is formed according to the selected second type of data set to obtain a plurality of data subsets.

[0029] In one embodiment, the method further comprises:

[0030] Obtain the measurement results of each data in each data subset;

[0031] According to the account book associated with each data, the measurement results of each data in different data subsets are summarized.

[0032] In a second aspect, the present application also provides a device for performing distributed metering on a data set, including:

[0033] A metering operator acquisition module is used to encapsulate each metering element to obtain the metering operator corresponding to each metering element;

[0034] The computing unit deployment module is used to select corresponding metering operators for combination, obtain each metering tool and deploy it on multiple computing nodes;

[0035] A data subset acquisition module is used to evenly divide the data set into multiple data subsets;

[0036] The data subset metering module is used to send the multiple data subsets to different computing nodes, so that each computing node measures the data subset according to a corresponding metering tool.

[0037] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the above method.

[0039] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to execute the above method.

[0040] The above-mentioned method, device, computer equipment, computer-readable storage medium and computer program product for distributed metering of data sets encapsulate each metering element to obtain the metering operator corresponding to each metering element; select the corresponding metering operator for combination to obtain each metering tool and deploy it on multiple computing nodes; divide the data set into multiple data subsets; send multiple data subsets to different computing nodes so that each computing node measures the data subset according to the corresponding metering tool. This application encapsulates each metering element to obtain the metering operator corresponding to each metering element, selects the corresponding metering operator for combination to obtain each metering tool, and reduces the coupling between metering elements; when a new metering tool needs to be generated, the corresponding metering operator can be selected according to the demand for combination to obtain a new metering tool, thereby improving the scalability of metering; the data set is evenly divided into multiple data subsets, and the multiple data subsets are sent to different computing nodes, so that the data processing volume of each computing node is the same, the computing resources are fully utilized, and the metering efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flowchart of a method for performing distributed metering on a data set in one embodiment;

[0043] Figure 2 It is a schematic diagram of a process of performing distributed metering of a total resource holding data set in one embodiment;

[0044] Figure 3 is a structural block diagram of an apparatus for performing distributed metering on a data set in one embodiment;

[0045] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The embodiment of the present application provides a method for distributed metering of a data set. The embodiment can be executed by a computer device. The computer device can encapsulate and process each metering element to obtain a metering operator corresponding to each metering element; can select corresponding metering operators for combination to obtain each metering tool and deploy it on multiple computing nodes; can evenly divide the data set into multiple data subsets; and send multiple data subsets to different computing nodes so that each computing node meters the data subset according to the corresponding metering tool. It can be understood that the computer device can be implemented through a server, can be implemented through a terminal, and can also be implemented through an interactive system between a terminal and a server. In this embodiment, the method includes Figure 1 The steps shown are:

[0048] Step S101, encapsulate each metering element to obtain a metering operator corresponding to each metering element.

[0049] Measurement elements refer to the basic factors or indicators used to quantify and describe an object or phenomenon in the field of measurement.

[0050] Each measurement element has different measurement methods. For example, interest rate interpolation can include a series of interpolation methods such as linear interpolation, left interpolation, and right interpolation. Each measurement element can be encapsulated into a tool class, and the measurement methods can be divided through enumeration.

[0051] You can use the domain-driven design pattern to determine the measurement elements of the field to which the data set belongs, and modularize and encapsulate the measurement elements to obtain the corresponding measurement operators. For example, according to the domain-driven design concept, if the field to which the data set belongs is determined to be the asset-liability data field, you can modularly encapsulate the seven measurement elements of interest rate, amortization type, holiday adjustment, interest interpolation, interest basis, compound interest method, and interest calculation to obtain the measurement operators corresponding to the seven measurement elements.

[0052] Step S102 , selecting corresponding metering operators for combination, obtaining each metering tool and deploying it on multiple computing nodes.

[0053] Different measurement tools (also called financial measurement tools) require different measurement operators. According to the type of measurement tool, the corresponding measurement operators of the measurement tool can be selected and combined to form a measurement tool. In addition, the measurement elements used in the measurement tool are shared, so the consistency of measurement is also guaranteed.

[0054] For example, for fixed-rate bond measurement tools, the measurement elements required for measuring fixed-rate bonds include holiday adjustment, interest calculation basis, compound interest method and interest calculation. These measurement elements have corresponding measurement operators. Through Flink's streaming operation, the holiday adjustment measurement operator, interest calculation basis measurement operator, compound interest method measurement operator and interest calculation measurement operator can be selected and combined to obtain the fixed-rate bond measurement tool.

[0055] If a new metering operator is added, all metering tools can use the new metering operator. For example, if a new interest rate interpolation left step interpolation metering operator is added, the corresponding non-maturity metering tool, fixed interest rate metering tool, floating interest rate metering tool and fixed interest bond metering tool can all use this metering operator.

[0056] After obtaining the metering tool, the metering tool can be deployed on multiple computing nodes (also referred to as metering nodes).

[0057] Step S103, dividing the data set into multiple data subsets on an even basis.

[0058] The data set can be evenly divided into multiple data subsets according to the computing power of the computing nodes.

[0059] Step S104: sending the multiple data subsets to different computing nodes, so that each computing node measures the data subset according to a corresponding measurement tool.

[0060] Multiple data subsets can be sent to different computing nodes, so that each computing node measures the data subset according to the corresponding measurement tool, thereby completing the distributed measurement operation of the data set.

[0061] In the above, each metering element is encapsulated to obtain the metering operator corresponding to each metering element, and the corresponding metering operators are selected for combination to obtain each metering tool, thereby reducing the coupling degree between the metering elements; when a new metering tool needs to be generated, the corresponding metering operators can be selected for combination according to the needs to obtain a new metering tool, thereby improving the scalability of the metering; the data set is evenly divided into multiple data subsets, and the multiple data subsets are sent to different computing nodes, so that the data processing volume of each computing node is the same, the computing resources are fully utilized, and the metering efficiency is improved.

[0062] In one of the embodiments, a data set is evenly divided into multiple data subsets, and the specific steps are as follows: obtain the total amount of the data set; obtain the processable data amount of the computing node based on the computing power of the computing node; and divide the data set evenly into multiple data subsets based on the total amount and the processable data amount.

[0063] In traditional distributed processing solutions, data sets are grouped according to account books, and after grouping, metering tasks are distributed according to account books, where account books are a collection of a type of data. However, there are cases where the data corresponding to account books are unevenly distributed. For example, there are 100 pieces of personal regular resource transfer data, but there is only one piece of data in the resource storage account of a financial institution. In this case, the metering efficiency (batch efficiency) depends on the execution efficiency of the computing node where the task with the most account book data is located.

[0064] In order to improve the metering efficiency, we can combine the characteristics of Flink streaming computing and not group the data sets according to the account books. Instead, we can evenly distribute the data sets on each computing node according to the size of the data. This can solve the problem of data offset corresponding to different account books.

[0065] The total amount of the data set can be obtained; according to the computing power of the computing node, the amount of data that can be processed by the computing node can be obtained; according to the total amount and the amount of data that can be processed, the data set can be evenly divided into multiple data subsets.

[0066] In this embodiment, the data set is evenly divided into multiple data subsets according to the total amount of the data set and the amount of data that can be processed by the computing nodes, so that the amount of data in the data subsets sent to different computing nodes is the same, and the execution efficiency of different computing nodes is relatively close, thereby improving the metering efficiency.

[0067] In one of the embodiments, a data set is evenly divided into multiple data subsets based on the total amount and the processable data amount, and the specific steps are as follows: a data average amount is obtained based on the total amount and the number of computing nodes; when the data average amount is less than the processable data amount, the data set is evenly divided into multiple data subsets based on the data average amount; when the data average amount is greater than the processable data amount, the data set is divided into multiple batches of data sets based on the processable data amount and the number of computing nodes, and each batch of data sets is divided into corresponding data subsets based on the processable data amount and the number of computing nodes.

[0068] For example, the total amount of the data set is 1 million data records, and according to the total amount of 1 million data records and the number of computing nodes being 5, the average amount of data can be obtained to be 200,000 data records.

[0069] If the processable data volume of each computing node is 210,000 data records, the average data volume is less than the processable data volume. At this time, the data set can be evenly divided into 5 data subsets based on the average data volume, and the data volume of each data subset is 200,000 data records.

[0070] If the processable data volume of each computing node is 160,000 data records, the average data volume is greater than the processable data volume. At this time, the data set can be divided into two batches of data sets according to the processable data volume and the number of computing nodes, where the data volume of the first batch of data is the sum of the processable data volume and the number of computing nodes, that is, the data volume of the first batch of data sets is 800,000 data records, and the data volume of the second batch of data sets is the result of subtracting the data volume of the first batch of data sets from the total amount of the data set. When the total amount of the data set is 1 million data records and the data volume of the first batch of data sets is 800,000 data records, the data volume of the second batch of data sets is 200,000 data records; after the data set is divided, the first batch of data sets and the first batch of data sets can be divided into corresponding data subsets according to the processable data volume. For the first batch of data sets, it can be evenly divided into 5 data subsets, each with a data volume of 160,000 data records. For the second batch of data sets, it can be evenly divided into 5 data subsets, each with a data volume of 40,000 data records.

[0071] In this embodiment, the data average component is obtained according to the total amount and the number of computing nodes, and the data set is evenly divided into multiple data subsets according to the size relationship between the data average component and the processable data volume, so that the data volume of the data subsets sent to different computing nodes is the same, and the execution efficiency of different computing nodes is relatively close, thereby improving the measurement efficiency.

[0072] In one of the embodiments, a data set is evenly divided into multiple data subsets according to the data average component, and the specific steps are as follows: obtaining the processing time level of each data in the data set; dividing the data set according to the processing time level of each data in the data set to obtain first-category data sets corresponding to different processing time levels; evenly dividing the first-category data set corresponding to each processing time level according to the number of computing nodes to obtain corresponding multiple second-category data sets; obtaining multiple data subsets according to the multiple second-category data sets corresponding to different processing time levels.

[0073] When the amount of data in data subsets sent to different computing nodes is the same, if the tasks corresponding to the data are different and the processing time difference of different tasks is relatively large, there is a situation where the data corresponding to the tasks with long processing time are allocated to the same computing node, resulting in lower execution efficiency of the computing node, thereby resulting in lower measurement efficiency. To avoid this situation, the data set can be evenly divided into multiple data subsets according to the processing time level corresponding to the data, so that the processing time difference of the data subsets sent to different computing nodes is smaller.

[0074] Exemplarily, the processing time level corresponding to each of the 1 million data in the data set can be obtained, where the processing time level can have three levels, namely, long time level, medium and long time level, and short time level; according to the processing time level of each data in the data set, the data set is divided to obtain a first category of data sets corresponding to the long time level, with a data volume of 200,000 data sets, a first category of data sets corresponding to the medium and long time level, with a data volume of 400,000 data sets, and a first category of data sets corresponding to the short time level, with a data volume of 400,000 data sets.

[0075] According to the number of computing nodes 5, the first-category data set corresponding to each processing time level can be evenly divided to obtain multiple corresponding second-category data sets. Specifically, the first-category data set corresponding to the long-time level can be evenly divided into five corresponding second-category data sets, and the data volume of each second-category data set is 40,000 data records; the first-category data set corresponding to the medium- and long-time level can be evenly divided into five corresponding second-category data sets, and the data volume of each second-category data set is 80,000 data records; the first-category data set corresponding to the short-time level can be evenly divided into five corresponding second-category data sets, and the data volume of each second-category data set is 80,000 data records; and multiple data subsets can be obtained according to the multiple second-category data sets corresponding to different processing time levels.

[0076] In this embodiment, the data set is divided according to the processing time level of each data in the data set to obtain the first type of data set corresponding to different processing time levels; the first type of data set corresponding to each processing time level is evenly divided according to the number of computing nodes to obtain corresponding multiple second type of data sets, so as to obtain multiple data subsets, so that the processing time difference of the data subsets sent to different computing nodes is small, thereby improving the measurement efficiency.

[0077] In one of the embodiments, the processing time level of each data in the data set is obtained, and the specific steps are as follows: according to the measurement tools required for measuring each data, the estimated processing time corresponding to each data is obtained; according to the corresponding relationship between the estimated processing time and the processing time level, the processing time level corresponding to each data is determined.

[0078] Different tasks correspond to different data, so different measurement tools are needed to measure the data. Each measurement tool can have a corresponding estimated processing time. According to the estimated processing time corresponding to the measurement tool required for the task, the estimated processing time of the task can be obtained. Therefore, according to the measurement tool required for data measurement, the estimated processing time corresponding to each piece of data can be obtained.

[0079] The corresponding relationship between the estimated processing time and the processing time level can be preset according to actual conditions.

[0080] According to the corresponding relationship between the estimated processing time and the processing time level, the processing time level corresponding to each piece of data is determined.

[0081] In this embodiment, the estimated processing time corresponding to each piece of data is obtained according to the measurement tools required for measuring each piece of data; and a relatively accurate processing time level for each piece of data can be obtained according to the corresponding relationship between the estimated processing time and the processing time level.

[0082] In one of the embodiments, multiple data subsets are obtained based on multiple second-category data sets corresponding to different processing time levels, and the specific steps are as follows: select one second-category data set from the multiple second-category data sets corresponding to different processing time levels; form a single data subset based on the selected second-category data set to obtain multiple data subsets.

[0083] Exemplarily, the long time level has five corresponding second-category data sets, namely A, B, C, D and E, and the data volume of each of the A, B, C, D and E data sets is 40,000 data records; the medium and long time level has five corresponding second-category data sets, namely F, G, H, I and J, and the data volume of each of the F, G, H, I and J data sets is 80,000 data records; the short time level has five corresponding second-category data sets, namely K, L, M, N and O, and the data volume of each of the K, L, M, N and O data sets is 80,000 data records.

[0084] You can select any one second category data set, such as A, from the five second category data sets A, B, C, D and E corresponding to the long time level, respectively; you can select any one second category data set, such as H, from the five second category data sets F, G, H, I and J corresponding to the medium and long time level; you can select any one second category data set, such as O, from the five second category data sets K, L, M, N and O corresponding to the short time level; based on the selected second category data sets A, H and O, a single data subset Q is formed, and the data volume of the Q data set is 200,000 data records. According to the above method for obtaining the data subset Q, multiple data subsets can be obtained.

[0085] In this embodiment, one second type of data set is selected from multiple second type of data sets corresponding to different processing time levels; a single data subset is formed based on the selected second type of data set to obtain multiple data subsets, so that the data amounts of data subsets sent to different computing nodes are the same and the processing time differences are small, thereby improving measurement efficiency.

[0086] In one of the embodiments, the method provided in the present application also includes: obtaining the measurement results of each data in each data subset; and summarizing the measurement results of each data in different data subsets according to the account book associated with each data.

[0087] After each computing node measures the data subset according to the corresponding measurement tool, the measurement result of each data in each data subset can be obtained; each data in the data subset has an associated account book, and the measurement results of each data in different data subsets can be summarized based on the account book associated with each data.

[0088] In this embodiment, the measurement results of each piece of data in different data subsets are summarized according to the account book associated with each piece of data, and the corresponding summary can be obtained according to business requirements.

[0089] In order to better understand the above method, an application example of the method for performing distributed metering on a data set in the present application is described in detail below.

[0090] The total resource holding engine can be used to perform distributed metering on the total resource holding data set. There are many frameworks that can be combined with the total resource holding engine to achieve distributed metering, among which the most commonly used ones are the spark framework, akka framework, hive framework, and flink framework. Among them, the flink framework is more suitable for the total resource holding engine, because the calculation of the total resource holding engine is relatively complex, the hive framework is not suitable, the spark framework is only suitable for the big data platform, and the memory consumption is relatively large, and it is heavily dependent on the big data platform. The akka framework is more suitable as a framework for task distribution and management, and its corresponding data offset processing capability is relatively weak. The flink framework can perform streaming processing, and the resource requirements are not very strict. The flink framework can also load multiple types of data and does not rely solely on a certain database type. In addition, flink itself also supports distributed deployment, which can meet the needs of the total resource holding engine research and development.

[0091] The engine that holds the total amount of resources can use the flink architecture to achieve integrated stream and batch metering; it can make full use of server resources and perform distributed deployment based on the provided servers to achieve distributed metering; it can split the metering steps to perform streaming processing on data processing and metering.

[0092] The total resource holdings engine can load the total resource holdings data set through the data stream (interface DataStream API) of the Flink framework. Through the interface, not only can the total resource holdings data of the relational database be loaded through the Structured Query Language (SQL), but it also supports loading the total resource holdings data of large databases, files, and message queues.

[0093] According to the method provided in this application, the total resource data set can be distributedly measured. The specific steps are as follows: Figure 2 shown.

[0094] Step S201, physical resource flow measurement: according to the domain-driven design concept, each measurement element in the field of total resource holdings data is encapsulated and processed to obtain the measurement operator corresponding to each measurement element, and the corresponding measurement operator is selected for combination to obtain the physical resource flow measurement tool and the single market value duration measurement tool and deploy them on multiple computing nodes. The physical resource flow measurement tool can be called a conversion operator; the loaded total resource holdings data set is divided into multiple total resource holdings data subsets, and the multiple total resource holdings data subsets are sent to different computing nodes, so that each computing node measures each data in the total resource holdings data subset according to the physical resource flow measurement tool. For example, for a personal 20-year mortgage, the amount and time of each future principal and interest repayment can be calculated according to the business term, repayment frequency, repayment method, and interest rate factors, and a resource flow is formed according to time, which is called a physical resource flow.

[0095] Step S202, single market value duration measurement: the single physical resource flow data calculated by the single data in the total resource holding data subset is combined with the corresponding single market value duration measurement tool to calculate the accrued interest, market value and duration indicators of each data in the total resource holding data subset.

[0096] Step S203, after each computing node measures the total resource data subset according to the physical resource flow metering tool and the market value duration metering tool, the metering result can be sent to the summary node through streaming processing, and the summary node can be used as a task node of the flink framework; the corresponding metering operators can be selected for combination to obtain the initial resource total amount summary metering tool, the time step summary metering tool and the account book market value duration summary metering tool and deploy them on the summary node, and the initial resource total amount summary metering tool, the time step summary metering tool and the account book market value duration summary metering tool can be called aggregation operator. The initial resource total amount summary, the time step summary and the account book market value duration summary can be batch processed, that is, the initial resource total amount summary, the time step summary and the account book market value duration summary can be processed in parallel.

[0097] Summary of initial total resources: Obtain the measurement results of a single data item in each subset of total resources held data, and summarize the measurement results of each data item in different subsets of total resources held data according to the account book associated with the single data item to obtain the initial total resources.

[0098] Time step summary: Get the time step plan from the cache according to the scenario number. The time step is a collection of time periods divided according to business needs. For example, the liquidity time step is generally divided into 1 day, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, one month, three months, half a year, one year, two years, three years, five years, ten years, twenty years, and more than twenty years.

[0099] For a certain time step plan, for example, the total initial resource amount for the first half of 2024, the total initial resource amount can be summarized according to the corresponding time step interval into which the time step and the physical resource flow fall.

[0100] If multiple time step schemes are configured, they are processed cyclically.

[0101] Account book market value duration summary: The accrued interest, yield to maturity, market value, Maclean duration, and convexity index of a single data calculated in the single market value duration measurement in step S302 can be weighted and summarized according to the account book and the initial total amount of resources for each data.

[0102] After the metering tools in the above steps complete their work, the resource total amount engine can create an output operator sink, where sinks can be divided into KafkaSink and MysqlSink. You can customize the sink. Depending on the corresponding data type, you can create sinks separately. Generally, MysqlSink is the main one.

[0103] In this embodiment, through flink streaming processing combined with the reference-driven design concept, business metering tools can be quickly assembled, and metering tool types can be connected in series as the main business line to form a complete computing chain, realizing engine process and componentization, and solving the scalability and efficiency problems of complex operations in large quantities. In addition, when a new metering operator is added, all metering tools can use the metering operator, and all business processes can use the metering operator by applying the metering tool; when a metering operator has a metering logic error, the metering operator can be modified, and all businesses will be corrected after the modification to maintain uniformity.

[0104] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a device for distributed metering of data sets for implementing the above-mentioned method for distributed metering of data sets. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for distributed metering of data sets provided below can refer to the limitations of the method for distributed metering of data sets above, and will not be repeated here.

[0106] In an exemplary embodiment, Figure 3 As shown, a device for distributed metering of a data set is provided, wherein:

[0107] The metering operator acquisition module 301 is used to encapsulate each metering element to obtain the metering operator corresponding to each metering element;

[0108] A computing unit deployment module 302 is used to select corresponding measurement operators for combination, obtain each measurement tool and deploy it on multiple computing nodes;

[0109] A data subset acquisition module 303 is used to evenly divide the data set into multiple data subsets;

[0110] The data subset metering module 304 is used to send the multiple data subsets to different computing nodes, so that each computing node meters the data subset according to a corresponding metering tool.

[0111] In one of the embodiments, the data subset acquisition module 303 is further used to: obtain the total amount of the data set; obtain the amount of data that can be processed by the computing node based on the computing power of the computing node; and divide the data set into multiple data subsets on average based on the total amount and the amount of data that can be processed.

[0112] In one of the embodiments, the data subset acquisition module 303 is further used to: obtain a data average component according to the total amount and the number of computing nodes; when the data average component is smaller than the processable data amount, evenly divide the data set into multiple data subsets according to the data average component; when the data average component is larger than the processable data amount, divide the data set into multiple batches of data sets according to the processable data amount and the number of computing nodes, and divide each batch of data sets into corresponding data subsets according to the processable data amount and the number of computing nodes.

[0113] In one of the embodiments, the data subset acquisition module 303 is further used to: obtain the processing time level of each data in the data set; divide the data set according to the processing time level of each data in the data set to obtain first-category data sets corresponding to different processing time levels; divide the first-category data set corresponding to each processing time level evenly according to the number of computing nodes to obtain corresponding multiple second-category data sets; obtain multiple data subsets according to the multiple second-category data sets corresponding to different processing time levels.

[0114] In one embodiment, the data subset acquisition module 303 is also used to: obtain the estimated processing time corresponding to each data according to the measurement tools required for measuring each data; and determine the processing time level corresponding to each data according to the corresponding relationship between the estimated processing time and the processing time level.

[0115] In one embodiment, the data subset acquisition module 303 is further used to: select one second type of data set from multiple second type of data sets corresponding to different processing time levels; form a single data subset based on the selected second type of data set to obtain multiple data subsets.

[0116] In one of the embodiments, the device further includes a metering result aggregation module, which is used to: obtain the metering result of each data in each data subset; and aggregate the metering results of each data in different data subsets according to the account book associated with each data.

[0117] Each module in the above-mentioned apparatus for distributed metering of data sets can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of a method for distributed metering of a data set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for distributed metering of a data set is implemented.

[0119] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0121] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0122] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0126] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for distributed measurement of a data set, characterized in that: The method comprises: Encapsulating each metering element to obtain a metering operator corresponding to each metering element; Select corresponding metering operators and combine them to obtain each metering tool and deploy it on multiple computing nodes; Divide the data set into multiple data subsets evenly; The multiple data subsets are sent to different computing nodes, so that each computing node measures the data subset according to a corresponding measurement tool.

2. The method according to claim 1, characterized in that: The data set is evenly divided into multiple data subsets, including: Obtaining the total amount of the data set; According to the computing capacity of the computing node, the amount of data that can be processed by the computing node is obtained; According to the total amount and the processable data amount, the data set is evenly divided into a plurality of data subsets.

3. The method according to claim 2, characterized in that The step of evenly dividing the data set into a plurality of data subsets according to the total amount and the processable data amount includes: Obtaining a data average according to the total amount and the number of computing nodes; When the data average amount is smaller than the processable data amount, dividing the data set into a plurality of data subsets on average according to the data average amount; When the data average component is greater than the processable data volume, the data set is divided into multiple batches of data sets according to the processable data volume and the number of computing nodes, and each batch of data sets is divided into corresponding data subsets according to the processable data volume and the number of computing nodes.

4. The method according to claim 3, characterized in that The step of evenly dividing the data set into a plurality of data subsets according to the data average component comprises: Obtaining the processing time level of each data in the data set; According to the processing time level of each data in the data set, the data set is divided to obtain a first type of data set corresponding to different processing time levels; According to the number of computing nodes, the first type of data set corresponding to each processing time level is evenly divided to obtain a corresponding plurality of second type of data sets; A plurality of data subsets are obtained according to the plurality of second-category data sets corresponding to different processing time levels.

5. The method according to claim 4, characterized in that The obtaining of the processing time level of each piece of data in the data set includes: According to the measurement tools required for measuring each piece of data, the estimated processing time corresponding to each piece of data is obtained; According to the corresponding relationship between the estimated processing time and the processing time level, the processing time level corresponding to each piece of data is determined.

6. The method according to claim 4, characterized in that The plurality of second-category data sets corresponding to different processing time levels are used to obtain a plurality of data subsets, including: Selecting one second-category data set from a plurality of second-category data sets corresponding to different processing time levels; A single data subset is formed according to the selected second type of data set to obtain a plurality of data subsets.

7. The method according to claim 1, characterized in that The method further comprises: Obtain the measurement results of each data in each data subset; According to the account book associated with each data, the measurement results of each data in different data subsets are summarized.

8. A device for distributed metering of a data set, characterized in that: The device comprises: A metering operator acquisition module is used to encapsulate each metering element to obtain the metering operator corresponding to each metering element; The computing unit deployment module is used to select corresponding metering operators for combination, obtain each metering tool and deploy it on multiple computing nodes; A data subset acquisition module is used to evenly divide the data set into multiple data subsets; The data subset metering module is used to send the multiple data subsets to different computing nodes, so that each computing node measures the data subset according to a corresponding metering tool.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images