A multi-dimensional data pricing method and system for a trusted data space

Abstract

The present application relates to the technical field of data processing and charging, and provides a multi-dimension data pricing method and system for trusted data space, comprising: splitting a federal query into multiple sub-queries and respectively issuing to multiple data subsystems for execution; determining multiple cost anchor points according to cumulative variance of cardinality estimation in an execution plan, based on which the execution plan is divided into multiple settlement segments; for each settlement segment, obtaining runtime statistical information through a data source connector, based on which the segment value of the settlement segment is determined; at the cost anchor point, when the sum of the segment value of the completed settlement segment and the estimated value of the remaining settlement segment is greater than the upper limit of the value, performing a downgrade rewriting on the sub-query corresponding to the remaining settlement segment, and determining the segment value of the remaining settlement segment according to the actual resource consumption after the downgrade rewriting; and sequentially aggregating the segment values of the settlement segments to obtain the unified value of the federal query.

Description

Technical Field

[0001] This invention relates to the field of data processing and billing technology, specifically to a multi-dimensional data pricing method and system for trusted data spaces. Background Technology

[0002] A trusted data space is an infrastructure for cross-entity data sharing and value transfer, emphasizing that data should not leave its domain, be usable but not visible, and have controllable and measurable uses. Data weaving technology, through unified metadata and catalogs, enables the logical integration and federated queries of heterogeneous data sources (such as relational databases, big data warehouses, search engines, and object storage). In a trusted data space scenario, a federated query involves not only the hardware resource consumption of the data provider's data production subsystem due to computational pushdown, but also the weaving service and privacy computation executed by the data weaving platform, as well as the value of the data itself. Therefore, it needs to be measured and aggregated for pricing across multiple dimensions.

[0003] Existing data billing methods mainly fall into three categories: subscription-based billing, billing by query or job, and pricing based on the overall value of the data asset. In addition, hardware services are typically priced based on the usage of resources such as CPU, memory, and network, while software services include professional business value.

[0004] However, in existing pricing practices, the estimation error of execution cost accumulates and amplifies along the execution plan. For example, the execution plan of a federated query consists of multiple join operators. If each join operator has an error, the errors will accumulate, and the data later in the process will become less reliable. If verification is only performed at the end of the billing process, the accumulation of errors cannot be detected.

[0005] Furthermore, during query execution, the resource consumption within the query is inseparable among the operators. Typically, multiple query operators share the scanning phase and data buffer during execution, and often proceed in parallel in a pipelined manner. If the query has resource constraints, the resource consumption already incurred cannot be clearly divided. Resource consumption is usually calculated using a preset ratio and overall usage. For trusted data spaces across multiple entities, it's impossible to determine the accuracy of the pricing; either the pricing result is accepted, or a dispute arises. Dispute resolution usually involves re-execution and cost recalculation, but this results in even greater resource consumption.

[0006] For query tasks, resource consumption is usually reduced to achieve optimization. However, most resource optimization methods require sacrificing the accuracy of the results. Therefore, existing pricing technologies all assume that the results are accurate and do not consider the situation where the accuracy of the results is reduced. Summary of the Invention

[0007] This application proposes a multi-dimensional data pricing method and system for trusted data spaces, which solves the technical problems in the prior art where the pricing granularity is decoupled from the query execution progress and the inability to accurately segment billing when the query is interrupted or downgraded.

[0008] To achieve the above objectives, this application provides the following technical solution: Firstly, this application proposes a multi-dimensional data valuation method for trusted data spaces, applied to a data weaving platform, including the following steps: Receive federal queries and determine the execution plan; According to the execution plan, the federated query is split into multiple subqueries, which are then distributed to multiple data subsystems for execution. Based on the cumulative variance of the cardinality estimate in the execution plan, multiple cost anchors are determined in the execution plan, and the execution plan is divided into multiple settlement segments using the cost anchors; For each settlement segment, obtain the runtime statistics after the execution of each subquery within the corresponding settlement segment, and determine the segment settlement value of the corresponding settlement segment; At the cost anchor point, when the sum of the segment value of the completed settlement segment and the estimated value of the remaining settlement segment is greater than the preset value upper limit, the subquery corresponding to the remaining settlement segment is downgraded and rewritten before being sent for execution, and the segment value of the remaining settlement segment is determined according to the actual resource consumption after the downgraded rewriting. The segmented values ​​of each settlement segment are aggregated according to the order of the cost anchor points to generate the unified value of the federated query.

[0009] In the above scheme, when executing the query task, the reliability of the pricing result of the query object and the estimated variance accumulated along the plan are considered.

[0010] In conjunction with the first aspect, determining multiple cost anchors in the execution plan based on the cumulative variance of the cardinality estimate in the execution plan includes: Based on the execution order of the execution plan, the variance of the cardinality estimate of the corresponding connection operator is accumulated at each connection operator to determine the cumulative variance; When the cumulative variance exceeds a preset variance threshold, a cost anchor is set after the corresponding connection operator, and the cumulative variance is set to zero before continuing to accumulate; wherein, The total number of cost anchors set shall not exceed the preset maximum number of anchors.

[0011] In conjunction with the first aspect, the variance threshold is determined based on a preset upper bound for the probability of default and a preset error ratio threshold, such that the probability that the relative amount by which the estimated value of any settlement segment deviates from its expected value exceeds the error ratio threshold is not greater than the upper bound for the probability of default.

[0012] In conjunction with the first aspect, determining the segment value of the corresponding settlement segment includes: Based on the runtime statistics, the actual resource consumption of each subquery is determined, and the actual resource consumption is mapped to a unified resource cost based on the resource correction factor corresponding to each data subsystem. The value of the data itself is determined based on the unit price of the data objects involved in the settlement segment and the scale of the data participating in the pricing. The platform service cost is determined based on the scale of data processed by the data weaving platform in performing weaving services and privacy computing, and the corresponding service unit price. The unified resource cost, the data's intrinsic value, and the platform service cost are summed to determine the segment value for the corresponding settlement segment; wherein... The resource correction factor is determined based on the resource processing capabilities of each data subsystem and is updated based on the runtime statistics.

[0013] In conjunction with the first aspect, according to the execution plan, the federated query is split into multiple subqueries, including: Based on the execution plan, a unique subquery identifier is generated for each subquery. Based on the operator types involved in the federated query, the resource correction factors corresponding to each data subsystem, and the data unit price of each data object, the pricing caliber identifier of the federated query is generated. The pricing caliber identifier is bound to the sub-query identifier, and the bound sub-query identifier and the pricing caliber identifier are appended to the query statement as comments, and then sent to multiple data subsystems for execution.

[0014] In conjunction with the first aspect, the step of performing a downgrade rewrite on the subquery corresponding to the remaining settlement segment and then issuing it for execution includes: Among a set of preset downgrade levels, the downgrade level that minimizes the predicted value of the remaining settlement segment from the upper limit of the value and has the lowest downgrade degree is selected, and the downgrade rewriting is performed according to the selected downgrade level; wherein, Each of the aforementioned downgrade levels corresponds to a sampling rate and a maximum pushdown data size.

[0015] In conjunction with the first aspect, the step of performing downgrade rewriting according to the selected downgrade level includes: Replace the exact aggregation operator in the subquery corresponding to the remaining settlement segment with an approximate aggregation operator based on stratified sampling; wherein, The sampling rate of the approximate aggregation operator is equal to the sampling rate corresponding to the selected downgrade level. Set the upper limit of the pushdown data size of the subquery corresponding to the remaining settlement segment to the upper limit of the pushdown data size corresponding to the selected degradation level; wherein, The downgrade rewriting maintains the same result data pattern as the federated query.

[0016] In conjunction with the first aspect, the method further includes: When a settlement segment is completed, an incremental settlement voucher for the corresponding settlement segment is generated; where, The incremental settlement voucher includes a segment number, the segment value of the settlement segment, the location identifier of the cost anchor point at the start and end of the settlement segment, and a summary of the incremental settlement voucher of the previous settlement segment. Execute the generation of incremental settlement vouchers for the corresponding settlement segment, including: Based on the segment number, the summary in each incremental settlement voucher is sequentially compared with the summary of the incremental settlement voucher in the previous settlement segment. If the verification passes, the segment value in the corresponding incremental settlement voucher is accumulated. The proportion of each data provider's actual resource consumption within a settlement segment to the sum of the actual resource consumption of all data subsystems within the corresponding settlement segment is used to allocate the unified resource cost of the corresponding settlement segment to each data provider.

[0017] In conjunction with the first aspect, the method further includes: Based on the unified valuation, determine whether there are any settlement segment objections; wherein... When an objection arises to any settlement segment in the unified value of the federated query, the boundary of the subquery corresponding to that settlement segment is determined based on the position identifier of the cost anchor point at the start and end of the disputed settlement segment. The subquery corresponding to that settlement segment is then re-executed to determine the recalculated value of the corresponding settlement segment. The unified value is obtained by summing the segment values ​​of multiple settlement segments in the order of the settlement segments; The settlement segment is divided by the cost anchor point set in the execution plan of the federated query based on the cumulative variance estimated by the cardinality. The segment value of each settlement segment is fixed in the incremental settlement voucher of the corresponding settlement segment; The incremental settlement voucher includes a segment number, a segment value, and the location identifier of the cost anchor point at the start and end of the settlement segment; The recalculated value is compared with the segment value of the settlement segment in the incremental settlement voucher. When the deviation between the two is greater than a preset deviation threshold, the segment value of the corresponding settlement segment is replaced with the recalculated value, and the unified value is corrected. Specifically, when the objection is raised by the data provider and points to the uniform resource cost allocated to the data provider, only the subquery corresponding to the data provider in the settlement segment is re-executed, and the uniform resource cost allocated to the data provider is adjusted according to the recalculation result.

[0018] Secondly, this application proposes a multi-dimensional data pricing system for trusted data spaces, including: The query receiving module receives federated queries and determines the execution plan. The query splitting module, based on the execution plan, splits the federated query into multiple subqueries, which are then distributed to multiple data subsystems for execution. The settlement segment division module determines multiple cost anchors in the execution plan based on the cumulative variance of the cardinality estimate in the execution plan, and divides the execution plan into multiple settlement segments using the cost anchors; The segment value determination module obtains runtime statistics after the execution of each sub-query within the corresponding settlement segment for each settlement segment, and determines the segment value of the corresponding settlement segment. The budget control module, at the cost anchor point, when the sum of the segment cost value of the completed settlement segment and the estimated cost value of the remaining settlement segment is greater than the preset cost value upper limit, performs a downgrade rewrite on the subquery corresponding to the remaining settlement segment and sends it down for execution, and determines the segment cost value of the remaining settlement segment according to the actual resource consumption after the downgrade rewrite; The aggregation module aggregates the segment values ​​of each settlement segment according to the order of the cost anchor points, and generates the unified value of the federated query.

[0019] The beneficial effects of this invention are as follows: This application compares the sum of the segment values ​​of completed settlement segments with the sum of the estimated values ​​of remaining settlement segments at each cost anchor point before the execution of the remaining subquery, against the upper limit of the estimated value. Execution continues as planned only if the sum is not greater than the upper limit, ensuring that the unified estimated value will not exceed the upper limit. This property is guaranteed by the execution process itself and does not rely on statistical information, environmental assumptions, or empirical parameters. The estimated value of the settlement segment has a controllable upper bound on the error probability, making the anchor point no longer an arbitrarily selected segmentation point, but a control point for the reliability of budget verification. Although the downgrade rewriting proposed in this application reduces the accuracy of the results, it reduces the value of the data itself according to the downgrade ratio, and this reduction does not affect the unified resource cost or platform service cost. This allows resource consumption control and pricing accuracy to be achieved simultaneously under the same mechanism, and the conflict between reducing accuracy and accurate pricing is resolved through the reduction rule. Since the segment value is fixed segment by segment with the incremental settlement voucher, a dispute over any settlement segment only requires re-executing the subquery corresponding to that segment, resulting in a constructive lower bound improvement in the amount of data involved in dispute recalculation. In addition, in this application, each incremental settlement voucher carries a summary of the previous voucher. Any modification to the historical segment billing value will change the voucher summary, thereby causing the summary comparison of all subsequent vouchers to fail, thus making the segment billing results tamper-proof.

[0020] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0023] In the attached diagram: Figure 1 This is a flowchart illustrating a multi-dimensional data pricing method for a trusted data space, as described in an embodiment of this application. Figure 2 This is a system composition diagram of a multi-dimensional data pricing system for a trusted data space according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of setting cost anchor points and determining variance thresholds in the embodiments of this application; Figure 4 This is a schematic diagram showing the division of the settlement section and the location of the budget verification in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the value composition of the intermediate calculation in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the binding and annotation distribution of pricing caliber identifiers in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the process of selecting and rewriting the downgrade level in the embodiments of this application; Figure 8 This is a schematic diagram of the downgrade discount process in the embodiments of this application; Figure 9 This is a schematic diagram of the incremental settlement voucher chain in the embodiments of this application; Figure 10 This is a schematic diagram of partial recalculation of disputed segments and cross-entity apportionment arbitration in an embodiment of this application. Detailed Implementation

[0024] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1

[0025] like Figure 1As shown, this application proposes a multi-dimensional data valuation method for trusted data spaces, applied to a data weaving platform, including the following steps: Step S1: Receive federated queries and determine the execution plan; Specifically, a federated query is a query request initiated by the data consumer of the data weaving platform. Then, by querying the source of the data, a specific query plan is determined, that is, the query tasks that different data subsystems need to perform.

[0026] Step S2: According to the execution plan, the federated query is split into multiple subqueries and distributed to multiple data subsystems for execution. Specifically, the query task of a federated query will involve different types of data, so different data subsystems need to execute it. Therefore, the federated query is first split into multiple subqueries based on the data source, and then distributed to the corresponding data subsystems for execution.

[0027] Step S3: Based on the cumulative variance of the cardinality estimate in the execution plan, determine multiple cost anchors in the execution plan, and divide the execution plan into multiple settlement segments using the cost anchors; Specifically, during the query phase, the platform generates an execution plan for the federated query before execution begins. This execution plan includes several cost anchors based on the cumulative variance estimated by the cardinality. Query execution proceeds in units of settlement segments. A settlement segment is the set of operators and subqueries contained between two adjacent cost anchors in the federated query's execution plan. When the execution plan has k cost anchors, it is divided into k+1 settlement segments. Each settlement segment corresponds to one or more subqueries during execution, and the set of subqueries corresponding to each settlement segment is fixed. Cost anchors are location markers in the execution plan used to divide settlement segments. Their setting condition is that at each location marker, the cumulative variance of the cardinality estimates of the connecting operators is greater than a preset variance threshold.

[0028] Step S4: For each settlement segment, obtain the runtime statistics after the execution of each subquery within the corresponding settlement segment, and determine the segment settlement value of the corresponding settlement segment; Specifically, after each settlement segment is completed, the platform collects the runtime statistics of each subquery within that segment through the unified interface of each data source connector, calculates the segment's settlement value based on this, and performs a budget check at the cost anchor point before entering the next settlement segment.

[0029] Step S5: At the cost anchor point, when the sum of the segment value of the completed settlement segment and the estimated value of the remaining settlement segment is greater than the preset value upper limit, the subquery corresponding to the remaining settlement segment is downgraded and rewritten before being sent for execution, and the segment value of the remaining settlement segment is determined according to the actual resource consumption after the downgraded rewriting. Specifically, if the verification passes, the process continues as planned; if the verification fails, the lowest degradation level among multiple degradation tiers is selected, and the remaining subqueries are rewritten and deployed after degradation. The estimated cost is the cost of the settlement segment before execution, calculated in advance based on the estimated cost of the subquery corresponding to the settlement segment, the data unit price of the data objects involved in the settlement segment, and the data size. The segment cost is the cost calculated after the settlement segment is executed, based on the actual resource consumption of each subquery and the actual data size involved in the pricing. The maximum cost is the maximum value allowed for this federated query. The maximum cost is specified by the data consumer when submitting the federated query; or, it is estimated by the data weaving platform based on the operator types involved in the federated query, the resource correction factors corresponding to each data subsystem, and the data unit price of each data object, and then provided to the data consumer for confirmation. The degradation level characterizes the degree of impact of the degradation rewriting on the accuracy of the results. The degree of degradation is measured as follows: taking the execution without degradation rewriting as the benchmark, the order of the selected degradation level in the preset degradation level sequence is taken, and the later the order, the higher the degree of degradation; where the order is negatively correlated with the sampling rate corresponding to each degradation level, that is, the lower the sampling rate of the degradation level, the higher the degree of degradation.

[0030] Step S6: Summarize the segment values ​​of each settlement segment according to the order of the cost anchor points to generate the unified value of the federated query.

[0031] Specifically, after all settlement segments are completed, the value of each segment is summarized in the order of the anchor points, which is the unified value of the federated query, and then allocated among the data providers according to the proportion of resource consumption.

[0032] Through the above steps, the cost accounting granularity of the federal query is kept in sync with the execution progress. Regardless of whether the query ends normally, is downgraded due to exceeding limits, or is terminated for other reasons, the completed part has a definite segmented cost record, and the summary result is more accurate. Example 2

[0033] like Figure 3 As shown, in one specific embodiment, multiple cost anchors in the execution plan are determined based on the cumulative variance of the cardinality estimate in the execution plan, including: Specifically, the location of the cost anchor point is used to determine the boundary of the settlement segment, and thus the reliability of the budget verification.

[0034] Based on the execution order of the execution plan, the variance of the cardinality estimate of the corresponding join operator is accumulated at each join operator to determine the cumulative variance; In one specific embodiment, the federated query determines an execution plan based on the query information, and the cardinality estimate variance of the i-th join operator in the execution plan is... The platform accumulates the operators one by one in the execution order: Obtain the cumulative variance ; When the cumulative variance exceeds a preset variance threshold, a cost anchor is set after the corresponding connection operator, and the cumulative variance is set to zero before continuing to accumulate; wherein, The total number of cost anchors set shall not exceed the preset maximum number of anchors.

[0035] In this application, when the cumulative variance is greater than the variance threshold, it indicates that the estimated uncertainty since the previous anchor point has accumulated to the point that the probability that the relative error of the estimated value of this segment exceeds the preset error proportion threshold will exceed the preset upper bound of the default probability. At this time, a cost anchor point is set after the connection operator, and the cumulative variance is set to zero, so that the uncertainty is remeasured from this anchor point. At this time, the anchor point is no longer an arbitrarily selected segment point, but a control point for the reliability of budget verification. The probability that the relative amount of the estimated value deviating from the true value at any anchor point exceeds the preset error proportion threshold is limited to within the preset upper bound of the default probability.

[0036] Based on the above embodiments, it can be determined that: the variance threshold is determined according to the preset upper bound of the default probability and the preset error ratio threshold, so that the probability that the relative amount of the estimated value of any settlement segment deviates from its expected value by more than the error ratio threshold is not greater than the upper bound of the default probability.

[0037] In a specific implementation, to avoid an increase in the management and storage costs of incremental settlement vouchers due to an excessive number of settlement segments, the total number of anchor points is limited to a preset upper limit. The upper limit of the number of anchor points can be determined based on the storage costs of incremental settlement vouchers within the validity period of the pricing commitment, and is determined based on the fact that the sum of the upper limit of the number of vouchers and the field length of a single voucher does not exceed a preset storage quota. Example 3

[0038] like Figure 5 As shown, in one embodiment of this application, determining the segment value of the corresponding settlement segment includes: Based on the runtime statistics, the actual resource consumption of each subquery is determined, and the actual resource consumption is mapped to a unified resource cost based on the resource correction factor corresponding to each data subsystem. Specifically, runtime statistics refer to the statistics returned by the data subsystem after executing a subquery, reflecting the actual execution status of the subquery, including at least one of the following: actual scanned data volume, actual returned data volume, actual data volume involved in calculation, and central processing unit execution time.

[0039] The value of the data itself is determined based on the unit price of the data objects involved in the settlement segment and the scale of the data participating in the pricing. The platform service cost is determined based on the scale of data processed by the data weaving platform in performing weaving services and privacy computing, and the corresponding service unit price. The unified resource cost, the value of the data itself, and the platform service cost are summed to determine the segment value of the corresponding settlement segment; The resource correction factor is determined based on the resource processing capabilities of each data subsystem and is updated based on the runtime statistics.

[0040] Specifically, the segment value consists of three independent components, each measured by different criteria: The unified resource cost corresponds to the objectively incurred hardware resource consumption, measured by runtime statistics recovered after each subquery execution, converted to a comparable standard using a resource correction factor; the data intrinsic value corresponds to the amount of information provided by the data provider and the data participating in the calculation of this settlement segment, measured by the data object's unit price and the scale of data involved in the pricing; and the platform service cost corresponds to the weaving services and privacy computation provided by the data weaving platform within this settlement segment, measured by the scale of data processed and the corresponding service unit price. The sum of these three components yields the segment value of the settlement segment.

[0041] In one definite embodiment of this application, the measurement objects, measurement units, and attributable entities of the three are all different. The attributable entity for the unified resource cost is the data subsystem providing computing power, the attributable entity for the value of the data itself is the data holder providing the data, and the attributable entity for the platform service cost is the operator of the data weaving platform. If the measurement is combined without distinction, it will be impossible to allocate the cost to the correct attributable entity in a cross-entity trusted data space scenario.

[0042] In one embodiment of this application, the update of the resource correction factor only applies to settlement segments after the corresponding update occurs; the segment value of completed settlement segments remains unchanged due to the update. This application addresses the constitutionality of the resource correction factor. In a cross-entity trusted data space scenario, the segment value of completed settlement segments is generated along with the incremental settlement voucher for that settlement segment and notified to the relevant data providers. If subsequent updates to the resource correction factor are allowed to retroactively change the segment value of completed settlement segments, the fees already notified to all parties will be in a state of continuous change, making it impossible for the parties to make reconciliation and settlement arrangements accordingly. In contrast, adaptive optimization methods typically re-evaluate subsequent execution plans after correcting model parameters, with the optimization objective being optimal execution performance. However, the constraint object of this application is the certainty of settled fees; the objectives of the two are not the same.

[0043] In one understandable embodiment of this application, the initialization and updating of the resource correction factor, and the principles for determining the value of the data scale involved in the pricing, are addressed. The resource correction factor is initialized through capability probing: when the data subsystem first accesses the data weaving platform, a set of probing requests is sent to the data subsystem; the probing requests include, but are not limited to, metadata queries, basic queries, small-scale limited return count queries, count aggregation queries, and simple predicate filtering queries. Based on the execution results, the resource processing capability of the corresponding data subsystem is determined; the resource processing capability includes CPU computing power, data scanning throughput, network communication latency, aggregation computing power, and predicate filtering efficiency, and the resource correction factor is determined based on the resource processing capability. For example, if the probing reveals that the data scanning throughput of a columnar database is approximately 1.5 times that of the reference system, then its input-output resource correction factor is initialized as the ratio of the corresponding processing capability of the reference system to the probing processing capability.

[0044] Since capability detection only reflects the initial performance at the time of access, the processing capacity may change with system load and data scale. Therefore, after the subquery is executed, the platform compares the actual resource consumption with the optimizer's estimate, updates the resource correction factor according to the error ratio, for example, by using an exponential moving average, and applies it to the segment value calculation of subsequent settlement segments.

[0045] The data size used for pricing is the actual data size when runtime statistics exist; the actual data size includes the actual amount of scanned data, the actual amount of returned data, or the actual amount of data used in the calculation. When runtime statistics do not exist, the predicted data size estimated by the query optimizer based on the statistics is used. The data size processed by the weaving service and privacy computation also preferentially uses the actual statistical results from the execution feedback. It should be noted that since segment value is calculated immediately after each settlement segment is completed, runtime statistics always exist for completed settlement segments; therefore, there will be no situation where runtime statistics are not available. This mainly occurs when estimating the value of unexecuted settlement segments. Example 4

[0046] like Figure 6 As shown, in one embodiment of this application, the step of splitting the federated query into multiple subqueries according to the execution plan includes: Based on the execution plan, a unique subquery identifier is generated for each subquery. Based on the operator types involved in the federated query, the resource correction factors corresponding to each data subsystem, and the data unit price of each data object, the pricing caliber identifier of the federated query is generated. The pricing caliber identifier is bound to the sub-query identifier, and the bound sub-query identifier and the pricing caliber identifier are appended to the query statement as comments, and then sent to multiple data subsystems for execution.

[0047] In this application, the pricing caliber identifier is bound to the subquery identifier, and the bound identifier is appended to the query statement as a comment before being issued. In this application, the data consumer obtains the pricing commitment for the query upon submission of a federated query. The platform generates a pricing caliber identifier based on the types of operators involved in the query, including whether they include joins, aggregations, sorting, etc., the currently effective resource correction factors of each data subsystem, and the configured data unit price of each data object. This identifier uniquely identifies the resource unit price version, correction factor version, and discount strategy version used in this pricing, thus ensuring that the original pricing caliber is determined when the same query is recalculated at different times. After binding the pricing caliber identifier to the subquery identifier, the platform appends it as a comment to the query statement issued to each data subsystem. Therefore, regardless of whether the data subsystem records its execution process in the form of query logs, execution logs, or indicator data, the corresponding execution record can be located by the subquery identifier in the comment, and the pricing caliber identifier can be used to trace back to the pricing caliber used when the platform generated the subquery.

[0048] In one embodiment of this application, regardless of whether the data subsystem records its execution process in the form of query logs, execution logs or indicator data, the corresponding execution record can be located by the subquery identifier in the annotation, and the pricing caliber identifier can be used to trace back to the pricing caliber used when the platform generated the subquery. Example 5

[0049] like Figure 4 and Figure 7 As shown, in one embodiment of this application, the step of performing a downgrade rewrite on the subquery corresponding to the remaining settlement segment and then issuing it for execution includes: Among a set of preset downgrade levels, the downgrade level that minimizes the predicted value of the remaining settlement segment from the upper limit of the value and has the lowest downgrade degree is selected, and the downgrade rewriting is performed according to the selected downgrade level; wherein, Each of the aforementioned downgrade levels corresponds to a sampling rate and a maximum pushdown data size.

[0050] In this application, multiple downgrade levels are preset, each downgrade level corresponding to a sampling rate and a maximum pushdown data size. The sampling rates corresponding to two adjacent downgrade levels differ by an order of magnitude, and the maximum pushdown data size corresponding to any level is proportional to the sampling rate of that level.

[0051] When the budget verification determines that the sum of the cumulative accounting value and the estimated accounting value of the remaining settlement segment is greater than the accounting value limit, the platform does not directly stop the query. Instead, it selects the level among the above levels that ensures the predicted accounting value of the remaining settlement segment is not greater than the accounting value limit and has the lowest degree of downgrade. Example 6

[0052] like Figure 7 and Figure 8 As shown, in one embodiment of this application, the step of performing downgrade rewriting according to the selected downgrade level includes: Replace the exact aggregation operator in the subquery corresponding to the remaining settlement segment with an approximate aggregation operator based on stratified sampling; wherein, The sampling rate of the approximate aggregation operator is equal to the sampling rate corresponding to the selected downgrade level. Set the upper limit of the pushdown data size of the subquery corresponding to the remaining settlement segment to the upper limit of the pushdown data size corresponding to the selected degradation level; wherein, The downgrade rewriting maintains the same result data pattern as the federated query.

[0053] Specifically, the downgrade rewriting includes: replacing the exact aggregation operator in the subquery corresponding to the remaining settlement segment with an approximate aggregation operator based on stratified sampling, with a sampling rate equal to the sampling rate corresponding to the selected downgrade level; and setting the upper limit of the pushdown data size of the subquery to the upper limit of the pushdown data size corresponding to the selected downgrade level. The downgrade rewriting maintains the data pattern of the query results unchanged, that is, the column names, column types, and column numbers of the returned results are completely consistent before and after the rewriting.

[0054] Furthermore, the platform determines the downgrade ratio based on the sampling rate of the selected tier, and the downgrade ratio is equal to one minus the sampling rate. This can be understood as follows: the data value of the remaining settlement segment is reduced according to the downgrade ratio, and the reduced segment value is: unified resource cost + data value × (1 - downgrade ratio) + platform service cost.

[0055] It can be clearly stated that the reduction only applies to the intrinsic value of the data. This is because the unified resource cost corresponds to the objectively incurred hardware resource consumption, and the platform service cost corresponds to the objectively provided weaving and privacy computing services provided by the platform. The actual amount of these two does not decrease due to the reduction in result accuracy. However, the intrinsic value of the data corresponds to the amount of data information actually obtained by the consumer. This amount of information decreases linearly with the sampling rate, so the sum of the downgrade ratio and the sampling rate is one.

[0056] Furthermore, it is understood that the downgrade rewriting only applies to the subqueries corresponding to the remaining settlement segments that have not yet been executed. The execution results and segment billing values ​​of completed settlement segments remain unchanged due to the downgrade rewriting. This application applies this limitation because the actual reason is that the downgrade rewriting occurs at the cost anchor point, at which time the results of completed settlement segments have already been generated and their segment billing values ​​have been determined. If the rewriting retroactively affects completed settlement segments, the previously determined segmented billing results will become meaningless, and the budget verification and reconciliation mechanism built on segmented billing will also fail. Therefore, the scope of the rewriting is limited to the portion that has not yet been executed after the anchor point. Example 7

[0057] like Figure 9 As shown, in one embodiment of this application, the method further includes: When a settlement segment is completed, an incremental settlement voucher for the corresponding settlement segment is generated; Specifically, upon completion of each settlement segment, the platform generates an incremental settlement voucher. Its fields include: segment number, segment value (comprising unified resource cost, data intrinsic value, and platform service cost), the cost anchor point identifier corresponding to the start and end positions of the settlement segment, and a summary of the incremental settlement voucher from the previous settlement segment. When a settlement segment undergoes a downgrade rewrite, the voucher also includes a downgrade identifier, the selected downgrade tier identifier, the downgrade ratio, and a result accuracy declaration for that settlement segment. The result accuracy declaration includes the sampling rate corresponding to the selected downgrade tier.

[0058] The incremental settlement document includes a segment number, the segment value of the settlement segment, the location identifier of the cost anchor point at the start and end of the settlement segment, and a summary of the incremental settlement document of the previous settlement segment; the summary can be a hash digest. Since each document carries the summary of the previous document, all documents form a chain in the order of their generation: any modification to the segment value in a historical document will change the summary of that document, thus causing the summary comparison of all subsequent documents to fail. Example 8

[0059] In one practical embodiment of this application, the result accuracy declaration is written into the incremental settlement voucher. This is because, in a cross-entity trusted data space scenario, the data consumer needs to determine which parts of the result were generated after reducing accuracy, and to what extent, in order to judge whether the result can be used for its established business decisions; the data provider also needs to explain the completeness of the data it provides being used. Placing the result accuracy declaration and the segmented value in the same voucher ensures that the correspondence between cost and accuracy is fixed along with the voucher chain, preventing either party from unilaterally modifying either one.

[0060] The process generates incremental settlement vouchers for the corresponding settlement segment. The incremental settlement vouchers include summary verification, two-level local recalculation, and cross-entity allocation. The summary is divided into two steps: verification and accumulation.

[0061] According to the segment number, the summary in each incremental settlement voucher is compared with the summary of the incremental settlement voucher in the previous settlement segment. After the verification is passed, the segment settlement value in the corresponding incremental settlement voucher is accumulated. Specifically, the verification process iterates through all incremental settlement vouchers in segment order, recalculates the summary of the previous voucher for each voucher, and compares it with the summary recorded in that voucher; after all comparisons pass, the sums are accumulated. If any voucher fails to be compared, the billing result of that federated query is marked as pending verification.

[0062] The proportion of each data provider's actual resource consumption within a settlement segment to the sum of the actual resource consumption of all data subsystems within the corresponding settlement segment is used to allocate the unified resource cost of the corresponding settlement segment to each data provider.

[0063] Specifically, the reconciliation process handles two types of disputes. The first type is segment-level disputes: for settlement segments marked with a downgrade flag, only the subquery corresponding to that settlement segment is re-executed to obtain a recalculated value. When the deviation between the recalculated value and the original segment value exceeds a preset deviation threshold, the recalculated value replaces the original segment value, and the unified value is corrected. The second type is allocation disputes: when any data provider objects to the unified resource cost allocated to them, only the subquery corresponding to that data provider within that settlement segment is re-executed, and the unified resource cost allocated to that data provider is adjusted based on the recalculation result. In practice, because the recalculation scope is limited to a single settlement segment, or even a single data provider's subquery within that segment, the resources and data volume consumed by the recalculation are significantly reduced. Example 9

[0064] like Figure 10 As shown, in one embodiment of this application, the method further includes: Based on the unified valuation, determine whether there are any settlement segment objections; When an objection arises to any settlement segment in the unified value of the federated query, the boundary of the subquery corresponding to that settlement segment is determined based on the position identifier of the cost anchor point at the start and end of the disputed settlement segment. The subquery corresponding to that settlement segment is then re-executed to determine the recalculated value of the corresponding settlement segment. The unified value is obtained by summing the segment values ​​of multiple settlement segments in the order of the settlement segments; The settlement segment is divided by the cost anchor point set in the execution plan of the federated query based on the cumulative variance estimated by the cardinality. The segment value of each settlement segment is fixed in the incremental settlement voucher of the corresponding settlement segment; The incremental settlement voucher includes a segment number, a segment value, and the location identifier of the cost anchor point at the start and end of the settlement segment; The recalculated value is compared with the segment value of the settlement segment in the incremental settlement voucher. When the deviation between the two is greater than a preset deviation threshold, the segment value of the corresponding settlement segment is replaced with the recalculated value, and the unified value is corrected. Specifically, when the objection is raised by the data provider and points to the uniform resource cost allocated to the data provider, only the subquery corresponding to the data provider in the settlement segment is re-executed, and the uniform resource cost allocated to the data provider is adjusted according to the recalculation result.

[0065] In this application, within a cross-entity trusted data space scenario, the unified billing value needs to be shared among multiple data providers, making billing disputes an inevitable occurrence. Existing technologies, by only retaining overall usage records at the query level, require re-executing the entire federated query to recalculate costs in the event of a dispute, which means consuming resources and data from all parties again. The key to this embodiment is aligning the recalculation scope with the fixed billing unit: since the segment billing value of each settlement segment is fixed with the incremental settlement voucher, and the voucher records the location identifier of the cost anchor point at the start and end of the segment, the platform can directly use this location identifier to retrieve the set of subqueries covered by the settlement segment, locking the recalculation scope within this set without affecting other settlement segments. For example, in a federated query containing 3 settlement segments and involving 6 subqueries, if the second settlement segment is disputed, only the two subqueries involved in that segment need to be re-executed.

[0066] Furthermore, a voucher chain verification is performed before recalculation: the summary in each incremental settlement voucher is compared sequentially by segment number to see if it matches the summary of the incremental settlement voucher in the previous settlement segment. Recalculation is only performed if the verification passes; if the verification fails, it means that the voucher or the voucher before it has been modified. In this case, the recalculation process is not initiated, but the billing result of the federated query is marked as pending verification. The reason for this sequential arrangement is that if the voucher chain itself is broken, the recalculated value will lose its benchmark for comparison, making recalculation meaningless.

[0067] Furthermore, when an objection is raised by a data provider and only targets the uniform resource cost allocated to it, the scope of recalculation can be narrowed one level further on the settlement segment, and only the subquery corresponding to that data provider within that settlement segment can be re-executed.

[0068] Regarding the determination of deviation, the segment value includes three components: unified resource cost, data value itself, and platform service cost. The deviation between the recalculated value and the segment value is determined by the ratio of the changed component to the original component.

[0069] Specifically, recalculation usually only changes the components that can be measured immediately, such as the cost of unified resources; while the value of the data itself and the cost of platform services do not change if the result remains unchanged; if the value is calculated in segments and the overall ratio is taken, the denominator includes the components that do not change, which will dilute the actual deviation, causing the deviation that should be corrected to be judged as not exceeding the threshold. Example 10

[0070] like Figure 2 As shown, this application proposes a multi-dimensional data pricing system for trusted data spaces, including a query receiving module, a query splitting module, a settlement segment division module, a segment value determination module, a budget control module, and a summary module.

[0071] The query receiving module receives federated queries and determines the execution plan. The query splitting module, based on the execution plan, splits the federated query into multiple subqueries, which are then distributed to multiple data subsystems for execution. Specifically, the query splitting module is used to split the received federated query into multiple subqueries, generate a unique subquery identifier for each subquery and bind it to the pricing caliber identifier of the federated query, and then attach the bound identifier to the query statement as a comment before sending it to multiple data subsystems for execution.

[0072] The settlement segment division module determines multiple cost anchors in the execution plan based on the cumulative variance of the cardinality estimate in the execution plan, and divides the execution plan into multiple settlement segments using the cost anchors; Specifically, the settlement segment division module is used to accumulate the variance of the cardinality estimate at each connection operator according to the execution order of the execution plan to obtain the cumulative variance. When the cumulative variance is greater than the preset variance threshold, a cost anchor point is set after the connection operator and the cumulative variance is set to zero. The execution plan is divided into multiple settlement segments by the cost anchor point. The variance threshold is determined based on the preset upper bound of the default probability and the preset error ratio threshold.

[0073] The segment value determination module obtains runtime statistics after the execution of each sub-query within the corresponding settlement segment for each settlement segment, and determines the segment value of the corresponding settlement segment. Specifically, the segment pricing module is used to obtain runtime statistics of each subquery within each settlement segment through the unified interface of the data source connector, determine the actual resource consumption of each subquery, map the actual resource consumption to a unified resource cost based on the resource correction factor corresponding to each data subsystem, determine the value of the data itself based on the data scale and data unit price involved in the pricing, determine the platform service cost based on the data scale and corresponding service unit price of the platform's weaving service and privacy computing, and sum the three to obtain the segment pricing value of the settlement segment.

[0074] The budget control module, at the cost anchor point, when the sum of the segment cost value of the completed settlement segment and the estimated cost value of the remaining settlement segment is greater than the preset cost value upper limit, performs a downgrade rewrite on the subquery corresponding to the remaining settlement segment and sends it down for execution, and determines the segment cost value of the remaining settlement segment according to the actual resource consumption after the downgrade rewrite; Specifically, the budget control module is used to accumulate the segment value of completed settlement segments at the cost anchor point to obtain the cumulative value, and predict the estimated value corresponding to the remaining settlement segments. When the sum of the two is greater than the preset upper limit of the value, the module selects the lowest degradation level among the preset multiple degradation levels that makes the predicted value no greater than the upper limit of the value and has the lowest degree of degradation. The module then performs degradation rewriting on the remaining subqueries according to the selected degradation level and sends it down for execution.

[0075] The aggregation module aggregates the segment values ​​of each settlement segment according to the order of the cost anchor points, and generates the unified value of the federated query.

[0076] Specifically, the aggregation module is used to aggregate the segmented values ​​of each settlement segment in the order of the cost anchor points to obtain a unified value, and to allocate the unified resource cost according to the proportion of the actual resource consumption of each data provider in a settlement segment to the sum of the actual resource consumption of all data subsystems in that settlement segment.

[0077] This application considers that all the above modules are deployed within the data weaving platform and the billing engine. Specifically, the query splitting module and the settlement segment division module are deployed within the data weaving platform; the segment pricing module, the budget control module, and the aggregation module are deployed within the billing engine. Example 11

[0078] In one alternative embodiment, the downshift gear is subject to bidirectional adaptive adjustment.

[0079] After rewriting the remaining subquery according to the selected degradation level, if the sum of the cumulative value and the predicted value of the remaining settlement segment is still greater than the upper limit of the value, the next degradation level is selected in ascending order of degradation degree and the prediction is re-made until the upper limit is met or all levels are exhausted. If the sum of the cumulative value and the predicted value of the remaining settlement segment is less than the upper limit of the value and reaches a preset margin, the degradation level is reverted to the previous degradation level in descending order of degradation degree to restore the accuracy of the result. Through the above two-way adjustment, the degradation degree is maintained within the optimal range that just meets the budget constraint, rather than being reduced to the bottom all at once.

[0080] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-dimensional data valuation method for a trusted data space, applied to a data weaving platform, characterized in that, Including the following steps: Receive federal queries and determine the execution plan; According to the execution plan, the federated query is split into multiple subqueries, which are then distributed to multiple data subsystems for execution. Based on the cumulative variance of the cardinality estimate in the execution plan, multiple cost anchors are determined in the execution plan, and the execution plan is divided into multiple settlement segments using the cost anchors; For each settlement segment, obtain the runtime statistics after the execution of each subquery within the corresponding settlement segment, and determine the segment settlement value of the corresponding settlement segment; At the cost anchor point, when the sum of the segment value of the completed settlement segment and the estimated value of the remaining settlement segment is greater than the preset value upper limit, the subquery corresponding to the remaining settlement segment is downgraded and rewritten before being sent for execution, and the segment value of the remaining settlement segment is determined according to the actual resource consumption after the downgraded rewriting. The segmented values ​​of each settlement segment are aggregated according to the order of the cost anchor points to generate the unified value of the federated query.

2. The multi-dimensional data pricing method for a trusted data space according to claim 1, characterized in that, The step of determining multiple cost anchors in the execution plan based on the cumulative variance estimated from the cardinality in the execution plan includes: Based on the execution order of the execution plan, the variance of the cardinality estimate of the corresponding connection operator is accumulated at each connection operator to determine the cumulative variance; When the cumulative variance exceeds a preset variance threshold, a cost anchor is set after the corresponding connection operator, and the cumulative variance is set to zero before continuing to accumulate; wherein, The total number of cost anchors set shall not exceed the preset maximum number of anchors.

3. The multi-dimensional data pricing method for a trusted data space according to claim 2, characterized in that, The variance threshold is determined based on a preset upper bound for the probability of default and a preset error ratio threshold, such that the probability that the relative amount by which the estimated value of any settlement segment deviates from its expected value exceeds the error ratio threshold is not greater than the upper bound for the probability of default.

4. The multi-dimensional data pricing method for a trusted data space according to claim 1, characterized in that, The determination of the segment value of the corresponding settlement segment includes: Based on the runtime statistics, the actual resource consumption of each subquery is determined, and the actual resource consumption is mapped to a unified resource cost based on the resource correction factor corresponding to each data subsystem. The value of the data itself is determined based on the unit price of the data objects involved in the settlement segment and the scale of the data participating in the pricing. The platform service cost is determined based on the scale of data processed by the data weaving platform in performing weaving services and privacy computing, and the corresponding service unit price. The unified resource cost, the data's intrinsic value, and the platform service cost are summed to determine the segment value for the corresponding settlement segment; wherein... The resource correction factor is determined based on the resource processing capabilities of each data subsystem and is updated based on the runtime statistics.

5. A multi-dimensional data pricing method for a trusted data space according to claim 1, characterized in that, The process of splitting the federated query into multiple subqueries according to the execution plan includes: Based on the execution plan, a unique subquery identifier is generated for each subquery. Based on the operator types involved in the federated query, the resource correction factors corresponding to each data subsystem, and the data unit price of each data object, the pricing caliber identifier of the federated query is generated. The pricing caliber identifier is bound to the sub-query identifier, and the bound sub-query identifier and the pricing caliber identifier are appended to the query statement as comments, and then sent to multiple data subsystems for execution.

6. The multi-dimensional data pricing method for a trusted data space according to claim 1, characterized in that, The step of performing a downgrade rewrite on the subquery corresponding to the remaining settlement segment and then issuing it for execution includes: Among a set of preset downgrade levels, the downgrade level that minimizes the predicted value of the remaining settlement segment from the upper limit of the value and has the lowest downgrade degree is selected, and the downgrade rewriting is performed according to the selected downgrade level; wherein, Each of the aforementioned downgrade levels corresponds to a sampling rate and a maximum pushdown data size.

7. A multi-dimensional data pricing method for a trusted data space according to claim 6, characterized in that, The process of performing a downgrade rewrite according to the selected downgrade level includes: Replace the exact aggregation operator in the subquery corresponding to the remaining settlement segment with an approximate aggregation operator based on stratified sampling; wherein, The sampling rate of the approximate aggregation operator is equal to the sampling rate corresponding to the selected downgrade level. Set the upper limit of the pushdown data size of the subquery corresponding to the remaining settlement segment to the upper limit of the pushdown data size corresponding to the selected degradation level; wherein, The downgrade rewriting maintains the same result data pattern as the federated query.

8. A multi-dimensional data pricing method for a trusted data space according to claim 4, characterized in that, The method further includes: When a settlement segment is completed, an incremental settlement voucher for the corresponding settlement segment is generated; where, The incremental settlement voucher includes a segment number, the segment value of the settlement segment, the location identifier of the cost anchor point at the start and end of the settlement segment, and a summary of the incremental settlement voucher of the previous settlement segment. Execute the generation of incremental settlement vouchers for the corresponding settlement segment, including: Based on the segment number, the summary in each incremental settlement voucher is sequentially compared with the summary of the incremental settlement voucher in the previous settlement segment. If the verification passes, the segment value in the corresponding incremental settlement voucher is accumulated. The proportion of each data provider's actual resource consumption within a settlement segment to the sum of the actual resource consumption of all data subsystems within the corresponding settlement segment is used to allocate the unified resource cost of the corresponding settlement segment to each data provider.

9. A multi-dimensional data pricing method for a trusted data space according to claim 1, characterized in that, The method further includes: Based on the unified valuation, determine whether there are any settlement segment objections; wherein... When an objection arises to any settlement segment in the unified value of the federated query, the boundary of the subquery corresponding to that settlement segment is determined based on the position identifier of the cost anchor point at the start and end of the disputed settlement segment. The subquery corresponding to that settlement segment is then re-executed to determine the recalculated value of the corresponding settlement segment. The unified value is obtained by summing the segment values ​​of multiple settlement segments in the order of the settlement segments; The settlement segment is divided by the cost anchor point set in the execution plan of the federated query based on the cumulative variance estimated by the cardinality. The segment value of each settlement segment is fixed in the incremental settlement voucher of the corresponding settlement segment; The incremental settlement voucher includes a segment number, a segment value, and the location identifier of the cost anchor point at the start and end of the settlement segment; The recalculated value is compared with the segment value of the settlement segment in the incremental settlement voucher. When the deviation between the two is greater than a preset deviation threshold, the segment value of the corresponding settlement segment is replaced with the recalculated value, and the unified value is corrected. Specifically, when the objection is raised by the data provider and points to the uniform resource cost allocated to the data provider, only the subquery corresponding to the data provider in the settlement segment is re-executed, and the uniform resource cost allocated to the data provider is adjusted according to the recalculation result.

10. A multi-dimensional data pricing system for a trusted data space, characterized in that, include: The query receiving module receives federated queries and determines the execution plan. The query splitting module, based on the execution plan, splits the federated query into multiple subqueries, which are then distributed to multiple data subsystems for execution. The settlement segment division module determines multiple cost anchors in the execution plan based on the cumulative variance of the cardinality estimate in the execution plan, and divides the execution plan into multiple settlement segments using the cost anchors; The segment value determination module obtains runtime statistics after the execution of each sub-query within the corresponding settlement segment for each settlement segment, and determines the segment value of the corresponding settlement segment. The budget control module, at the cost anchor point, when the sum of the segment cost value of the completed settlement segment and the estimated cost value of the remaining settlement segment is greater than the preset cost value upper limit, performs a downgrade rewrite on the subquery corresponding to the remaining settlement segment and sends it down for execution, and determines the segment cost value of the remaining settlement segment according to the actual resource consumption after the downgrade rewrite; The aggregation module aggregates the segment values ​​of each settlement segment according to the order of the cost anchor points, and generates the unified value of the federated query.

Images