Heterogeneous data quality evaluation and virtual cross-domain sharing method under cloudy environment

Abstract

This invention relates to the field of computer data processing technology, and discloses a method for heterogeneous data quality assessment and virtualized cross-domain sharing in a multi-cloud environment. The invention includes: generating virtual data objects from heterogeneous data sources in multiple cloud domains and establishing correspondences between fields, data sources, permissions, lineage, caches, and candidate sources; obtaining field status, data source running status, cache status, permission tag consistency, and historical execution status to generate a basic quality profile; after a cross-domain query request passes access verification, generating a cross-domain quality profile based on the requester's domain context, and converting it into field constraints, data source constraints, and execution constraints to update the virtual data object metadata and rewrite the cross-domain query execution plan; and updating the profile and quality constraint description based on execution feedback. This invention can improve the reliability of cross-domain queries and the identifiability of result credibility boundaries.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a method for heterogeneous data quality assessment and virtualized cross-domain sharing in a multi-cloud environment. Background Technology

[0002] Heterogeneous data virtualization access and cross-domain sharing in multi-cloud environments are crucial technical aspects of cloud computing, big data governance, and data circulation platforms. As business systems in enterprise, government, finance, healthcare, and industrial internet scenarios are increasingly deployed across public, private, industry, and hybrid cloud platforms, data sources exhibit significant heterogeneity in storage structure, access protocols, field schemas, permission tags, update cycles, caching mechanisms, and lineage relationships. To reduce cross-cloud data migration costs and mitigate data redundancy and consistency risks associated with redundant synchronization, existing systems typically employ methods such as data catalogs, metadata management, data virtualization, federated queries, and cross-domain access control to achieve unified access and sharing of heterogeneous data across multiple clouds.

[0003] Existing solutions related to secure cross-domain data sharing across multiple clouds typically focus on addressing issues such as cross-domain trust establishment, access authorization, attribute mapping, and record-keeping during the sharing process. For example, patent CN113132103B discloses a secure cross-domain data sharing system and method. This solution includes a user module, a multi-cloud autonomy module, and a data storage module. It uses an attribute mapping table for fine-grained cross-domain attribute mapping and leverages blockchain technology to establish trust relationships between multiple clouds, enabling file information to be uploaded to the blockchain, access requests, and access decisions. The shortcomings of this solution are: its focus is on cross-domain access control and the establishment of multi-cloud trust relationships, primarily addressing the access qualification issue for users or data requesters, without further handling the impact of changes in the operational status of heterogeneous multi-cloud data sources on the virtualization query execution process after access permission is granted.

[0004] Existing solutions related to secure data weaving and cross-domain access policy integration have been publicly disclosed. For example, patent application US20220021711A1 discloses a security platform and a method for efficient access and discovery. This solution involves secure data weaving, data cataloging, data virtualization, attribute-based access control, audit trails, and cross-domain data access policy integration, and can be used for data access discovery and policy control across networks and data sources. The shortcomings of this solution are: its focus is on security policies, access control, data discovery, and audit management, while it pays insufficient attention to data source reachability, field status, cache status, and virtualization execution control under changes in historical query execution status. This makes it difficult to solve the problem of query failures or inconsistent results caused by changes in data source status during cross-domain query execution.

[0005] Existing solutions related to data governance and heterogeneous data quality assessment typically focus on multi-source heterogeneous data access, data exploration, cleaning, standardization, cataloging, lineage determination, and quality closed-loop management. For example, patent CN112699175B discloses a data governance system and method. This solution reads multi-source heterogeneous data through a data access module and performs data exploration, data definition, and data reconciliation; it extracts, converts formats, and corrects or removes abnormal data through a data processing module; it forms various types of metadata through a data organization module; and it integrates catalogs, classifies and categorizes data, determines lineage, manages quality, performs data operation and maintenance, and provides data services through a data governance module. The shortcomings of this solution are: its quality assessment and quality closed-loop are mainly geared towards data access, cleaning, cataloging, and service management within the data governance platform. The quality results are primarily used for governance, operation and maintenance, and data service management, making it difficult to directly impact the query execution process of the virtualization engine. It also struggles to promptly address execution risks arising from changes in field quality, cache status, or data source malfunctions during cross-domain query execution.

[0006] Existing solutions related to heterogeneous data sources, virtual data models, and data quality assessment have been publicly disclosed. For example, patent US11409735B2 discloses a scheme for selective preprocessing of data stored in heterogeneous data sources. This scheme involves connecting multiple heterogeneous data sources, acquiring metadata, generating a virtual data model, and performing quality assessment on data assets or fields based on a data quality assessment module, thereby assisting in data source matching, data preprocessing, or execution plan selection. The shortcomings of this scheme are that its quality assessment is mainly reflected in quality scores, data asset ranking, preprocessing suggestions, or execution plan selection criteria, without fully considering the differences in the combination of virtual objects, fields, physical data sources, and requesting domains in multi-cloud cross-domain queries, and without further transforming the quality assessment results into constraint information that can directly participate in the virtualization query execution control.

[0007] Existing solutions related to virtual data objects and cross-engine unified querying have been publicly disclosed. For example, patent CN114647716B discloses a system suitable for generalized data warehouses. This solution designs virtual data objects, real-time data processing, and cross-engine unified data querying based on data warehouse technology. Virtual data objects can establish mapping relationships with multi-source heterogeneous data sources and other virtual objects. Cross-engine unified querying supports multiple database query engines and unstructured data query engines. The shortcomings of this solution are: its focus is on virtual data object modeling, real-time data processing, and cross-engine unified querying, which usually relies on virtual object metadata, field mapping relationships, and predetermined query paths for access; when the physical data source is unreachable, field schemas drift, cache versions expire, or permission tags change, the original query path may still be invoked, leading to query failures, inconsistent returned results, or unclear result credibility boundaries.

[0008] In summary, the existing technology has the following shortcomings: First, existing multi-cloud cross-domain sharing solutions focus primarily on access control, attribute mapping, blockchain notarization, or cross-domain authorization decisions, addressing the question of whether access is permitted, while paying insufficient attention to virtualization query execution control after access permission is granted.

[0009] Second, existing data governance and data quality assessment solutions are mostly used for data access, cleaning, cataloging, lineage, quality reporting, or governance closed loop. The quality results usually affect the management layer or service release layer, and are difficult to directly participate in field calls, data source selection, cache calls, and execution path adjustments in the virtualization engine.

[0010] Third, while existing virtual data models and federated query schemes can generate query execution plans based on heterogeneous data sources, they typically rely on static metadata, fixed field mapping relationships, or preset query paths. When data source reachability, field schemas, cached versions, permission tags, or historical execution states change, the original virtual object metadata and cross-domain query execution plans may not be able to adapt in a timely manner.

[0011] Fourth, existing caching or candidate data source invocation solutions typically focus on query acceleration, disaster recovery, or cache hit rate, failing to fully integrate field status, data source running status, cache status, permission tag consistency, and historical execution status to uniformly constrain cache invocation, candidate source selection, and failure rollback processes in cross-domain queries.

[0012] Fifth, existing technologies do not adequately utilize feedback from cross-domain query execution. Feedback information generated during cross-domain queries, such as source unreachability, field schema drift, cache invalidation, permission label changes, abnormal response latency, query failure, or field quality degradation, is usually not used in a timely manner to correct subsequent virtualized query execution processes. This makes it difficult for virtualized query execution processes to adapt to the dynamic changes of multi-cloud heterogeneous data sources in a timely manner. Summary of the Invention

[0013] Given that existing multi-cloud cross-domain sharing solutions focus on access control but pay insufficient attention to virtualization query execution control after access, the quality results of existing data governance and quality assessment solutions cannot directly affect the query execution process of the virtualization engine, existing virtual data models and federated query solutions rely on static metadata and cannot adapt to dynamic changes in data sources in a timely manner, and existing technologies do not make sufficient use of cross-domain query execution feedback, this invention proposes a method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment.

[0014] Its purpose is to generate quality characterization information based on the field status, running status, cache status, permission tag consistency, and historical execution status of the data source during the cross-domain sharing process of multi-cloud heterogeneous data virtualization. This quality characterization information is then converted into constraint information that the virtualization engine can parse and execute. The process of virtual object metadata, field calls, data source selection, cache calls, failure rollback, and result trust identification generation is coordinated and controlled. Through cross-domain query execution feedback, the subsequent virtualization query execution process is continuously corrected, thereby reducing the situation where invalid fields, unreachable data sources, or expired caches are continued to be called, reducing the risk of cross-domain query failures and result inconsistencies, and improving the identifiability of result trust boundaries.

[0015] The technical solution of this invention is a method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment, applied to a data sharing system including a virtualization engine, comprising: S1: Generate virtual data objects for heterogeneous data sources in multiple cloud domains registered to the virtualization engine, and establish correspondences between virtual data objects and field mapping relationships, physical data source identifiers, cloud domain identifiers, access interface protocols, permission tags, lineage identifiers, cache version identifiers, and candidate data source association information.

[0016] S2: Obtain quality collection information associated with the virtual data object. The quality collection information includes field status information, data source running status information, cache status information, source-side permission tag consistency information, and historical execution status information.

[0017] S3: Based on the correspondence and quality collection information, generate a basic quality profile that is associated with virtual data objects, fields and physical data sources.

[0018] S4: In response to a cross-domain query request that has passed access permission verification, obtain the requesting domain context.

[0019] S5: Based on the basic quality profile, requester domain context, and corresponding relationships, generate a cross-domain quality profile associated with virtual data objects, fields, physical data sources, and requester domains. The cross-domain quality profile includes cross-domain permission label consistency, which is formed by the matching results between the requester domain permission labels in the requester domain context, the permission labels of virtual data objects, and the permission labels of physical data sources.

[0020] S6: Convert cross-domain quality profiles into quality constraint descriptions that can be parsed and executed by the virtualization engine. The quality constraint descriptions include field constraints, data source constraints, and execution constraints.

[0021] S7: Determine the set of callable fields and field transformation methods based on field constraints, determine the data source to be called based on data source constraints, including physical data source, cached data source or candidate data source, and determine cache validity conditions, failure fallback paths and result trust identifier generation rules based on execution constraints.

[0022] S8: Update the metadata of the virtual data object based on the quality constraint description, and generate or rewrite the cross-domain query execution plan.

[0023] S9: Perform cross-domain data access according to the cross-domain query execution plan, obtain the cross-domain query execution feedback, and update the basic quality profile, cross-domain quality profile, and quality constraint description based on the cross-domain query execution feedback.

[0024] Furthermore, establishing the corresponding relationship in S1 includes: establishing a field mapping relationship between the fields in the virtual data object and the fields in the physical data source; recording the cloud domain, access interface protocol, source data update information, cache version identifier, and lineage identifier of the physical data source fields; recording the permission tags corresponding to the virtual data object, the fields in the virtual data object, and the fields in the physical data source respectively; and recording the association relationship between the fields in the virtual data object and the cache data source, mirror data source, synchronous replica data source, semantically compatible data source, and candidate alternative data source.

[0025] Furthermore, the quality collection information in S2 includes: field status information, data source running status information, cache status information, source-side permission tag consistency information, and historical execution status information; field status information includes field missing rate, format error rate, field schema version, field semantic identifier, and field update time; data source running status information includes interface reachability detection results, response latency, source data update time, source version identifier, semantic mapping version status, data synchronization status, and access failure records; cache status information includes cache version identifier, cache generation time, cache validity period, cache source identifier, and source data version matching relationship; source-side permission tag consistency information includes the matching results between virtual data object permission tags and physical data source permission tags; historical execution status information includes cross-domain query failure rate, failure type, failure rollback records, response latency anomaly records, and result trust identifier anomaly records.

[0026] Furthermore, the requester domain context in S4 includes requester domain permission tags, requester domain timeliness requirements, and result granularity requirements; the cross-domain quality profile in S5 includes cross-domain permission tag consistency, which is formed by the matching results between the requester domain permission tags, the permission tags of virtual data objects, and the permission tags of physical data sources.

[0027] Furthermore, the basic quality profile in S3 is represented as follows: Cross-domain quality profile is represented as follows: The vector structure representation of cross-domain quality profiling is as follows: In the formula, Indicates time Below and virtual data objects Fields and physical data sources The associated basic quality profile, Indicates time Below and virtual data objects Fields Physical data source and requesting domain Related cross-domain quality profiles Indicates the requester's domain context. This indicates a portrait generation function that incorporates the requester's domain context into the base quality portrait.

[0028] Furthermore, Indicates structural integrity. Indicates pattern consistency. Indicates the basic semantic matching degree. This indicates the cross-domain semantic matching degree after considering the granularity requirements of the requester's domain result. Indicates basic timeliness. This indicates the cross-domain timeliness after considering the timeliness requirements of the requesting domain. Indicates interface reachability. Indicates cache validity. This indicates consistency of the source-side permission tags. This indicates consistency between cross-domain permission tags following the requester's domain permission tag. Indicates the integrity of bloodline. This indicates historical execution stability.

[0029] Furthermore, the quality constraint description in S6 includes field constraint items. Data source constraints and execution constraints And expressed as: In the formula, Represents virtual data objects In the requesting domain The following quality constraint description, Indicates a collection of callable fields. This indicates that accessing the field collection is prohibited. Represents a set of field transformation operators. Indicates the collection of physical data sources that can be invoked. This indicates a collection of cached data sources that can be invoked. This indicates the collection of candidate data sources that can be invoked. Indicates cache validity conditions. Indicates the fallback path in case of failure. This indicates the rules for generating a reliable identifier for the result.

[0030] Furthermore, in S7, the virtualization engine inserts field transformation operators into the cross-domain query execution plan based on field constraints, and writes physical data source call paths, cached data source call paths, or candidate data source call paths into the cross-domain query execution plan based on data source constraints. Field transformation operators include field pruning operators, field desensitization operators, field aggregation operators, format conversion operators, and granularity adjustment operators. Candidate data source call paths are generated from candidate data sources that satisfy field mapping relationships, source-side permission label consistency information, cross-domain permission label consistency, data source running status information, and version status, semantic mapping version status, data synchronization status, or cache status corresponding to the candidate data source type.

[0031] Furthermore, the field constraint items determine the set of callable fields, the set of prohibited fields, and the set of field transformation operators based on field status information, source-side permission tag consistency information, cross-domain permission tag consistency, and cross-domain quality profile; the virtualization engine updates the field call flags, field masking flags, and field transformation flags in the virtual data object metadata according to the field constraint items, and inserts field transformation operators into the cross-domain query execution plan.

[0032] Furthermore, the data source constraints determine the physical data source set, cache data source set, and candidate data source set based on data source running status information, cache status information, source-side permission tag consistency information, cross-domain permission tag consistency, and cross-domain quality profile. The execution constraints determine the cache validity conditions, failure fallback paths, and result trust identifier generation rules based on cache status information, data source running status information, and historical execution status information. The cache validity conditions are formed by the cache version identifier, cache generation time, source data update time, requester domain timeliness requirements, and cache source identifier. When a physical data source call fails, the failure fallback path generates subsequent execution paths in the order of cache data source call path, candidate data source call path, and path that refuses execution and triggers quality re-evaluation. The result trust identifier generation rules generate a result trust identifier carrying field transformation flags, cache source flags, candidate data source call flags, and quality constraint version flags.

[0033] Furthermore, the cross-domain query execution feedback in S9 includes source unreachability information, field schema drift information, permission tag change information, cache invalidation information, response latency anomaly information, query failure information, and field quality degradation information; the cross-domain quality profile is updated according to the following formula: In the formula, This indicates the updated cross-domain quality profile. This indicates the update step size corresponding to the cross-domain query execution feedback type. This indicates feedback from the cross-domain query execution. The resulting profile correction items are as follows: source unreachability information is converted into interface reachability correction items, field pattern drift information is converted into pattern consistency correction items, cache invalidation information is converted into cache validity correction items, permission tag change information is converted into cross-domain permission tag consistency correction items, and query failure information is converted into historical execution stability correction items. Profile correction items unrelated to the requester domain are used to update the basic quality profile, and profile correction items related to the requester domain are used to update the cross-domain quality profile. The virtualization engine regenerates the quality constraint description based on the updated basic quality profile and cross-domain quality profile, and updates the virtual data object metadata and subsequent cross-domain query execution plan based on the regenerated quality constraint description.

[0034] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention converts quality collection information into a basic quality profile, and then combines it with the requester domain context to generate a cross-domain quality profile. This enables the same virtual data object to form differentiated quality views for different requester domains. It can introduce requester domain permission tags, requester domain timeliness requirements, and result granularity requirements into the cross-domain quality judgment process. This helps to improve the adaptability between field calls, data source calls, and result granularity processing methods and requester domain requirements in a multi-cloud heterogeneous environment, and reduces the situation of incorrect field calls or result granularity mismatch caused by ignoring the differences between requester domains.

[0035] 2. This invention converts cross-domain quality profiles into quality constraint descriptions that include field constraints, data source constraints, and execution constraints. These descriptions are then parsed and executed by the virtualization engine. This allows the quality assessment results to directly impact virtual data object metadata updates and cross-domain query execution plan rewriting. The quality assessment results are no longer merely used as a basis for quality reports, governance recommendations, or data asset ranking; instead, they can enter the query execution chain of the virtualization engine, which helps improve the ability of the quality assessment results to constrain the cross-domain data access process.

[0036] 3. When generating or rewriting a cross-domain query execution plan, this invention determines the field transformation operator, physical data source call path, cached data source call path, or candidate data source call path based on the quality constraint description. It also adjusts the query plan based on cache validity conditions and failure fallback paths. Furthermore, it dynamically adjusts the cross-domain query execution path by combining the data source running status, cache version matching relationship, data synchronization status, semantic mapping version status, and permission tag consistency. This helps reduce the situation where invalid fields, unreachable data sources, or expired caches are continued to be called when the data source is unreachable, the cache is invalid, or the field mode drifts.

[0037] 4. Based on cross-domain query execution feedback, this invention converts source unreachability information, field pattern drift information, permission tag change information, cache invalidation information, response latency anomaly information, query failure information, or field quality degradation information into profile correction items, and updates the basic quality profile and cross-domain quality profile. In turn, it regenerates the quality constraint description and updates the subsequent cross-domain query execution plan. This enables the virtualized query execution process to be continuously corrected according to the changes in the running status of multi-cloud heterogeneous data sources, providing a technical basis for reducing the risk of cross-domain query failure and inconsistent results.

[0038] 5. After performing cross-domain data access, this invention generates a result trust identifier carrying field transformation markers, cache source markers, candidate data source call markers, and quality constraint version markers according to the result trust identifier generation rules. The trust identifier can characterize the field transformation, cache source, candidate data source call, and quality constraint version involved in the query result, which helps the requesting domain to identify the source, transformation process, and execution basis of the query result, and improves the identifiability of the result trust boundary. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to the present invention. Figure 2 This is a diagram illustrating the correspondence between virtual data objects and multi-cloud heterogeneous data sources. Figure 3 A schematic diagram illustrating the generation of basic quality profiles, cross-domain quality profiles, and quality constraint descriptions; Figure 4 A diagram illustrating the rewriting of the execution plan for cross-domain queries; Figure 5 A diagram illustrating the closed-loop process of cross-domain query execution feedback update. Detailed Implementation

[0040] To make the above-mentioned objectives, features, and advantages of the present invention more apparent and understandable, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the following embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can adaptively adjust the data source name, field name, statistical window, detection period, and parameter values ​​according to different multi-cloud deployment environments, data source types, access interface protocols, permission tag systems, and field structures.

[0041] Example 1, referring to Figures 1-5 This is the first embodiment of the present invention, which provides a method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment.

[0042] This embodiment illustrates the application scenario of cross-cloud business analysis of sales, inventory, and logistics data for a group enterprise. The group enterprise's sales system is deployed in the first cloud domain, the inventory system in the second cloud domain, the logistics system in the third cloud domain, and the business analysis system in the fourth cloud domain. The sales system includes a sales order database, the inventory system includes an inventory detail database and an inventory synchronization copy data source, and the logistics system includes a logistics status interface and a logistics cache data source.

[0043] The business analysis system initiates cross-domain query requests through the virtualization engine to obtain data such as order amount, inventory quantity, and shipping status.

[0044] Specifically, the sales order database is accessed via an SQL interface, the inventory details database via a data lake interface, the logistics status interface via a REST interface, and the logistics cache data source via an object storage interface. The order-related fields in the different data sources are named `order_id`, `sales_order_no`, and `tracking_order_code`, respectively; the field names differ but their business semantics are related. The inventory quantity field is `stock_qty`, and the delivery status fields include `delivery_status` and `cached_delivery_status`.

[0045] The field schema, update time, access interface, cache status, permission tag, and lineage record of the above fields are all obtained from the metadata, interface probe results, cache logs, and historical query logs of the corresponding data source.

[0046] S1: Generate virtual data objects and establish corresponding relationships: Reference Figure 1 and Figure 2 The virtualization engine generates virtual data objects for heterogeneous data sources in multiple cloud domains registered to the virtualization engine, and establishes correspondences between virtual data objects and field mapping relationships, physical data source identifiers, cloud domain identifiers, access interface protocols, permission tags, lineage identifiers, cache version identifiers, and candidate data source association information.

[0047] Specifically, the virtualization engine reads the data source registration information from the sales order database, inventory detail database, inventory synchronization copy data source, logistics status interface, and logistics cache data source to generate virtual data objects for business analysis. These virtual data objects include virtual fields such as order number, order amount, inventory quantity, shipment status, logistics update time, and region identifier.

[0048] The virtualization engine establishes a field mapping relationship between the order number field in the business analysis virtual data object and the order_id field in the sales order database, the sales_order_no field in the inventory details database, and the tracking_order_code field in the logistics status interface.

[0049] Establish a field mapping relationship between the inventory quantity field and the stock_qty field in the inventory details database, as well as the sync_stock_qty field in the inventory synchronization copy data source.

[0050] Establish a field mapping relationship between the shipping status field and the delivery_status field in the logistics status interface, as well as the cached_delivery_status field in the logistics cache data source.

[0051] The virtualization engine records the cloud domain, access interface protocol, source data update information, cache version identifier, and lineage identifier of each physical data source.

[0052] For example, the sales order database corresponds to the first cloud domain and the SQL interface, the inventory details database corresponds to the second cloud domain and the data lake interface, the logistics status interface corresponds to the third cloud domain and the REST interface, and the logistics cache data source corresponds to the third cloud domain and the object storage interface.

[0053] The virtualization engine also records the permission tags corresponding to virtual data objects, fields within virtual data objects, and fields in the physical data source.

[0054] For example, the business analysis virtual data object is labeled as internal business analysis, the order amount field is labeled as internal business field, and the shipment status field is labeled as internal logistics field.

[0055] The permission labels for physical data source fields come from the data directory, field metadata, or data source permission tags, and are not manually specified temporarily during query execution.

[0056] S2: Obtain quality acquisition information: Reference Figure 1 The virtualization engine acquires quality acquisition information associated with virtual data objects. This quality acquisition information includes field status information, data source runtime status information, cache status information, source-side permission tag consistency information, and historical execution status information.

[0057] Specifically, field status information is obtained from field statistical tasks, field schema records, and field sampling results. Field status information includes field missing rate, format error rate, field schema version, field semantic identifier, and field update time.

[0058] The field missing rate is denoted as . This represents the ratio of the number of blank records to the total number of records in the sampled data or full statistics; the format error rate is denoted as... This represents the ratio of the number of records that do not conform to the field type, field format, enumeration constraint, or format validation rule to the total number of records.

[0059] For example, if the virtualization engine samples 1000 records from the stock_qty field of the inventory quantity, and 12 records are missing and 8 records are malformed, then the field missing rate is... The format error rate is 0.012. It is 0.008.

[0060] The above values ​​are merely illustrative data to illustrate the calculation process. The actual sample size and statistical window can be determined by the data source size, data update cycle, and system sampling strategy.

[0061] Data source runtime status information is obtained from interface probing tasks, access logs, and source version records. This information includes interface return status, response latency, access failure records, source version identifier, source data update time, semantic mapping version status, and data synchronization status. The source data update time is denoted as... .

[0062] For example, the virtualization engine initiates a lightweight probe request to the logistics status interface, recording the interface return status, response latency, and access failure records; reads the synchronization logs from the inventory synchronization replica data source, recording the data synchronization status; and reads the semantic mapping version status from the semantically compatible data source to determine whether the field semantic mapping rules are valid.

[0063] Cache status information is obtained from cache metadata and cache generation logs. For example, the logistics cache data source records the cache version identifier cache_v12, cache generation time, cache validity period, cache source identifier, and source data version matching relationship.

[0064] If the source data version corresponding to the cached version is the same as the current source version of the logistics status interface, the source data version matching relationship is recorded as a match; if the source data has been updated but the cached version has not been updated, it is recorded as a mismatch.

[0065] Source-side permission tag consistency information is obtained by comparing the permission tags of virtual data objects with the permission tags of physical data source fields.

[0066] For example, if the delivery status field in the virtual object is an internal logistics field, and the delivery_status field in the physical data source is also an internal logistics field, then the source-side permission label consistency record is a match; if the physical field label is a restricted logistics field and the virtual field label is an internal logistics field, and there is no allowed mapping relationship, then the record is a mismatch.

[0067] Historical execution status information is obtained from cross-domain query logs. For example, the virtualization engine counts the number of failed calls to the logistics status interface, the failure type, the response latency exception records, and the failure rollback records within the most recent execution window. The execution window can be determined based on the system log retention period and query frequency, and is not a fixed limitation.

[0068] S3: Generate a basic quality profile: Reference Figure 1 and Figure 3 The virtualization engine generates a basic quality profile associated with virtual data objects, fields, and physical data sources based on the correspondence and quality collection information.

[0069] Specifically, the basic quality profile reflects the objective quality status on the source side between virtual fields and physical data sources, and does not depend on a specific request domain. The basic quality profile is represented as follows: In the formula, Indicates time Below and virtual data objects Fields and physical data sources The associated basic quality profile, Indicates structural integrity. Indicates pattern consistency. Indicates the basic semantic matching degree. Indicates basic timeliness. Indicates interface reachability. Indicates cache validity. This indicates consistency of the source-side permission tags. Indicates the integrity of bloodline. This indicates historical execution stability.

[0070] in, Determined by field missing rate and formatting error rate. Field missing rate The format error rate is obtained by comparing the number of blank records to the total number of records in the sampled data or full statistics. The structural integrity score is calculated as the ratio of the number of records that do not conform to the field type, field format, enumeration constraints, or format validation rules to the total number of records. The structural integrity score is expressed as: For example, the field missing rate of the inventory quantity field. The format error rate is 0.012. If it is 0.008, then .

[0071] in and All data were obtained from field statistics tasks.

[0072] The schema version is determined by the field schema version, field type compatibility, and records of successful field conversions. If the schema version of the physical data source field is consistent with the schema version of the virtual data object field, If the two are inconsistent but an executable format conversion rule exists, then... The success rate of this format conversion rule in historical execution records is used to determine the outcome; if the field does not exist or the field type cannot be converted, then... Take 0.

[0073] It is determined by the field semantic identifier, semantic mapping version status, and semantic transformation record.

[0074] If the semantic identifier of the physical data source field is consistent with the semantic identifier of the virtual data object field, If the two are inconsistent but a valid semantic mapping version state exists, and matching can be achieved through unit conversion, format conversion, or semantic mapping relationship, then... The determination is based on the validity of the semantic mapping version and the historical conversion power; if the semantic mapping version is invalid or there is no executable semantic mapping relationship, Take 0.

[0075] It is determined by the current time and the source data update time. Let the current time be denoted as . The source data update time is Source data age The basic allowed age for data is The basic timeliness is then expressed as: in, It is determined by the data source update cycle, historical update frequency, or data source metadata.

[0076] For example, if the historical update cycle of a certain inventory data source is 30 minutes, and the current time differs from the source data update time by 12 minutes, then... .

[0077] The result is determined by the interface probe results and the response latency. Let (I_{reach}) represent the interface probe result; when the interface probe is successful... When interface detection fails ;remember This indicates a delay in the response to this detection. The maximum allowable response latency, determined based on historical response latency distribution, interface service level information, or operational monitoring records, is represented as follows: When the interface is unreachable , The value is 0; when the interface is reachable and the response latency is lower than the maximum allowable response latency, A continuous evaluation value is generated based on the magnitude of the response delay.

[0078] It can be determined by the quantile value of historical response delay, interface service level information, or operation monitoring records.

[0079] It is determined by the cache version identifier, cache generation time, cache validity period, and source data version matching relationship.

[0080] remember This indicates whether the cached version matches the source data version. If they match, then... When there is a mismatch ;remember Indicates the cache generation time. If the cache expiration period is indicated, then cache validity is expressed as: When the cached version does not match the source version , The value is 0; when the cache version matches and the cache generation time has not exceeded the cache validity period, A continuous evaluation value is generated based on the cache generation time and cache validity period.

[0081] It is determined by the matching result between the permission tags of the virtual data object and the permission tags of the physical data source field.

[0082] If the two are consistent or satisfy the permission tag mapping relationship. Take 1; if the two are inconsistent and there is no allowed mapping relationship, Take 0.

[0083] Permission tag consistency is used to generate profiles and quality constraints, but does not replace access permission verification.

[0084] It is determined by lineage identifier, cache source identifier, synchronized copy source identifier, and candidate alternative source identifier. If the field source, transformation process, cache generation source, and candidate alternative source can all be traced, then... Take the higher value; if the bloodline is broken or the source cannot be traced, then reduce.

[0085] In one implementation, It is determined by the ratio of the number of traceable bloodline nodes to the number of bloodline nodes that should be traced.

[0086] It is determined by the cross-domain query failure rate, response latency anomaly rate, and failure rollback rate.

[0087] remember This indicates the cross-domain query failure rate. Indicates the response delay anomaly rate. If the failure rollback rate is represented, then the historical execution stability is expressed as: in, , and All of these statistics are derived from historical execution logs.

[0088] S4: Receive cross-domain query requests and obtain the requester's domain context.

[0089] Reference Figure 1 and Figure 3 The business analysis system initiates a cross-domain query request, requesting order amount, inventory quantity, and delivery status. Access permission verification is performed by the existing identity authentication or cross-domain access control module. In this embodiment, quality profiling and quality constraints do not replace access permission verification; they only participate in virtualized query execution control after access permission is granted.

[0090] Specifically, after a request passes access permission verification, the virtualization engine obtains the requester's domain context. The requester's domain context includes the requester's domain permission tag, the requester's domain timeliness requirements, and the result granularity requirements.

[0091] For example, the requesting domain permission label is "Internal Business Analysis Domain"; the requesting domain timeliness requirements include the maximum age of data allowed in the requesting domain, denoted as... For example, the corresponding logistics delivery status data The timeframe is 15 minutes; the result granularity requirement is to return aggregated results by region and date, rather than returning all logistics details for each order.

[0092] The aforementioned request domain context comes from the request metadata carried in the cross-domain query request, the request domain attributes output by the access permission verification, the request domain description registered in the data sharing directory, or the request domain registration information maintained in the virtualization engine.

[0093] S5: Generate cross-domain quality profiles: Reference Figure 3The virtualization engine generates cross-domain quality profiles associated with virtual data objects, fields, physical data sources, and requesting domains based on the basic quality profile, requesting domain context, and corresponding relationships.

[0094] Specifically, the cross-domain quality profile is generated from the base quality profile and the requester's domain context, and is represented as follows: The vector structure representation of cross-domain quality profiling is as follows: In the formula, Indicates time Below and virtual data objects Fields Physical data source and requesting domain Related cross-domain quality profiles Indicates the requester's domain context. This indicates a portrait generation function that incorporates the requester's domain context into the base quality portrait.

[0095] function Correct the consistency of source-side permission tags based on the requester's domain permission tags. Generate consistent cross-domain permission tags The basic timeliness is adjusted according to the timeliness requirements of the requesting domain. Generate cross-domain timeliness Adjust the basic semantic matching degree according to the result granularity requirements. Generate cross-domain semantic matching degree .

[0096] In one implementation, the image generation function Follow these steps: Read the requester domain permission tags, requester domain timeliness requirements, and result granularity requirements from the requester domain context; match the requester domain permission tags with the virtual data object permission tags and the physical data source field permission tags respectively. If the three are consistent or satisfy the preset permission tag mapping relationship, generate cross-domain permission tag consistency in the callable state; if they are inconsistent and there is no allowed mapping relationship, generate cross-domain permission tag consistency in the non-callable state; calculate the source data age based on the maximum data age and source data update time in the requester domain timeliness requirements, and generate cross-domain timeliness; based on the result granularity requirements, field semantic identifiers, field pattern versions, and semantic mapping version status, determine whether the field can meet the requester domain result granularity requirements through field aggregation operators, format conversion operators, or granularity adjustment operators. If it can, generate the corresponding cross-domain semantic matching degree and write the corresponding operator into the field constraint item; if it cannot, reduce the cross-domain semantic matching degree or write the corresponding field into the prohibited call field set.

[0097] for Let the maximum allowed data age in the requesting domain be . The age of the source data is Then, cross-domain timeliness is expressed as: For example, the requesting domain requires the maximum data age of logistics status data. If the time interval is 15 minutes, and the current time differs from the source data update time by 10 minutes, then... If the age of the source data is 5 minutes, then .

[0098] This calculation is used to form a continuous timeliness evaluation. The larger the value, the further away from the upper limit of the timeliness of the requesting domain. In actual implementation, whether the timeliness requirements of the requesting domain are met can also be directly used as part of the cache validity conditions. The continuous timeliness evaluation is used for candidate path sorting, field constraint generation, or execution constraint generation.

[0099] for The virtualization engine compares the granularity of the results with the field semantic identifier, field schema version, and semantic mapping version status. If the field semantic granularity is consistent with the granularity of the requester's domain result, then... If a field can meet the requester's domain result granularity requirements through field aggregation operators, format conversion operators, or granularity adjustment operators, then The determination is based on the operator's availability and historical execution success rate; if the field semantic granularity cannot be converted to the requester's domain result granularity, then... Set to 0 or reduce to an uncallable state.

[0100] For example, if the requesting domain requires the return of shipping status statistics by region and date, and if the shipping status field can generate corresponding statistics using the region aggregation operator, then... Keep it available and write the region aggregation operator into the field constraint.

[0101] for The virtualization engine compares the matching results between the requester's domain permission tags, the virtual data object permission tags, and the physical data source field permission tags. If the three match or satisfy the tag mapping relationship, then... Set to 1; if the requester's domain permission label cannot cover the virtual data object permission label or the physical data source field permission label, then Set to 0 or reduce to an uncallable state.

[0102] Used for generating subsequent field constraints and data source constraints, not for bypassing access permission checks.

[0103] Meanwhile, the requester's domain timeliness requirement is also passed to the execution constraint item to generate cache validity conditions; the result granularity requirement is also passed to the field constraint item to determine the field aggregation operator or granularity adjustment operator.

[0104] Therefore, the requester's domain context It participates in both cross-domain quality profile generation and quality constraint description generation.

[0105] S6: Convert cross-domain quality profiles into quality constraint descriptions: Reference Figure 3 and Figure 4 The virtualization engine converts cross-domain quality profiles into quality constraint descriptions that can be parsed and executed by the virtualization engine.

[0106] Specifically, quality constraints are described as parsable data structures within the virtualization engine.

[0107] Quality constraint descriptions include field constraint items. Data source constraints and execution constraints And expressed as: In the formula, Represents virtual data objects In the requesting domain The following quality constraint description, Represents a collection of callable fields. This indicates that accessing the field collection is prohibited. Represents a set of field transformation operators. Indicates the collection of physical data sources that can be invoked. This indicates a collection of cached data sources that can be invoked. This indicates the collection of candidate data sources that can be invoked. Indicates cache validity conditions. Indicates the fallback path in case of failure. This indicates the rules for generating a reliable identifier for the result.

[0108] Field constraints Used to generate field call tags, field mask tags, and field transformation tags; data source constraints. Used to generate physical data source call paths, cached data source call paths, and candidate data source call paths; execute constraint terms. Used to generate cache validity judgment nodes, failure rollback nodes, and result trust identifier generation nodes.

[0109] Quality constraint descriptions can be implemented using key-value structures, JSON structures, intermediate representation structures of execution plans, or internal object structures of the virtualization engine.

[0110] The above set is generated based on field pattern compatibility, permission tag matching results, interface detection results, cache version matching, requester domain timeliness requirements, result granularity requirements, lineage records, and historical execution records.

[0111] For example, if the structural integrity, schema consistency, cross-domain permission tag consistency, and historical execution stability of the order amount field all meet the execution requirements, then the order amount field will be written... .

[0112] If the cross-domain permission tag consistency of a certain logistics detail field does not meet the requesting domain requirements, then that field will be written to... .

[0113] If the shipping status field can meet the result granularity requirements through region-level aggregation, then the field aggregation operator is written. .

[0114] For data source constraints, if the sales order database interface is reachable and historical execution is stable, then the sales order database will be written to. .

[0115] If the cache version, cache generation time, source data update time, requester domain timeliness requirements, and cache source identifier of the logistics cache data source all meet the cache validity conditions, then the logistics cache data source will be written to... .

[0116] If the data synchronization status and source version identifier of the inventory synchronization replica data source meet the calling conditions, then the inventory synchronization replica data source will write... .

[0117] For execution constraints, Record cache version determination, cache generation time determination, source data update time determination, and requester domain timeliness requirement determination; Record the cached data source call path, candidate data source call path, and path that is rejected and triggers quality reassessment after a physical data source call failure; Rules for generating trusted identifiers for recorded results.

[0118] S7: Define field constraints, data source constraints, and execution constraints: The virtualization engine determines the set of callable fields and field transformation methods based on field constraints, determines the data source to be called based on data source constraints, including physical data sources, cached data sources or candidate data sources, and determines cache validity conditions, failure rollback paths and result trust identifier generation rules based on execution constraints.

[0119] Specifically, the virtualization engine inserts field transformation operators into the cross-domain query execution plan based on field constraints, and writes the physical data source call path, cached data source call path, or candidate data source call path into the cross-domain query execution plan based on data source constraints.

[0120] Field transformation operators include field trimming operators, field desensitization operators, field aggregation operators, format conversion operators, and granularity adjustment operators.

[0121] The candidate data source call path is generated from the candidate data sources that satisfy the field mapping relationship, source side permission tag consistency information, cross-domain permission tag consistency, data source running status information, and the version status, semantic mapping version status, data synchronization status or cache status corresponding to the candidate data source type.

[0122] The field constraint items determine the set of callable fields, the set of prohibited fields, and the set of field transformation operators based on field status information, source-side permission label consistency information, cross-domain permission label consistency, and cross-domain quality profile.

[0123] The virtualization engine updates the field call flags, field mask flags, and field transformation flags in the virtual data object metadata according to the field constraints, and inserts field transformation operators into the cross-domain query execution plan.

[0124] The data source constraints determine the physical data source set, cache data source set, and candidate data source set based on data source running status information, cache status information, source-side permission tag consistency information, cross-domain permission tag consistency, and cross-domain quality profile.

[0125] The execution constraints determine cache validity conditions, failure fallback paths, and result trust identifier generation rules based on cache status information, data source runtime status information, and historical execution status information. Cache validity conditions are formed by the cache version identifier, cache generation time, source data update time, requester domain timeliness requirements, and cache source identifier.

[0126] When a physical data source call fails, the fallback path generates subsequent execution paths in the order of cached data source call path, candidate data source call path, and path that is rejected and triggers quality reassessment.

[0127] The result trust identifier generation rule generates a result trust identifier that carries field transformation markers, cache source markers, candidate data source call markers, and quality constraint version markers.

[0128] S8: Update virtual data object metadata and generate or rewrite cross-domain query execution plans: Reference Figure 4 The virtualization engine updates the metadata of virtual data objects based on the quality constraint description and generates or rewrites cross-domain query execution plans.

[0129] Specifically, the virtualization engine updates the field call flag, field mask flag, and field transformation flag in the virtual data object metadata based on the field constraints.

[0130] Field call flags are used to indicate whether a field can participate in this cross-domain query; field mask flags are used to indicate whether a field is excluded from the query results; field transformation flags are used to indicate whether a field needs to be inserted with field pruning operators, field desensitization operators, field aggregation operators, format conversion operators, or granularity adjustment operators.

[0131] The cross-domain query execution plan includes physical data source read nodes, cached data source read nodes, candidate data source read nodes, field transformation nodes, cache validity judgment nodes, failure rollback nodes, and result trust identifier generation nodes.

[0132] In this embodiment, the original query plan includes: reading the order amount from the sales order database, reading the inventory quantity from the inventory details database, and reading the shipping status from the logistics status interface.

[0133] If the virtualization engine determines that the logistics status interface is unreachable based on the quality constraint description, but the logistics cache data source meets the cache validity conditions, then the virtualization engine will replace the logistics status interface read node with the logistics cache data source read node.

[0134] If a field schema drift occurs in the inventory details database, but the data synchronization status and source version identifier of the inventory synchronization replica data source meet the calling conditions, the virtualization engine will replace the inventory details database read node with the inventory synchronization replica data source read node.

[0135] If the requesting domain requires both regional and date-level results, the virtualization engine inserts a field aggregation operator in the execution branch corresponding to the shipping status field.

[0136] If the cross-domain permission tag consistency requirement for a certain field is not met, the virtualization engine writes a field masking flag in the virtual object metadata and removes the field from the projection field set of the execution plan.

[0137] S9: Perform cross-domain data access and update based on execution feedback: Reference Figure 4 and Figure 5 The virtualization engine executes cross-domain data access according to the rewritten cross-domain query execution plan.

[0138] After execution, the virtualization engine generates a result trust identifier according to the result trust identifier generation rules, obtains the cross-domain query execution feedback, and updates the basic quality profile, cross-domain quality profile, and quality constraint description based on the cross-domain query execution feedback.

[0139] Specifically, the result trust identifiers include field transformation markers, cache source markers, candidate data source call markers, and quality constraint version markers.

[0140] For example, if the order amount field comes from the sales order database, the inventory quantity field comes from the inventory synchronization copy data source, and the shipping status field comes from the logistics cache data source, and the shipping status field has undergone regional-level aggregation processing, then the result trust identifier records the candidate data source call flag for the inventory quantity field, the cache source flag for the shipping status field, the field aggregation flag for the shipping status field, and the quality constraint version flag used in this query.

[0141] The requesting domain can identify whether a field has been transformed, whether it comes from a cache, whether a candidate source call has occurred, and the constraint version on which the execution plan is based, based on the result trust identifier.

[0142] Cross-domain query execution feedback includes information such as source unreachability, field schema drift, permission tag change, cache invalidation, response latency exception, query failure, and field quality degradation.

[0143] In this embodiment, the image correction function Based on whether the cross-domain query execution feedback is related to the requesting domain, it is categorized into basic profile correction functions. and cross-domain portrait correction function .

[0144] in, This indicates an execution response that is independent of the requesting domain. This indicates the execution feedback related to the requesting domain.

[0145] Cross-domain quality profiles are updated according to the following formula: In the formula, This indicates the updated cross-domain quality profile. This indicates the update step size corresponding to the cross-domain query execution feedback type. This indicates feedback from the cross-domain query execution. The resulting image correction items.

[0146] In the formula, This indicates the updated base quality profile. This indicates the update step size of the basic quality profile. This represents the base profile correction item obtained from the execution feedback transformation independent of the requesting domain. This indicates an execution response that is independent of the requesting domain.

[0147] and Each outputs a correction vector with the same dimensions as the corresponding image.

[0148] Dimensions not affected by this feedback will retain their corresponding values ​​from the previous profile; dimensions affected by the feedback will generate new target correction values ​​based on the feedback type.

[0149] Source unreachability information is converted into interface reachability correction items. For example, if the logistics status interface is unreachable in this execution, then the relevant information related to the logistics status interface will be corrected. reduce.

[0150] Field schema drift information is converted into schema consistency correction items. For example, if an inventory quantity field changes from numeric to string and the current format conversion rule cannot handle it, then... reduce.

[0151] Cache invalidation information is converted into cache validity correction items. For example, if the logistics cache version does not match the source data version, then... reduce.

[0152] Permission label change information is converted into corresponding correction items based on the object of the change; when the permission label of a virtual data object or the permission label of a physical data source field changes, it is converted into a source-side permission label consistency correction item or a cross-domain permission label consistency correction item; when the permission label of the requesting domain changes, it is converted into a cross-domain permission label consistency correction item.

[0153] Response latency anomaly information is converted into interface reachability correction items or historical execution stability correction items.

[0154] Field quality degradation information is converted into structural integrity correction items, basic semantic matching degree correction items, or historical execution stability correction items.

[0155] Query failure information is converted into historical execution stability corrections.

[0156] When multiple feedback events occur simultaneously, the virtualization engine generates corresponding profile correction items for each event, and synthesizes profile correction items based on the feedback type, feedback severity, and number of times the feedback occurs.

[0157] Update step size and Determined based on the type and severity of feedback.

[0158] The severity of the feedback is determined by the error type, the number of consecutive occurrences, the number of fields affected, the number of data sources affected, or whether it caused the query to fail.

[0159] Source unreachability, field schema drift, permission label changes, and cache invalidation correspond to larger update steps; minor response delays or occasional access failures correspond to smaller update steps.

[0160] For data sources with high update frequency, the update step size for a single feedback is reduced to minimize the impact of short-term fluctuations on subsequent execution plans; for data sources with low update frequency and severe feedback, the update step size for a single feedback is increased to accelerate the correction of subsequent execution plans.

[0161] The virtualization engine regenerates the quality constraint description based on the updated base quality profile and cross-domain quality profile, and updates the virtual data object metadata and subsequent cross-domain query execution plans based on the regenerated quality constraint description.

[0162] Example 2, refer to Figures 1-5 This is the second embodiment of the present invention, which differs from the first embodiment in that the application scenario is cross-cloud sharing of industrial equipment operation monitoring data.

[0163] Specifically, the equipment monitoring system is deployed in the first cloud domain, the equipment alarm system in the second cloud domain, the maintenance record system in the third cloud domain, and the headquarters operations and maintenance analysis system in the fourth cloud domain. The equipment monitoring system includes a time-series database, the equipment alarm system includes a message queue and alarm synchronization copy data source, and the maintenance record system includes a relational database. The headquarters operations and maintenance analysis system uses a virtualization engine to query equipment temperature, equipment vibration, alarm status, and maintenance records across domains.

[0164] Reference Figure 2 In S1, the virtualization engine generates virtual data objects for equipment operation and maintenance, and establishes a mapping relationship between virtual fields such as equipment number, equipment temperature, equipment vibration, alarm status and maintenance records and fields of each physical data source.

[0165] Specifically, equipment temperature and equipment vibration are high-frequency time-series data, alarm status is message queue data, and maintenance records are low-frequency business records.

[0166] In S2, the virtualization engine performs interface probing on the time-series database to obtain interface reachability and response latency; reads message backlog status and consumption latency from the message queue, which serve as a type of data synchronization status; reads replica version identifier and synchronization latency from the alarm synchronization replica data source; and reads field schema version and field update time from the maintenance record database.

[0167] Reference Figure 3 In S3, the virtualization engine generates a basic quality profile based on the quality acquisition information.

[0168] In S4, after the headquarters operations and maintenance analysis system initiates a cross-domain query request and passes the access permission verification, the virtualization engine obtains the operations and maintenance domain permission tags, the timeliness requirements of device status data, and the granularity requirements of the results summarized by device number.

[0169] In S5, the virtualization engine combines the aforementioned requester domain context to generate a cross-domain quality profile.

[0170] Specifically, if the time-series database interface is unreachable but the cache version, cache generation time, and source data version of the edge cache data source meet the timeliness requirements, then in S7, the virtualization engine will switch the device temperature reading path from the time-series database to the edge cache data source.

[0171] If there is a consumption delay in the message queue, but the replica version identifier and synchronization delay of the alarm synchronization replica data source meet the requirements, the virtualization engine will switch the alarm status reading path to the alarm synchronization replica data source.

[0172] If the maintenance record field can be aggregated by device number and date, the virtualization engine in S8 inserts a field aggregation operator in the maintenance record branch.

[0173] Reference Figure 5 In S9, after execution, the virtualization engine converts time-series database unreachability, message queue latency anomalies, cache invalidation, or maintenance field pattern drift into corresponding profile correction items, updates the basic quality profile and cross-domain quality profile, and updates the quality constraint description and execution plan for subsequent equipment operation and maintenance queries accordingly. The remaining steps are the same as those in Example 1.

[0174] Example 3, referring to Figures 1-5 This is the third embodiment of the present invention, which differs from embodiment 1 in that the application scenario is cross-domain sharing of scientific research experimental data.

[0175] Specifically, experimental sample data is stored in the experimental database of the first cloud domain, experimental image feature data is stored in the object storage of the second cloud domain, model calculation results are stored in the data lake table of the third cloud domain, and the scientific research collaboration analysis system is deployed in the fourth cloud domain.

[0176] The scientific research collaboration analysis system uses a virtualization engine to query experimental sample numbers, experimental conditions, image features, model output results, and statistical summary information across domains.

[0177] Reference Figure 2 In S1, the virtualization engine generates virtual data objects for scientific research analysis and establishes field mapping relationships between virtual fields such as sample number, experimental conditions, image features and model output results and each physical field.

[0178] Specifically, the sample ID field in the experimental database is sample_id, the sample field in the object storage feature index is image_sample_code, and the sample field in the data lake table is model_sample_key.

[0179] Experimental conditions may have unit differences, model results may have version differences, and image feature data may only be suitable for returning statistical results and not detailed results.

[0180] In S2 and S3, the virtualization engine reads field semantic identifiers, semantic mapping version status, field pattern version, cache version identifiers, lineage identifiers, and permission tags, and generates a basic quality profile based on these.

[0181] Reference Figure 3 In S4, the scientific research collaboration analysis system initiates a cross-domain query request, and the virtualization engine obtains the scientific research collaboration domain permission tags, model result timeliness requirements, and statistical granularity requirements.

[0182] In S5, the virtualization engine introduces the result granularity requirements into the cross-domain quality profile and further passes them to field constraints to determine the field aggregation operator or granularity adjustment operator.

[0183] Specifically, if the experimental condition field requires desensitization, the virtualization engine in S7 will write the field desensitization operator into the field transformation operator set. If the image feature cache meets the cache validity conditions, the data source call path of the image feature field will be written into the cache data source call path.

[0184] If the model result data lake table is unreachable, but the replica version identifier and synchronization delay of the synchronous replica data source meet the requirements, then the data source call path of the model output result field is written into the candidate data source call path.

[0185] Reference Figure 5 In step S9, after execution, the virtualization engine generates a result trust identifier. This result trust identifier records the field anonymization flag for the experimental conditions field, the cache source flag and field aggregation flag for the image feature field, the candidate data source call flag for the model output result field, and the quality constraint version flag. The remaining steps are the same as in Example 1.

[0186] Example 4, refer to Figures 1-5 This is the fourth embodiment of the present invention. The difference between this embodiment and embodiment 1 is that it focuses on the method of generating the candidate data source call path.

[0187] Specifically, the candidate data source is not an arbitrary backup source, but a callable data source obtained by the virtualization engine after judging the status according to the candidate data source type.

[0188] Candidate data source types include cached data sources, mirrored data sources, synchronous replica data sources, semantically compatible data sources, and candidate alternative data sources.

[0189] Reference Figure 2 and Figure 4 In S7, the virtualization engine determines the corresponding judgment criteria based on the candidate data source type.

[0190] Specifically, the criteria for determining a cached data source include cache version, cache generation time, cache validity period, and source data version matching; the criteria for determining a mirrored data source include mirror version identifier, mirror synchronization time, and source version consistency; the criteria for determining a synchronized replica data source include replica version identifier, synchronization delay, and update log continuity; the criteria for determining a semantically compatible data source include field schema compatibility, field semantic identifier matching, semantic mapping version status, and unit or format conversion relationship; and the criteria for determining a candidate alternative data source include field mapping relationship, permission tag consistency, interface reachability, and historical call stability.

[0191] The virtualization engine filters candidate data sources based on the above criteria, and only data sources that meet the corresponding criteria are written into the candidate data source call path.

[0192] In S8, the virtualization engine configures a failure rollback node in the cross-domain query execution plan; in S9, when a physical data source call fails, the failure rollback node generates subsequent execution paths in the order of cached data source call path, candidate data source call path, and path that is rejected and triggers quality reassessment. The remaining steps are the same as those in Example 1.

[0193] Example 5, refer to Figures 1-5 This is the fifth embodiment of the present invention. The difference between this embodiment and embodiment 1 is that it focuses on explaining the method of generating the reliable identifier of the result.

[0194] Specifically, in S8, the virtualization engine configures the result trusted identifier generation node in the cross-domain query execution plan; in S9, after the cross-domain query is executed, the result trusted identifier generation node generates the result trusted identifier according to the result trusted identifier generation rules.

[0195] The results trust identifiers include field transformation markers, cache source markers, candidate data source call markers, and quality constraint version markers.

[0196] Reference Figure 4 and Figure 5 In the scenario of group enterprise operation analysis, the order amount field comes from the sales order database, the inventory quantity field comes from the inventory synchronization copy data source, the delivery status field comes from the logistics cache data source, and the delivery status field has undergone regional aggregation processing.

[0197] The reliability identifiers generated by the virtualization engine include candidate data source call flags for the inventory quantity field, cache source flags for the shipping status field, field aggregation flags for the shipping status field, and quality constraint version flags used in this query.

[0198] Specifically, in the scenario of industrial equipment operation monitoring, the equipment temperature field comes from the edge cache data source, the alarm status field comes from the synchronous replica data source, and the maintenance record field comes from the relational database and has undergone date-granularity aggregation processing.

[0199] The trusted identifiers generated by the virtualization engine include a cache source flag for the device temperature field, a candidate data source call flag for the alarm status field, a granularity adjustment flag for the maintenance record field, and a quality constraint version flag.

[0200] Specifically, in the scenario of cross-domain sharing of scientific research experimental data, the experimental condition field is processed by the desensitization operator, the image feature field comes from the cached data source and is processed by the aggregation operator, and the model output result field comes from the synchronous copy data source.

[0201] The reliability identifiers generated by the virtualization engine include field anonymization tags for the experimental conditions field, cache source tags and field aggregation tags for the image feature field, candidate data source call tags for the model output result field, and quality constraint version tags. The remaining steps are the same as those in Example 1.

[0202] Based on embodiments 1-5, the working principle of this invention is as follows: The virtualization engine first generates virtual data objects for multi-cloud heterogeneous data sources and establishes the correspondence between virtual fields, physical fields, cloud domains, interfaces, permission tags, lineage, caches, and candidate data sources; then, it generates a basic quality profile based on field status, data source running status, cache status, consistency of source-side permission tags, and historical execution status; when a cross-domain query request passes the access permission verification, the virtualization engine generates a cross-domain quality profile by combining the requester domain permission tags, requester domain timeliness requirements, and result granularity requirements; subsequently, the cross-domain quality profile is converted into a profile including field constraints, data source constraints, and execution constraints. Quality constraint description; the virtualization engine updates the virtual data object metadata based on the quality constraint description, and inserts field transformation operators, writes physical data source call paths, cached data source call paths or candidate data source call paths into the cross-domain query execution plan. At the same time, it generates cache validity judgment nodes, failure rollback nodes and result trust identifier generation nodes. After execution, the virtualization engine converts execution feedback such as source unreachability, field mode drift, cache invalidation, permission label change, query failure or field quality degradation into profile correction items, updates the basic quality profile and cross-domain quality profile, and regenerates the quality constraint description and subsequent cross-domain query execution plan based on the updated profile.

[0203] Through the above process, quality collection information comes from field statistics, interface detection, cached metadata, permission tag comparison, lineage records and historical execution logs. Profile dimensions and constraint descriptions are generated from the above objective operational data. Field calls, data source selection, cache calls, failure rollbacks and result trust identifier generation are all executed by the virtualization engine according to the quality constraint descriptions, avoiding the quality assessment from remaining at the level of manual rules, quality reports or data asset sorting.

[0204] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for heterogeneous data quality assessment and cross-domain sharing in a multi-cloud environment, applied to a data sharing system including a virtualization engine, characterized in that... include: Virtual data objects are generated for heterogeneous data sources in multiple cloud domains registered to the virtualization engine, and the correspondence between virtual data objects and field mapping relationships, physical data source identifiers, cloud domain identifiers, access interface protocols, permission tags, lineage identifiers, cache version identifiers, and candidate data source association information is established. Obtain quality collection information associated with virtual data objects. The quality collection information includes field status information, data source running status information, cache status information, source-side permission tag consistency information, and historical execution status information. Based on the correspondence and quality collection information, a basic quality profile is generated that is associated with virtual data objects, fields and physical data sources; In response to a cross-domain query request that has passed access permission verification, obtain the requester's domain context; Based on the basic quality profile, the requester domain context, and the corresponding relationship, a cross-domain quality profile is generated that is associated with the virtual data object, field, physical data source, and requester domain. The cross-domain quality profile includes cross-domain permission label consistency, which is formed by the matching results between the requester domain permission label in the requester domain context, the permission label of the virtual data object, and the permission label of the physical data source. The cross-domain quality profile is converted into a quality constraint description that can be parsed and executed by the virtualization engine. The quality constraint description includes field constraints, data source constraints, and execution constraints. The set of callable fields and field transformation methods are determined based on field constraints. The data source to be called is determined based on data source constraints. The data source to be called includes physical data source, cached data source or candidate data source. The cache validity conditions, failure fallback paths and result trust identifier generation rules are determined based on execution constraints. Update the virtual data object metadata based on the quality constraint description, and generate or rewrite the cross-domain query execution plan; Perform cross-domain data access according to the cross-domain query execution plan, obtain feedback on this cross-domain query execution, and update the basic quality profile, cross-domain quality profile, and quality constraint description based on this cross-domain query execution feedback.

2. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: Establish the correspondence between virtual data objects and field mapping relationships, physical data source identifiers, cloud domain identifiers, access interface protocols, permission tags, lineage identifiers, cache version identifiers, and candidate data source association information, including: Establish a field mapping relationship between the fields in the virtual data object and the fields in the physical data source; Record the cloud domain where the physical data source is located, the access interface protocol, the source data update information, the cache version identifier, and the lineage identifier; Record the permission tags corresponding to the virtual data object, the fields in the virtual data object, and the fields of the physical data source; Record the relationships between fields in the virtual data object and cached data sources, mirrored data sources, synchronous replica data sources, semantically compatible data sources, and candidate alternative data sources.

3. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: Quality collection information includes: Field status information includes field missing rate, format error rate, field schema version, field semantic identifier, and field update time; The data source runtime status information includes interface reachability detection results, response latency, source data update time, source version identifier, semantic mapping version status, data synchronization status, and access failure records; Cache status information includes cache version identifier, cache generation time, cache validity period, cache source identifier, and source data version matching relationship; Source-side permission tag consistency information includes the matching results between virtual data object permission tags and physical data source permission tags; Historical execution status information includes cross-domain query failure rate, failure type, failure rollback records, response latency anomaly records, and result reliability identifier anomaly records; The requester domain context includes the requester domain permission tags, the requester domain timeliness requirements, and the result granularity requirements.

4. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: The basic quality profile is represented as follows: Cross-domain quality profile is represented as follows: The vector structure representation of cross-domain quality profiling is as follows: In the formula, Indicates time Below and virtual data objects Fields and physical data sources The associated basic quality profile, Indicates time Below and virtual data objects Fields Physical data source and requesting domain Related cross-domain quality profiles Indicates the requester's domain context. This indicates a profile generation function that incorporates the requester's domain context into the base quality profile. Indicates structural integrity. Indicates pattern consistency. Indicates the basic semantic matching degree. This indicates the cross-domain semantic matching degree after considering the granularity requirements of the requester's domain result. Indicates basic timeliness. This indicates the cross-domain timeliness after considering the timeliness requirements of the requesting domain. Indicates interface reachability. Indicates cache validity. This indicates consistency of the source-side permission tags. This indicates consistency between cross-domain permission tags following the requester's domain permission tag. Indicates the integrity of bloodline. This indicates historical execution stability.

5. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: Quality constraint descriptions include field constraint items. Data source constraints and execution constraints And expressed as: In the formula, Represents virtual data objects In the requesting domain The following quality constraint description, Indicates a collection of callable fields. This indicates that accessing the field collection is prohibited. Represents a set of field transformation operators. Indicates the collection of physical data sources that can be invoked. This indicates a collection of cached data sources that can be invoked. This indicates the collection of candidate data sources that can be invoked. Indicates cache validity conditions. Indicates the fallback path in case of failure. This indicates the rules for generating a trustworthy identifier for the result.

6. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: The virtualization engine inserts field transformation operators into the cross-domain query execution plan based on field constraints, and writes the physical data source call path, cached data source call path, or candidate data source call path into the cross-domain query execution plan based on data source constraints. Field transformation operators include field pruning operators, field desensitization operators, field aggregation operators, format conversion operators, and granularity adjustment operators; The candidate data source call path is generated from the candidate data sources that satisfy the field mapping relationship, source side permission tag consistency information, cross-domain permission tag consistency, data source running status information, and the version status, semantic mapping version status, data synchronization status or cache status corresponding to the candidate data source type.

7. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 5, characterized in that: The field constraint items determine the set of callable fields, the set of prohibited fields, and the set of field transformation operators based on field status information, source-side permission label consistency information, cross-domain permission label consistency, and cross-domain quality profile. The virtualization engine updates the field call flags, field mask flags, and field transformation flags in the virtual data object metadata according to the field constraints, and inserts field transformation operators into the cross-domain query execution plan.

8. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 5, characterized in that: The data source constraints determine the physical data source set, cache data source set, and candidate data source set based on data source running status information, cache status information, source-side permission tag consistency information, cross-domain permission tag consistency, and cross-domain quality profile. The execution constraints determine cache validity conditions, failure fallback paths, and result trust identifier generation rules based on cache status information, data source runtime status information, and historical execution status information. The validity criteria for a cache are formed by the cache version identifier, the cache generation time, the source data update time, the requester's domain timeliness requirements, and the cache source identifier. When a physical data source call fails, the failure fallback path generates subsequent execution paths in the order of cached data source call path, candidate data source call path, and path that is rejected and triggers quality reassessment. The result trust identifier generation rule generates a result trust identifier that carries field transformation markers, cache source markers, candidate data source call markers, and quality constraint version markers.

9. The method for heterogeneous data quality assessment and virtualization cross-domain sharing in a multi-cloud environment according to claim 1, characterized in that: Cross-domain query execution feedback includes source unreachability information, field schema drift information, permission tag change information, cache invalidation information, response latency exception information, query failure information, and field quality degradation information; Cross-domain quality profiles are updated according to the following formula: In the formula, This indicates the updated cross-domain quality profile. This indicates the update step size corresponding to the cross-domain query execution feedback type. This indicates feedback from the cross-domain query execution. The resulting image correction items; Source unreachability information is converted into interface reachability correction items, field pattern drift information is converted into pattern consistency correction items, cache invalidation information is converted into cache validity correction items, permission tag change information is converted into cross-domain permission tag consistency correction items, and query failure information is converted into historical execution stability correction items. Image correction items unrelated to the requesting domain are used to update the base quality image, while image correction items related to the requesting domain are used to update the cross-domain quality image. The virtualization engine regenerates the quality constraint description based on the updated base quality profile and cross-domain quality profile, and updates the virtual data object metadata and subsequent cross-domain query execution plans based on the regenerated quality constraint description.

Images