Demand-driven data resource dynamic identification and discovery method

Abstract

The invention provides a demand-driven data resource dynamic identification and discovery method, and relates to the technical field of data resource discovery, and the method comprises the following steps: inputting a natural language scene; performing semantic analysis and map reasoning; generating a federal candidate set; performing scene quality evaluation; and returning the federal identification code and the quality evaluation list of the data resource meeting the quality requirement. According to the method, a clear knowledge graph modeling process is given, and the knowledge graph is a federated knowledge graph which is not used for federated machine learning; a traditional extensive evaluation mode that multiple pieces of data use one general data quality evaluation is changed, the data value can be fully released, differentiated quality evaluation can be obtained in different scenes, whether the data are shared and easy to use or not is more accurately judged, and the sharing exchange efficiency and accuracy of government affair data are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data resource discovery, and in particular to a demand-driven data resource dynamic identification and discovery method. Background Art

[0002] Over the long-term development of information technology, various departments have accumulated vast amounts of data resources. However, these resources are often organized and managed in relatively static directories within various information centers. This leads to significant drawbacks in application, such as fragmented directories and a lack of complete business semantics. Each department and each level often independently maintains multiple types of directories. Due to a lack of unified cross-departmental planning, data resources across regions and departments have formed a complex "directory maze," which not only makes it difficult for users to find data but also increases the complexity and maintenance costs of data management.

[0003] According to Chinese patent CN109522465A, a semantic search method and device based on knowledge graph is provided. This solution overcomes the limitations of traditional keyword search through semantic understanding and knowledge association, and provides more accurate and intelligent search results. However, according to the above solution, the main focus of the existing technology is on the presentation of search results. It does not carry out ontology design of how the data resources themselves are oriented towards the "resource discovery" scenario, and assigns a unique, dynamic, and semantically rich code to the identifier of each resource. It only considers the relevance ranking of semantic search, but does not include a refined multi-dimensional quality assessment of the data itself for specific scenarios.

[0004] Therefore, existing technologies for cross-domain data resource sharing suffer from the following flaws throughout the entire process: Different business departments independently built their own data center systems, each without fully considering the requirements for cross-system data collaboration at the outset. This lack of unified metadata exchange standards, management practices, and cross-domain synchronization mechanisms has led to the formation of numerous "data silos." Furthermore, concerns about data sovereignty, security, and privacy protection have made data holders reluctant or lacking the motivation to fully open up their data or migrate it to a centralized data platform.

[0005] On the other hand, in the existing data governance domain, data quality assessments often use a universal, static set of metrics to score all data. For example, universal basic rules are used to assess data integrity and standardization. This approach ignores the differences in key quality dimensions of data across different application scenarios. As a result, the quality scores received by users in the "data consumption domain" fail to truly reflect the data's applicability to their specific businesses. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a demand-driven dynamic identification and discovery method for data resources to solve the problems raised in the above-mentioned background technology. The present invention provides a clear knowledge graph modeling process, and it is a federated knowledge graph. This federated knowledge graph is not used for federated machine learning; it changes the traditional extensive evaluation model of "multiple copies of data using one common data quality evaluation", can fully release the value of data, obtain differentiated quality evaluation in different scenarios, and more accurately judge whether the data is suitable and easy to use.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a demand-driven data resource dynamic identification and discovery method, comprising the following steps: S1, natural language scenario input, users describe their specific application scenarios in natural language rather than directly describing the data; S2, semantic parsing and graph reasoning: The system uses LLM technology to understand the scenario description entered by the user, extract core business terms, and then execute predefined semantic network reasoning rules to find matching data resources in the same metadata graph built based on federated metadata; S3. Federation candidate set generation. Based on the matching results, the system will initiate requests to each data node in the metadata federation system step by step or in parallel. Each node will detect the candidate set in different federation nodes. S4: Scenario-based quality assessment: The system performs quality assessment on the multi-dimensional data contracts that have been applied to the candidate set, and then sends the assessment results back to the requesting node for sorting; S5. Return the federal identification codes and quality assessment list of data resources that meet the quality requirements.

[0008] Furthermore, it also includes the construction of a business semantic knowledge graph. The construction plan includes: defining core metaclasses and their attributes, establishing core relationships, and establishing technical constraints. Among them, the establishment of technical constraints uses RDFS for business ontology modeling, combined with SHACL to define the constraints of the data model.

[0009] Furthermore, the core metaclass includes application scenarios, business terms, data resources and quality indicators, wherein the quality indicators are used to customize data contracts; the establishment of core relationships includes the application scenarios using certain business terms to describe their data requirements, the business terms being mapped to specific data resources or their fields, and the data resources being evaluated using certain quality indicators.

[0010] Furthermore, step S2 establishes a metadata federation mechanism, which builds a four-tiered federated node system: national, provincial, municipal, and county. Each node is interconnected, logically forming a unified metadata management domain. Each node is responsible for its own level of data resource metadata, which is aggregated and synchronized with the national node, while retaining a local copy.

[0011] Furthermore, the metadata federation mechanism includes two synchronization mechanisms. One is that national-level nodes regularly pull enhanced metadata from local copies; the other is that local copies complete real-time synchronization by actively triggering event streams; scheduled batch-incremental updates are used between provincial-national nodes, and active triggering is used between city-county nodes.

[0012] Furthermore, it also includes a scenario-based data contract mechanism. The data contract is defined in the form of a template using YAML format. The template is edited by humans and mapped to specific functions that need to be executed internally by the system parsing module.

[0013] Furthermore, the data contract template includes the following elements: contractId: unique identifier of the contract; scene: Description of the specific business scenario to which this contract applies; metrics: A list of one or more quality metrics.

[0014] Furthermore, step S4 includes the following scoring process: S4.1 When a user initiates a data request for a specific scenario, the system matches the most appropriate data contract template based on the scenario description; S4.2 applies the contract template to the candidate data resources and calculates the normalized score of each indicator based on the scoring function and target value defined in the contract. ; S4.3 Use the weights of each indicator defined in the contract , calculate the final total score of the candidate resource in this scenario in a weighted manner, ,in ; S4.4 uses the total quality score and the scores of each individual indicator in the metadata of the DataResource corresponding to the quality assessment vector for subsequent sorting, recommendation, reuse and auditing.

[0015] Furthermore, it also includes a set of demand-driven data resource encoding algorithms, which are used to establish a close relationship between the uniquely identified form of successfully applied scenario requirements and distributed heterogeneous data resources for auditing and reference by other external systems; the data resource encoding algorithm instantly generates a globally unique dynamic code containing rich contextual information for the finally selected high-quality data resources.

[0016] Furthermore, the data resource encoding algorithm includes the following contents: Semantic parsing: extracting business terms from natural language scenario descriptions; Term to SPARQL conversion: build SPARQL queries based on business terms; Distributed federated query: executes queries in the federated metadata network and aggregates candidate resources; Scenario-based quality assessment and ranking: Score and rank candidate resources based on scenarios and data contracts; Dynamic code generation: Generate dynamic code for resources that meet the conditions.

[0017] Beneficial effects of the present invention: 1. This demand-driven dynamic data resource identification and discovery method eliminates the need for users to learn and understand existing global data resource catalogs. Instead, users can directly describe their business scenarios using natural language. The system automatically understands their intent and, leveraging the federated metadata knowledge graph, provides a single-step process from "scenario requirements" to "available data resources." This significantly improves the efficiency and accuracy of government data sharing and exchange, while reducing the mental burden of browsing through extensive resource catalogs.

[0018] 2. This demand-driven dynamic identification and discovery method for data resources uses a federated metadata mechanism. The original data resources are retained locally in their respective ownership departments, and metadata is synchronized and calculated only within the federated system. This can effectively protect the rights and interests of data owners, eliminate concerns about large and small data sharing, and thus promote cross-domain data collaborative applications.

[0019] 3. This demand-driven dynamic identification and discovery method for data resources uses a scenario-based data contract model, enabling the same data to display multi-dimensional dynamic quality evaluations based on different needs. Users can clearly understand the applicability of the data in specific scenarios, thereby making clearer and more specific data selection decisions.

[0020] 4. The present invention generates a unique identifier containing the demand context through a demand-driven dynamic resource encoding algorithm, providing technical support for data in multi-dimensional lifecycle management, precise traceability, version control and auditing, and also provides a platform for data-using departments across the globe to learn from each other's best application practices. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is an overall flow chart of a demand-driven data resource dynamic identification and discovery method of the present invention; Figure 2 This is a system overall framework diagram of a demand-driven data resource dynamic identification and discovery method of the present invention; Figure 3 The semantic knowledge graph of the urban black and odorous water treatment business provided in the embodiment of the present invention; Figure 4 This is a flowchart of data resource processing in a four-level federation architecture according to an embodiment of the present invention; Figure 5 This is the metadata federation incremental synchronization sequence in an embodiment of the present invention; Figure 6 This is a quality scoring flow chart of the present invention; Figure 7 This is a flow chart of a demand-driven data resource encoding algorithm in an embodiment of the present invention; Figure 8 This is an input page for an example of the system operation effect in an embodiment of the present invention; Figure 9 A real-time progress display diagram of the system in an embodiment of the present invention waiting for execution of multiple steps when the system is running; Figure 10 A diagram showing recommended results of system operation according to an embodiment of the present invention; Figure 11 This is the system runtime evaluation process in an embodiment of the present invention; Figure 12 A best practice page for aggregating, classifying, and clustering different application scenarios during system operation in an embodiment of the present invention; Figure 13 This is a diagram of a four-level federation architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0023] See also Figures 1 to 13 , the present invention provides the following technical solutions: a demand-driven data resource dynamic identification and discovery method, which allows users to use natural language to describe their business demand scenarios (not to describe the data resources themselves). The system can use NLP technology and knowledge graphs across the fields of natural resources and smart cities to perform deep semantic analysis and reasoning to match related data resources.

[0024] This invention also builds a distributed federated metadata management and synchronization mechanism. While ensuring the data sovereignty of data nodes in different departments and that raw data does not leave the domain, it achieves the secure and orderly circulation and calculation of metadata within the federated system through a standardized metadata exchange model, event-driven protocol, and hash reconciliation mechanism, promoting the cross-domain collaborative use of data resources. We have also established a scenario-oriented, multi-dimensional data contract evaluation model. By customizing data contracts for different business scenarios (including specific quality parameters and applied to specific data sets), we can achieve differentiated and refined evaluations of the same data resource in different application scenarios.

[0025] The overall process includes the following steps: (1) Natural language scenario input: Users describe their specific application scenarios using natural language (rather than traditional keywords) rather than directly describing the data.

[0026] (2) Semantic parsing and graph reasoning. The system uses LLM technology to understand the scenario description entered by the user and extract core business terms. It then executes predefined semantic network reasoning rules to find matching data resources in the same metadata graph built based on federated metadata.

[0027] (3) Generation of federated candidate sets. Based on the matching results, the system will initiate requests to each data node in the metadata federation system step by step or in parallel, and each node will detect the candidate sets in different federated nodes.

[0028] (4) Scenario-based quality assessment. The system performs quality assessment on the multi-dimensional data contracts that have been applied to the candidate set, and then sends the assessment results back to the requesting node for sorting.

[0029] (5) Return the federal identification codes and quality assessment lists of data resources that meet the quality requirements.

[0030] To enable data retrieval based on application scenarios described in natural language, it is necessary to address the "semantic gap" between scenario descriptions and traditional data resource metadata. This paper constructs and applies a business semantic knowledge graph (focused on knowledge graphs across two cross-business domains: natural resources and smart cities). The core goal is to connect dispersed data resources through business semantics across cross-business application scenarios, forming an understandable and reasonable knowledge network.

[0031] This embodiment also provides a specific method for constructing the above-mentioned business semantic knowledge graph as follows: (1) Define core metaclasses and their attributes Scene (application scenario): such as name (scene name, such as "urban dust pollution prevention and control", used to correspond to the knowledge package in the semantic reasoning engine), description (scene description, used to represent the knowledge content of the scene), domain (the field to which it belongs, such as "environmental protection", used for screening and classification).

[0032] BusinessTerm: such as label (term name, such as "PM2.5 concentration"), definition, and code.

[0033] DataResource (data resource): such as URI (unique resource identifier), owner (data owner), format (data format), qualityVector (quality assessment vector), version (data resource version).

[0034] QualityMetric (quality indicator, used for custom data contracts): such as metricName (metric name, such as "data update frequency"), unit (unit), weight (weight in a specific contract).

[0035] (2) Establishing core relationships Scene-uses-BusinessTerm (the application scenario uses certain business terms to describe its data requirements).

[0036] BusinessTerm-mapsTo-DataResource (business terms are mapped to specific data resources or their fields).

[0037] DataResource-evaluatedBy-QualityMetric (data resources are evaluated by certain quality metrics).

[0038] (3) Technical constraints RDFS (RDF Schema) is used for business ontology modeling, combined with SHACL (Shapes Constraint Language) to define data model constraints. For example, each DataResource must be referenced by at least one BusinessTerm and must contain a qualityVector field.

[0039] This invention conducts deep semantic modeling of scenarios, terminology, and data resources across business fields of natural resources and smart cities, and uses ontology reasoning and custom rules to discover data resources in a federated environment. It solves the practical pain points of complex business logic and hidden data relationships, and innovatively achieves direct access from demand to data.

[0040] Typical business semantic knowledge graph in the scenario ( Figure 3 ) Example: Treatment of urban black and odorous water bodies, as follows: Application Process (1) The user enters a scenario description, for example: "I need to query all black and odorous water body data in the built-up area of ​​​​Y city in X city in the past month, including water quality level, pollutants, monitoring locations and other information, in order to assess the impact of black and odorous water bodies in the future." (2) Semantic parsing: combining the intent understanding capability of LLM and the business terminology library to extract key business terms such as built-up area, water quality monitoring, geographic location, etc.

[0041] (3) Perform semantic network reasoning to map the extracted business terms and the pre-set and continuously expanded domain business knowledge package (SPARQL) into a structured query language.

[0042] (4) The SPARQL query will be executed in the federated metadata knowledge graph. The query request will traverse the relevant graph structure from the federated nodes that is aggregated and spliced ​​in the national nodes, and return the metadata information of the matching DataResource.

[0043] In the above scenario, the knowledge graph can link multiple scattered data resources, such as the "surface water quality automatic monitoring station data" (including water quality level, pollutant concentration, monitoring time, and station location) of the environmental protection department, the "urban drainage network distribution map" (including network direction and outlet location) of the water affairs department, and the "built-up area boundary vector data" of the urban construction department, through common business terms (such as "geographic location" and "water body") and scenario goals ("black and odorous water body treatment"), to provide users with a comprehensive data view.

[0044] This embodiment also provides a set of metadata federation mechanisms to achieve "dynamic weaving and identification of cross-domain data resources". Taking government data sharing as an example, a four-level federal node system is constructed at the national, provincial, municipal and county levels. The nodes at each level are interconnected, logically forming a unified metadata management domain. Each node is responsible for the metadata of the data resources at its level, and uniformly converges and synchronizes to the national node, while retaining a local copy. The architecture diagram is as follows Figure 4 shown.

[0045] This invention combines event-driven architecture, hash reconciliation, logical clock conflict resolution, and multi-level administrative hierarchy. Its mechanism is unique and specifically targets a long-standing problem in cross-domain government big data sharing and exchange. It cleverly balances the contradiction between data sharing needs and data sovereignty protection, addresses the difficulty of traditional centralized metadata management in adapting to large-scale, heterogeneous, and multi-sovereign data sources, and overcomes the security risks associated with simple data replication. It has important practical value in promoting the market-oriented allocation of data elements. Metadata synchronization mechanism (1) Enhanced synchronization and event-driven. This embodiment designs two synchronization mechanisms. First, the national-level node periodically pulls the enhanced metadata in the local copy; second, the local copy completes real-time synchronization by actively triggering the event stream. Between provincial and national nodes, because the overall update volume is relatively large, there will be greater pressure on the graph update task. A scheduled batch-incremental update is used to improve efficiency. For city-county nodes, because the update volume between local nodes is small, but the real-time requirement is high, an active triggering method is used.

[0046] (2) Event format. Choose the standardized JSON-LD format, for example: { "eventId":"unique-event-id-string", "op":"add" / "update" / "delete", / / Operation type "federationId":"D-330101", / / Federation node ID where the event was initiated "payload":{ / *JSON-LD format metadata fragment* / }, "hash":"SHA-256-of-payload", / / payload hash value for verification "timestamp":"ISO8601-timestamp" / / event timestamp }.

[0047] Figure 5To simplify the illustration, the actual sequential interaction is more complex, involving two-way confirmation and possible conflict resolution branches. For example, a user uploads a new data resource to the C-city node. The C-city node generates metadata in JSON-LD and publishes a `meta.add` event through the C-Broker. The P-province node's broker receives the event via a cross-domain bridge. The P-SyncSvc pushes the event to the P-RDF write triple, triggering the P-Reasoner to perform reasoning and update the P-GraphAPI view. Similar steps are followed for synchronization with the N-country node. In the event of a conflict, if the P node detects a version conflict, it generates a resolve message and writes it back.

[0048] With the aforementioned architecture established, it's theoretically possible to find data resources that meet scenario requirements. However, to address the issues presented by the fixed, coarse-grained, and fundamental evaluation methods used in traditional data quality assessments (though this traditional evaluation mechanism is certainly useful, representing a compromise to simplify the problem), this paper introduces a scenario-based data contract mechanism, providing refined, customizable, and dynamic quality assessment criteria for data selection based on different application objectives. This contract can be pre-set or dynamically defined and extended by the user.

[0049] This embodiment also provides a contract template structure as follows: Data contracts are defined as templates in a format that is easy for humans to read and machines to parse (YAML). Humans edit the templates, and the system parses the modules and maps them to specific functions that need to be executed internally. A contract template mainly contains the following elements: 1.contractId: The unique identifier of the contract (e.g., CNT-ENV-WATER-0007, automatically generated by the system).

[0050] 2. scene: Description of the specific business scenario to which this contract applies (e.g., "data support for the special action plan for the treatment of urban black and odorous water bodies").

[0051] 3. metrics: A list of one or more quality indicators. Each metric contains: 3.1name: indicator name (such as Timeliness, Completeness, Spatial Resolution, Reliability).

[0052] 3.2description: Detailed description of the indicator.

[0053] 3.3weight: The relative weight of this indicator in this contract (between 0 and 1, the sum of all indicator weights is 1).

[0054] 3.4target: The target value or threshold of the indicator (such as timeliness requirement "<=5 days", completeness requirement ">=98%").

[0055] 3.5func: Scoring function, used to calculate the score of the indicator based on the difference between the actual data performance and the target value. Common scoring functions include: 3.5.1 linear_decay: Linear decay function. The more the target is exceeded, the lower the score is.

[0056] 3.5.2 sigmoid: S-shaped function, which is sensitive to score changes near the target value.

[0057] 3.5.3 step: Step function, full score or zero score is obtained when a certain threshold is reached.

[0058] 3.5.4 identity: directly use the normalized actual value as the score Example contract YAML snippet contractId:CNT-ENV-WATER-0007 scene: "Data support for urban black and odorous water treatment" metrics: -name:Timeliness description:"Timeliness of data updates" weight:0.25 target:"<=5 days"#Target: Data delay should not exceed 5 days func:"linear_decay" -name:Completeness description:"Ratio of complete records for key fields" weight:0.35 Target: Completeness not less than 98% func:"sigmoid" -name:SpatialResolution description:"resolution of geospatial data" weight:0.20 target: "<=30m"#Target: Spatial resolution better than 30 meters func:"step"#For example, if it is better than 30 meters, 1 point will be awarded, otherwise 0 points will be awarded -name:Reliability description: "Reliability assessment of data sources" weight:0.20 Target: Reliability score no less than 0.95 (may come from other evaluations) func:"identity". The contract content is stored in a domain knowledge base driven by an LLM and retrieved based on the scenario requirements entered by the user. If no matching contract template is found, a vertical domain LLM is automatically generated using data samples of templates stored in the domain knowledge base, fine-tuned, and then manually verified / revised. Due to the simplicity of this generation task, the large model fine-tuning process uses the general LoRA fine-tuning method and will not be further described here.

[0059] This embodiment also provides quality scoring process content; 1. When a user initiates a data request for a specific scenario, the system matches the most appropriate data contract template based on the scenario description.

[0060] 2. Apply the contract template to the candidate data resources and calculate the normalized score of each indicator based on the func (scoring function) and target (target value) defined in the contract. (between 0 and 1).

[0061] 3. Use the weights of each indicator defined in the contract , calculate the final total score of the candidate resource in this scenario in a weighted manner, ,in .

[0062] 4. The quality assessment vector (qualityVector) consisting of the total quality score and the scores of each individual indicator is stored in the metadata of the DataResource corresponding to the quality assessment vector for subsequent sorting, recommendation, reuse and auditing.

[0063] An example of the significance of this process: For the same piece of POI data, in the context of travel route planning, its "information richness" (e.g., presence of photos, reviews, opening hours) and "update frequency" might be given higher weighting; whereas in the context of emergency shelter site selection, "geographic location accuracy" and "capacity information completeness" might be given higher weighting. Using different data contract templates, this POI data can be subjected to differentiated quality assessments, resulting in objective, multi-dimensional evaluations from an application perspective.

[0064] This embodiment also provides a set of demand-driven data resource encoding algorithms, which use this algorithm to establish a close relationship between the uniquely identified form of successfully applied scenario requirements and distributed heterogeneous data resources for auditing and reference by other external systems.

[0065] The algorithm's goal is to instantly generate a globally unique, dynamic code containing rich contextual information for the final selected high-quality data resource. Multiple such codes can be generated for the same data resource, which can be applied to different scenarios. The contextual information for this code includes metadata federation source, business semantics, version, and content hash.

[0066] The main idea behind generating this scene-dependent identification code is to trace back the entire detection process, select feature identifiers during the process, and then establish a structured code so that the entire recommendation process can be reproduced through this code in the future, forming an explainable recommendation rather than a "black box" recommendation.

[0067] The pseudo code for implementing the above algorithm flow in this embodiment is as follows: function generate_dynamic_code_for_scene(scene_text): / / Step 1: Semantic parsing - extracting business terms from natural language scenario descriptions extracted_terms = nlp_semantic_parser(scene_text) / / Enhanced by NLP and LLM / / Step 2: Term to SPARQL conversion - build SPARQL queries based on business terms sparql_query=build_sparql_from_terms(extracted_terms, knowledge_graph_schema) / / Step 3: Distributed federated query - execute the query in the federated metadata network and aggregate candidate resources candidate_resources execute_federated_sparql_query(sparql_query) / / Step 4: Scenario-based quality assessment and ranking - score and rank candidate resources based on scenarios and data contracts ranked_resources=rank_resources_by_scene_contract(candidate_resources, scene_text) / / Step 5: Dynamic code generation - Generate dynamic code for resources that meet the conditions results=[] terms_hash=calculate_hash(extracted_terms) / / Calculate the hash of the business term combination for resource in ranked_resources: if resource.quality_score>=threshold: / / Assuming there is a quality threshold dynamic_code=format_code(resource.federation_id, terms_hash, resource.version, resource.content_hash) add_to_results(results,{code:dynamic_code, score:resource.quality_score, resource_metadata:resource}) return results This embodiment also provides the above-mentioned dynamic encoding format description: The dynamic encoding format used is: federationId:businessKey:version:hashfederationId: identifies the federation node to which the data resource belongs. For example, it can be the administrative division code at the national level (N), provincial level (P11 for a province), municipal level (C3301 for a city), or county level (D330101 for a district or county). These codes are used to uniquely identify federation members when the federation metadata node is registered.

[0068] BusinessKey: A hash value of the core business term combination that drives data resource discovery (for example, a SHA256 hash of the sorted business term string). This allows the encoding to inherently capture the business context of resource discovery. The pre-hashed business term string can be cached for easy lookup in subsequent applications.

[0069] version: The version number of the data resource. This is a version identifier maintained by the federated metadata node provider.

[0070] Hash: The hash value of the data resource content (for example, the SHA256 hash of the key content of a data file or dataset). This is used to ensure the integrity and uniqueness of the data, facilitate version comparison and fault tolerance.

[0071] This embodiment also provides an example of system operation effect, such as Figures 8 to 12 shown.

[0072] Directly input application scenario requirements to automatically match cross-domain data resources, instead of entering keywords and searching for available resources in a fixed directory.

[0073] Directly input application scenario requirements to automatically match cross-domain data resources, rather than entering keywords and searching for available resources in a fixed directory. Progress is displayed in real time as multiple steps are executed. Multiple recommendations are obtained, clarifying which "data contract" requirements are met. The entire evaluation process can be traced back, making evaluation decisions explainable. Dynamically generated resource identifiers driven by demand automatically aggregate, categorize, and cluster best practices across different application scenarios for reference by other data users.

[0074] The basic principles, main features and advantages of the present invention are shown and described above. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0075] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A demand-driven data resource dynamic identification and discovery method, characterized in that: The following steps are involved: S1, natural language scenario input, users describe their specific application scenarios in natural language rather than directly describing the data; S2, semantic parsing and graph reasoning: The system uses LLM technology to understand the scenario description entered by the user, extract core business terms, and then execute predefined semantic network reasoning rules to find matching data resources in the same metadata graph built based on federated metadata; S3. Federation candidate set generation. Based on the matching results, the system will initiate requests to each data node in the metadata federation system step by step or in parallel. Each node will detect the candidate set in different federation nodes. S4: Scenario-based quality assessment: The system performs quality assessment on the multi-dimensional data contracts that have been applied to the candidate set, and then sends the assessment results back to the requesting node for sorting; S5. Return the federal identification codes and quality assessment list of data resources that meet the quality requirements.

2. The demand-driven data resource dynamic identification and discovery method according to claim 1, characterized in that: It also includes the construction of a business semantic knowledge graph. The construction plan includes: defining core metaclasses and their attributes, establishing core relationships, and establishing technical constraints. Among them, the establishment of technical constraints uses RDFS for business ontology modeling and combines SHACL to define the constraints of the data model.

3. The demand-driven data resource dynamic identification and discovery method according to claim 2, characterized in that: The core metaclass includes application scenarios, business terms, data resources and quality indicators, where quality indicators are used to customize data contracts; establishing core relationships includes application scenarios using certain business terms to describe their data requirements, business terms being mapped to specific data resources or their fields, and data resources being evaluated using certain quality indicators.

4. The demand-driven data resource dynamic identification and discovery method according to claim 2, characterized in that: A metadata federation mechanism is also established in step S2. This mechanism constructs a four-level federal node system at the national, provincial, municipal, and county levels. The nodes at each level are interconnected, logically forming a unified metadata management domain. Each node is responsible for the data resource metadata at its level, and uniformly converges and synchronizes to the national node, while retaining a local copy.

5. The demand-driven data resource dynamic identification and discovery method according to claim 4, characterized in that: The metadata federation mechanism includes two synchronization mechanisms. One is that national-level nodes regularly pull enhanced metadata from local copies; the other is that local copies complete real-time synchronization by actively triggering event streams; scheduled batch-incremental updates are used between provincial-national nodes, and active triggering is used between city-county nodes.

6. The demand-driven data resource dynamic identification and discovery method according to claim 4, characterized in that: It also includes a scenario-based data contract mechanism. The data contract is defined in the form of a template in YAML format. The template is edited by humans and mapped to specific functions that need to be executed internally by the system parsing module.

7. A demand-driven data resource dynamic identification and discovery method according to claim 6, characterized in that: The data contract template contains the following elements: contractId: unique identifier of the contract; scene: Description of the specific business scenario to which this contract applies; metrics: A list of one or more quality metrics.

8. The demand-driven data resource dynamic identification and discovery method according to claim 4, characterized in that: Step S4 includes the following scoring process: S4.1 When a user initiates a data request for a specific scenario, the system matches the most appropriate data contract template based on the scenario description; S4.2 applies the contract template to the candidate data resources and calculates the normalized score of each indicator based on the scoring function and target value defined in the contract. ; S4.3 Use the weights of each indicator defined in the contract , calculate the final total score of the candidate resource in this scenario in a weighted manner, ,in ; S4.4 uses the total quality score and the scores of each individual indicator in the metadata of the DataResource corresponding to the quality assessment vector for subsequent sorting, recommendation, reuse and auditing.

9. The demand-driven data resource dynamic identification and discovery method according to claim 1, characterized in that: It also includes a set of demand-driven data resource encoding algorithms, which are used to establish a close relationship between the uniquely identified form of successfully applied scenario requirements and distributed heterogeneous data resources for the purpose of auditing and reference by other external systems; the data resource encoding algorithm instantly generates a globally unique dynamic code containing rich contextual information for the finally selected high-quality data resources.

10. The demand-driven data resource dynamic identification and discovery method according to claim 9, characterized in that: The data resource encoding algorithm includes the following contents: Semantic parsing: extracting business terms from natural language scenario descriptions; Term to SPARQL conversion: build SPARQL queries based on business terms; Distributed federated query: executes queries in the federated metadata network and aggregates candidate resources; Scenario-based quality assessment and ranking: Score and rank candidate resources based on scenarios and data contracts; Dynamic code generation: Generate dynamic code for resources that meet the conditions.

Images