A demand-driven data resource dynamic identification and discovery method

Abstract

The application provides a demand-driven data resource dynamic identification and discovery method, and relates to the technical field of data resource discovery, and comprises the following steps: natural language scene input; semantic analysis and graph reasoning; federal candidate set generation; scene quality evaluation; and returning the federal identification code and quality evaluation list of the data resource meeting the quality requirements.The application gives a clear knowledge graph modeling process, and is a federal knowledge graph, which is not used for federal machine learning; changes the traditional extensive evaluation mode of 'using one general data quality evaluation for multiple data', can fully release the data value, can obtain differentiated quality evaluation under different scenes, can more accurately judge whether the data is usable and good to use, and greatly improves the sharing and exchange efficiency and accuracy of government data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data resource discovery, in particular to a demand-driven data resource dynamic identification and discovery method. BACKGROUND

[0002] Different departments have gathered a large amount of data resources in the long-term informationization construction, but these resources are often organized and managed in a relatively static directory manner in each information center, which has significant disadvantages in the application process, such as directory fragmentation and lack of complete business semantics. Each department and each level often maintains multiple types of directories independently; due to the lack of unified planning across departments, the data resources in the cross-regional and cross-departmental repositories form a complex "directory maze", which not only increases the difficulty of user search, but also increases the complexity and maintenance cost of data management.

[0003] According to the semantic search method and device based on a knowledge graph provided in Chinese patent CN109522465A, the limitations of traditional keyword search are overcome through semantic understanding and knowledge association, and more accurate and intelligent search results are provided. However, according to the above-mentioned scheme, the existing technology mainly focuses on the presentation of search results, and does not carry out ontology design on how the data resources themselves face the "resource discovery" scene, and does not give each resource identification a unique, dynamic, and semantic-rich code. Only the relevance ranking of semantic search is considered, but the fine multi-dimensional quality evaluation of data itself for specific scenarios is not included.

[0004] Therefore, in the whole process of cross-domain data resource sharing, the existing technology has the following defects: different business departments independently build their own data center systems, and at the beginning of construction, they do not fully consider the requirements of cross-system data collaboration, lack unified metadata exchange standards, management specifications and cross-domain synchronization mechanisms, and form many "data chimneys". At the same time, due to concerns about data sovereignty, data security and privacy protection, data holders are reluctant or lack the motivation to completely open or migrate data to a data centralized platform.

[0005] On the other hand, in the existing data governance domain, a set of general and static index system is often used to score all data, such as using general basic rules to evaluate the integrity and standardization of data. This evaluation method ignores the difference in key quality dimensions of data in different application scenarios, resulting in that the quality score obtained by the user in the "data consumption domain" cannot truly reflect the applicability of the data to the specific business. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application aims to provide a demand-driven data resource dynamic identification and discovery method to solve the problems raised in the background art, and the present application provides a clear knowledge graph modeling process, and it is a federated knowledge graph, which is not used for federated machine learning; it changes the traditional "one general data quality evaluation for multiple data" extensive evaluation mode, can fully release the value of data, and get differentiated quality evaluation in different scenarios, and more accurately judge whether the data is usable and good.

[0007] To achieve the above object, the present application is realized by the following technical scheme: a demand-driven data resource dynamic identification and discovery method, comprising the following steps:

[0008] S1, natural language scene input, the user describes the specific application scene through natural language description, rather than direct description about data;

[0009] S2, semantic analysis and graph reasoning, the system uses LLM technology to understand the scene description input by the user, extracts the core business terms, and then executes the pre-defined semantic network reasoning rules to find the matching data resources in the same metadata graph constructed based on the federated metadata;

[0010] S3, federated candidate set generation, according to the matching result, the system will initiate a request to each data node in the metadata federation system step by step or in parallel, and each node will check out the candidate set in different federated nodes;

[0011] S4, scene quality evaluation, the system executes quality evaluation on the multi-dimensional data contract applied to the candidate set, and then sends the evaluation result back to the request node for sorting;

[0012] S5, return the federated identification code and quality evaluation list of the data resources meeting the quality requirements.

[0013] Further, it also includes the construction of a business semantic knowledge graph, and the construction scheme includes: defining core meta-classes and their attributes, establishing core relationships, and establishing technical constraints; wherein the technical constraints are established by RDFS for business ontology modeling, and SHACL is combined to define the constraints of the data model.

[0014] Further, the core meta-classes include application scenes, business terms, data resources, and quality indicators, wherein the quality indicators are used to customize data contracts; the establishment of core relationships includes that the application scene uses certain business terms to describe its data requirements, the business terms are mapped to specific data resources or fields, and the data resources are evaluated by certain quality indicators.

[0015] Furthermore, step S2 establishes a metadata federation mechanism, which constructs a four-tiered federation node system at the national, provincial, municipal, and county levels. Each level of nodes is interconnected, logically forming a unified metadata management domain. Each node is responsible for its own data resource metadata, which is then uniformly aggregated and synchronized to the national-level node while retaining a local copy.

[0016] Furthermore, the metadata federation mechanism includes two synchronization mechanisms: one is that national-level nodes periodically pull enhanced metadata from local replicas; the other is that local replicas complete real-time synchronization by actively triggering event streams. Timed batch-incremental updates are used between provincial and national nodes, while active triggering is used between city and county nodes.

[0017] Furthermore, it also includes a scenario-based data contract mechanism. Data contracts are defined in template form, using YAML format. The templates are edited by humans, and the system parsing module maps them to specific functions that need to be executed internally.

[0018] Furthermore, the data contract template includes the following elements:

[0019] contractId: A unique identifier for the contract;

[0020] scene: A description of the specific business scenario to which this contract applies;

[0021] metrics: A list of one or more quality metrics.

[0022] Furthermore, step S4 includes the following scoring process:

[0023] S4.1 When a user initiates a data request for a specific scenario, the system matches the most suitable data contract template based on the scenario description;

[0024] S4.2 applies the contract template to the candidate data resources, and calculates the normalized score of each indicator based on the scoring function and target value defined in the contract. ;

[0025] S4.3 Use the weights of each indicator defined in the contract The final total score of the candidate resource in this scenario is calculated using a weighted method. ,in ;

[0026] S4.4 will use the metadata of the DataResource corresponding to the quality assessment vector composed of the total quality score and the scores of each individual indicator for subsequent sorting, recommendation, reuse and auditing.

[0027] Further, a set of demand-driven data resource coding algorithms are further included, which are used to establish a close relationship between the successfully applied scene requirements uniquely identified form and the distributed heterogeneous data resources, so as to audit and provide reference for other external systems; the data resource coding algorithm generates a globally unique dynamic code containing rich context information for the finally selected high-quality data resource.

[0028] Further, the data resource coding algorithm includes the following contents:

[0029] Semantic analysis: extracting business terms from natural language scene description;

[0030] Term to SPARQL conversion: constructing a SPARQL query based on business terms;

[0031] Distributed federated query: executing the query in the federated metadata network and aggregating candidate resources;

[0032] Scenario quality evaluation and sorting: scoring and sorting the candidate resources according to the scene and data contract;

[0033] Dynamic code generation: generating a dynamic code for the resource that meets the conditions.

[0034] Advantages of the present application:

[0035] 1. In the demand-driven data resource dynamic identification and discovery method, the user does not need to learn and understand the existing global data resource directory in advance, and directly uses natural language to describe the business scene, the system automatically understands the user's intention, and realizes one-hop direct access from "scene requirement" to "available data resource" with the help of federated metadata knowledge graph. The efficiency and accuracy of government data sharing and exchange are greatly improved, and the mental burden caused by browsing a large number of resource directories is reduced.

[0036] 2. The demand-driven data resource dynamic identification and discovery method can effectively protect the rights and interests of data owners through the federated metadata mechanism, and only synchronize and calculate metadata in the federated system, so as to promote cross-domain data collaboration application.

[0037] 3. The demand-driven data resource dynamic identification and discovery method adopts a scenario-based data contract model, so that the same data can exhibit multi-dimensional dynamic quality evaluation according to different requirements, and users can clearly understand the applicability of the data in a specific scene, so as to make more clear and explicit data selection decisions.

[0038] 4. The application generates a unique identifier containing demand context through a demand-driven dynamic resource coding algorithm, providing technical support for data in multi-dimensional life cycle management, accurate traceability, version control and audit final aspects, and providing a platform for global data departments to learn from each other's best application practices. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A demand-driven data resource dynamic identification and discovery method according to the present application is provided.

[0040] Figure 2 A demand-driven data resource dynamic identification and discovery method according to the present application is provided.

[0041] Figure 3 A city black and smelly water treatment business semantic knowledge graph is provided in an embodiment of the present application.

[0042] Figure 4 A data resource processing flowchart in a four-level federation architecture is provided in an embodiment of the present application.

[0043] Figure 5 A metadata federation incremental synchronization sequence is provided in an embodiment of the present application.

[0044] Figure 6 A quality score flowchart is provided in the present application.

[0045] Figure 7 A demand-driven data resource coding algorithm flowchart is provided in an embodiment of the present application.

[0046] Figure 8 An input page of a system running effect example in an embodiment of the present application is provided.

[0047] Figure 9 A real-time progress display chart when the system is running and waiting to execute multiple steps in an embodiment of the present application is provided.

[0048] Figure 10 A recommended result chart of system running in an embodiment of the present application is provided.

[0049] Figure 11 A system running time evaluation process in an embodiment of the present application is provided.

[0050] Figure 12 A page of best practices of different application scenarios in system running in an embodiment of the present application is provided.

[0051] Figure 13 A four-level federation architecture chart is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the following will further describe the present application in combination with specific embodiments.

[0053] Please refer to Figures 1 to 13 The present application provides the following technical solutions: a demand-driven data resource dynamic identification and discovery method, which allows users to describe their business demand scenarios (not the data resources themselves) using natural language, and the system can use NLP technology and knowledge graphs across natural resources and smart city fields to perform deep semantic analysis and reasoning to match related data resources.

[0054] The present application also constructs a distributed federated metadata management and synchronization mechanism. Under the premise of guaranteeing the data sovereignty of different department data nodes and the non-domain export of original data, through standardized exchange metadata models, event-driven protocols and hash accounting mechanisms, the safe and orderly circulation and calculation of metadata within the federation system are realized, and the cross-field collaborative use of data resources is promoted;

[0055] And a multi-dimensional data contract evaluation model for application scenarios is established. By customizing data contracts for different business scenarios (which contain specific quality parameters and are applied to specific data sets), differentiated and refined evaluation of the same data resource in different application scenarios is realized.

[0056] Among them, the overall process includes the following steps:

[0057] (1) Natural language scenario input. Users describe their specific application scenarios through natural language (rather than traditional keywords) rather than direct data descriptions.

[0058] (2) Semantic analysis and graph reasoning. The system uses LLM technology to understand the user's input scenario description and extracts core business terms. Then it executes predefined semantic network reasoning rules to find matching data resources in the same metadata graph constructed based on federated metadata.

[0059] (3) Generation of federated candidate set. According to the matching results, the system will sequentially or in parallel request each data node in the metadata federation system, and each node will check out the candidate set in different federated nodes.

[0060] (4) Scenario-based quality evaluation. The system performs quality evaluation on the multi-dimensional data contract applied to the candidate set, and then sends the evaluation results back to the requesting node for sorting.

[0061] (5) Return the federated identification code and quality evaluation list of the data resources that meet the quality requirements.

[0062] In order to realize the data search by the application scene described by natural language, the semantic gap between the scene description method and the traditional data resource metadata description needs to be solved. The business semantic knowledge graph (the field category is the knowledge graph of two cross-business fields of natural resources and smart city) is constructed and applied, and the core target is to associate the scattered data resources through the business semantics in the cross-business application scene, and form a knowledge network which can be understood and reasoned.

[0063] The embodiment also provides a specific construction method of the above business semantic knowledge graph as follows.

[0064] (1) Defining the core meta-class and its attributes

[0065] Scene (application scene): such as name (scene name, such as “city dust pollution prevention and control”, used for corresponding with the knowledge package in the semantic reasoning engine), description (scene description, used for knowledge representation of scene connotation), domain (belonging field, such as “environmental protection”, used for screening and classification).

[0066] BusinessTerm (business term): such as label (term name, such as “PM2.5 concentration”), definition, code (coding).

[0067] DataResource (data resource): such as URI (unique resource identifier), owner (data owner), format (data format), qualityVector (quality evaluation vector), version (data resource version).

[0068] QualityMetric (quality metric, used for customizing data contract): such as metricName (index name, such as “data update frequency”), unit, weight (weight in a specific contract).

[0069] (2) Establishing core relationships

[0070] Scene-uses-BusinessTerm (application scene uses some business terms to describe its data requirements).

[0071] BusinessTerm-mapsTo-DataResource (business term is mapped to a specific data resource or its field).

[0072] DataResource-evaluatedBy-QualityMetric (data resource is evaluated by some quality metrics).

[0073] (3) Technological constraints

[0074] RDFS (RDF Schema) is used for business ontology modeling, combined with SHACL (Shapes Constraint Language) to define constraints for the data model. For example, each DataResource should be referenced by at least one BusinessTerm and must contain a qualityVector field.

[0075] This invention performs deep semantic modeling of scenarios, terms, and data resources across business domains of natural resources and smart cities. It uses ontology reasoning and custom rules to discover data resources in a federated environment, solving the practical pain points of complex business logic and hidden data relationships, and innovatively realizing direct access from demand to data.

[0076] Typical business semantic knowledge graphs in the scenario Figure 3 Example: Treatment of urban black and odorous water bodies, as follows:

[0077] Application process

[0078] (1) User input scenario description, for example: "I need to query all black and odorous water bodies in the Y built-up area of ​​City X over the past month, including water quality level, pollutants, monitoring location and other information, in order to assess the impact of black and odorous water bodies on the future."

[0079] (2) Semantic parsing: Combining the intent understanding capabilities of LLM with the business terminology database, key business terms are extracted, such as built-up area, water quality monitoring, and geographical location.

[0080] (3) Perform semantic network reasoning to map the extracted business terms and the pre-set and continuously expanding domain business knowledge package (SPARQL) into a structured query language.

[0081] (4) The SPARQL query will be executed in the federated metadata knowledge graph. The query request will traverse the aggregated and stitched graph structure from the federated nodes in the national nodes and return the metadata information of the matching DataResource.

[0082] In the above scenarios, knowledge graphs can connect multiple scattered data resources, such as "surface water quality automatic monitoring station data" from environmental protection departments (including water quality level, pollutant concentration, monitoring time, and station location), "urban drainage network distribution map" from water affairs departments (including network direction and outlet location), and "built-up area boundary vector data" from urban construction departments, through common business terms (such as "geographical location" and "water body") and scenario objectives ("black and odorous water body treatment"), to provide users with a comprehensive data view.

[0083] The embodiment also provides a set of metadata federation mechanisms, realizes the dynamic weaving and identification of cross-domain data resources. Taking government data sharing as an example, a four-level federal node system of the state, the province, the city and the county is constructed. Each node is connected to each other, and logically constitutes a unified metadata management domain. Each node is responsible for the metadata of the data resources at the level, and is uniformly converged and synchronized to the national node, while the local copy is reserved. The architecture diagram is shown in Figure 4 .

[0084] The application combines event-driven architecture, hash reconciliation, logical clock conflict resolution and multi-level administrative hierarchy characteristics, and is specially designed to overcome the long-standing problems of government cross-domain big data sharing and exchange. The mechanism is unique. The contradiction between data sharing demand and data sovereignty protection is skillfully balanced, the problem that the traditional centralized metadata management is difficult to adapt to large-scale, heterogeneous and multi-sovereign data sources is solved, and the security risk brought by simple data replication is overcome, which has important practical value for promoting the marketization configuration of data elements

[0085] Metadata synchronization mechanism

[0086] (1) Enhanced synchronization and event-driven. The embodiment designs two synchronization mechanisms, one is that the national node pulls the enhanced metadata in the local copy at a fixed time; the other is that the local copy completes real-time synchronization by actively triggering an event stream. Between the province and the state node, because the overall update amount will be larger, for the graph update task, there will be a large pressure, and batch-incremental update is adopted to improve the efficiency. For the city-county node, because the update amount between local nodes is small, but the real-time requirement is high, the active triggering mode is adopted.

[0087] (2) Event format. The standardized JSON-LD format is selected, for example:

[0088] {

[0089] "eventId":"unique-event-id-string",

[0090] "op":"add" / "update" / "delete", / / Operation type

[0091] "federationId":"D-330101", / / Event initiation federal node ID

[0092] "payload":{ / *JSON-LD format metadata fragment* / },

[0093] "hash":"SHA-256-of-payload", / / Hash value of payload, used for verification

[0094] "timestamp": "ISO8601-timestamp" / / event timestamp

[0095] }.

[0096] Figure 5 For simplicity, the actual sequence interaction is more complex, including bidirectional confirmation and possible conflict handling branches. For example, a user uploads a new data resource at the C city node, the C city node generates metadata JSON-LD, and publishes the `meta.add` event through the C-Broker. The Broker of the P province node receives the event through the cross-domain bridge, the P-SyncSvc pushes the event to the P-RDF write triple, triggers the P-Reasoner for reasoning, and updates the view of the P-GraphAPI. Similar steps continue to synchronize to the N country node. When there is a conflict, if the P node detects a version conflict, a resolve message is generated for backwriting.

[0097] After the above architecture is established, the data resources that can theoretically correspond to the scene requirements have been found. However, in order to solve the problems caused by the fixed and coarse-grained basic evaluation method in traditional data quality evaluation (of course, this traditional evaluation mechanism is also meaningful, which is a compromise to simplify the problem), the invention introduces a scene data contract mechanism to provide fine and customizable dynamic quality evaluation standards for data selection under different application goals. This contract can be pre-set or dynamically defined and extended by users.

[0098] The contract template of the present embodiment is constituted as follows:

[0099] The data contract is defined in the form of a template, adopts a format (YAML) easy for human reading and machine parsing, and is edited by a human template and mapped by a system analysis module into a specific function that needs to be executed internally. A contract template mainly includes the following elements:

[0100] 1. contractId: a unique identifier of the contract (such as CNT-ENV-WATER-0007, which is automatically generated by the system).

[0101] 2. scene: a description of the specific business scene to which the contract is applicable (such as "Urban Black and Odorous Water Treatment Special Action Data Support").

[0102] 3. metrics: a list of one or more quality indicators. Each indicator (metric) includes:

[0103] 3.1 name: Indicator name (e.g., Timeliness, Completeness, Spatial Resolution, Reliability).

[0104] 3.2 description: Detailed explanation of the indicator.

[0105] 3.3 weight: The relative weight of this indicator in this contract (between 0 and 1, with the sum of the weights of all indicators being 1).

[0106] 3.4 target: The target value or threshold of this indicator (e.g., timeliness requirement "<=5 days", completeness requirement ">=98%").

[0107] 3.5func: The scoring function, used to calculate the score of the indicator based on the difference between the actual data performance and the target value. Commonly used scoring functions include:

[0108] 3.5.1 linear_decay: linear decay function, the more the score exceeds the target, the more linearly it decreases.

[0109] 3.5.2 sigmoid: A sigmoid function that is sensitive to changes in score around the target value.

[0110] 3.5.3 Step: Step function, reaching a certain threshold will result in full marks or zero marks.

[0111] 3.5.4 Identity: The normalized actual value is used directly as the score.

[0112] Contract YAML snippet example

[0113] contractId:CNT-ENV-WATER-0007

[0114] Scene: "Data Support for Urban Black and Odorous Water Body Treatment"

[0115] metrics:

[0116] -name:Timeliness

[0117] Description: "Timeliness of data updates"

[0118] weight: 0.25

[0119] target: "<=5 days" # Target: Data delay should not exceed 5 days

[0120] func:"linear_decay"

[0121] -name:Completeness

[0122] description:"key field completeness"

[0123] weight:0.35

[0124] target:">=98%"#target: completeness >= 98%

[0125] func:"sigmoid"

[0126] -name:SpatialResolution

[0127] description:"spatial resolution of geospatial data"

[0128] weight:0.20

[0129] target:"<=30m"#target: spatial resolution <= 30m

[0130] func:"step"#e.g. 1 if <= 30m, 0 otherwise

[0131] -name:Reliability

[0132] description:"reliability assessment of data source"

[0133] weight:0.20

[0134] target:">=0.95"#target: reliability score >= 0.95 (may come from other assessments)

[0135] func:"identity".

[0136] The above contract content is placed in a domain knowledge base driven by LLM, and the scenario requirement input by the user is detected. When no contract template matching the scenario requirement is found, the vertical domain LLM can be automatically generated by fine-tuning the data samples stored in this domain knowledge base, and then manually confirmed / revised. Since this generation task is relatively simple, the general LoRA fine-tuning method is used for fine-tuning of the large model, which will not be described here.

[0137] The embodiment also provides the quality score process content;

[0138] 1. When the user initiates a data request for a specific scenario, the system matches the most suitable data contract template according to the scenario description.

[0139] 2. Apply the contract template to the candidate data resource, and calculate the normalized score of each metric defined in the contract according to the func (scoring function) and target (target value) (in the range of 0 to 1).

[0140] 3. Calculate the final total score of the candidate resource in this scenario in a weighted manner using the weights of each metric defined in the contract where .

[0141] 4. The quality evaluation vector (qualityVector) composed of the quality total score and the score of each single metric is used to form the metadata of the DataResource, which is used for subsequent sorting, recommendation, reuse and auditing.

[0142] Significance example of the above process: the same "POI data", its "information richness" (such as whether there are photos, comments, opening hours) and "update frequency" may have higher weights in the "tourist route planning" scenario; while in the "emergency shelter site selection" scenario, the "geographical location accuracy" and "capacity information integrity" will have higher weights. Through different data contract templates, the POI data can be differentiated in quality evaluation, so as to obtain multiple dimensional objective evaluation from the application point of view.

[0143] The embodiment also provides a demand-driven data resource coding algorithm, which uses the algorithm to establish a close relationship between the successfully applied scenario requirements in a unique form and the distributed heterogeneous data resources, so as to audit and provide reference for other external systems.

[0144] Algorithm goal: generate a globally unique dynamic code containing rich context information for the finally selected high-quality data resource, and when the same data resource can be applied to different scenarios, multiple such codes can be corresponded. The context information of the code includes metadata federation source, business semantics, version, and content hash.

[0145] The main idea of generating this scenario-related identification code is to trace back the entire detection process, select the feature identification in the process, and then establish a structured code to reproduce the entire recommendation process through the code in the future, forming an interpretable recommendation rather than a "black box" recommendation.

[0146] The pseudo code of the embodiment to realize the above algorithm process is as follows:

[0147] ​function generate_dynamic_code_for_scene(scene_text):

[0148] / / Step 1: Semantic Parsing - Extract business terms from natural language scene description

[0149] extracted_terms = nlp_semantic_parser(scene_text) / / Utilize NLP and LLM enhancements

[0150] / / Step 2: Term to SPARQL Conversion - Build SPARQL query based on business terms

[0151] sparql_query = build_sparql_from_terms(extracted_terms, knowledge_graph_schema)

[0152] / / Step 3: Distributed Federated Query - Execute query in federated metadata network, aggregate candidate resources

[0153] candidate_resources = execute_federated_sparql_query(sparql_query)

[0154] / / Step 4: Scene Quality Assessment and Ranking - Rank candidate resources based on scene and data contracts

[0155] ranked_resources = rank_resources_by_scene_contract(candidate_resources, scene_text)

[0156] / / Step 5: Dynamic Code Generation - Generate dynamic code for eligible resources

[0157] results = []

[0158] terms_hash = calculate_hash(extracted_terms) / / Calculate hash of business term combination

[0159] for resource in ranked_resources:

[0160] / / Check if resource meets eligibility criteria based on terms hash if resource.hash == terms_hash:

[0161] if resource.quality_score >= threshold: / / assume there is a quality threshold

[0162] dynamic_code = format_code(resource.federation_id,

[0163] terms_hash,

[0164] resource.version,

[0165] resource.content_hash)

[0166] add_to_results(results, {code: dynamic_code, score: resource.quality_score, resource_metadata: resource})

[0167] return results

[0168] The embodiment also provides the above-mentioned dynamic coding format specification:

[0169] The dynamic coding format structure adopted is: federationId:businessKey:version:hashfederationId: identifies the federation node to which the data resource belongs. For example, it can be a national-level (N), provincial-level (P11 represents a province), municipal-level (C3301 represents a city), and county-level (D330101 represents a district and county) administrative division code, which is a unique federation member identifier determined by the federation metadata node when registering.

[0170] businessKey: the hash value of the core business term combination driving the data resource discovery (for example, the SHA256 hash of the sorted business term string). This makes the code itself contain the business context semantics when the resource is discovered. The business term string before the hash can be cached for subsequent application reverse lookup.

[0171] version: the version number of the data resource. This is a version identifier maintained by the federation metadata node provider.

[0172] hash: the hash value of the data resource content (for example, the SHA256 hash of the key content of the data file or data set). This is used to ensure the integrity and uniqueness of the data, facilitating version comparison and fault tolerance.

[0173] The embodiment also provides a system running effect example, such asFigures 8 to 12 The application is shown.

[0174] Directly input the application scenario requirement, automatically match cross-domain data resources, instead of inputting keywords to find available resources in fixed directories.

[0175] Directly input the application scenario requirement, automatically match cross-domain data resources, instead of inputting keywords to find available resources in fixed directories. Wait for multiple steps to be executed, and display the real-time progress. Obtain multiple recommended results, and clarify which "data contract" requirements are met. And the entire evaluation process can be traced back, so that the evaluation decision has explainability. Based on the requirement-driven dynamically generated resource identification code, the best practices of different application scenarios can be automatically aggregated, classified and clustered for reference by other data departments.

[0176] The above shows and describes the basic principles and main features of the application and the advantages of the application, and it is obvious to those skilled in the art that the application is not limited to the details of the above exemplary embodiments, and the application can be implemented in other specific forms without departing from the spirit or essential characteristics of the application.

[0177] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for demand-driven dynamic identification and discovery of data resources, characterized in that, Comprise the following steps: S1, natural language scene input, users describe their specific application scenarios through natural language, rather than direct description of data; S2, semantic analysis and graph reasoning, the system uses LLM technology to understand the user's input scene description, extracts the core business terminology, and then executes the predefined semantic network reasoning rules to find matching data resources in the same metadata graph based on the federal metadata; S3, federal candidate set generation, according to the matching results, the system will request each data node in the metadata federal system, and each node will check out the candidate set in different federal nodes; S4, scene quality evaluation, the system executes quality evaluation on the multi-dimensional data contract applied to the candidate set, and then sends the evaluation results back to the request node for sorting; S5, return the federal identification code and quality evaluation list of the data resource that meets the quality requirements; Step S4 includes the following scoring process: S4.1 When the user initiates a data request for a specific scene, the system matches the most suitable data contract template according to the scene description; S4.2 Apply the contract template to the candidate data resource, and calculate the normalized score of each metric according to the scoring function and target value defined in the contract ; S4.3 Using the respective indicator weights defined in the contract the final total score of the candidate resource in this scenario is calculated in a weighted manner, wherein ; S4.4 The quality evaluation vector composed of this quality total score and each single indicator score is correspondingly stored in the metadata of DataResource, which is used for subsequent sorting, recommendation, reuse and audit.

2. The method of claim 1, wherein, It also includes the construction of business semantic knowledge graph, the construction scheme includes: defining core meta-classes and their attributes, establishing core relationships and establishing technical constraints; Among them, RDFS is used for business ontology modeling, and SHACL is used to define the constraints of data model.

3. The method of claim 2, wherein: The core meta-classes include application scenarios, business terms, data resources, and quality indicators, where quality indicators are used to customize data contracts; The establishment of core relationships includes that application scenarios use certain business terms to describe their data requirements, business terms are mapped to specific data resources or fields, and data resources are evaluated by certain quality indicators.

4. The method of claim 2, wherein the method further comprises: In step S2, a metadata federal mechanism is also established, which builds a national, provincial, municipal and county federal node system. Each node is connected to each other, forming a unified metadata management domain in logic, and each node is responsible for the metadata of the data resources at this level, and is synchronized to the national node while retaining a local copy.

5. The method of claim 4, wherein: The metadata federal mechanism includes two synchronization mechanisms: one is that the national node pulls the enhanced metadata in the local copy at a fixed time; The second is that the local copy completes real-time synchronization by actively triggering event flow; Between the province and the state nodes, batch-incremental update is used, and between the city and the county nodes, active triggering method is used.

6. The method of claim 4, wherein the method further comprises: It also includes a scene data contract mechanism, which defines data contracts in the form of templates, uses YAML format, and is edited by humans. Template, mapped to specific functions that need to be executed by the system parsing module.

7. The method of claim 6, wherein the method further comprises: The data contract template contains the following elements: contractId: unique identifier of the contract; scene: specific business scene description applicable to this contract; metrics: a list of one or more quality indicators.

8. The method of claim 1, wherein: Also included is a set of requirement-driven data resource encoding algorithms that are used to establish a close relationship between the successfully applied scenario requirements uniquely identified form and the distributed heterogeneous data resources, in order to audit and provide other external systems for reference; the data resource encoding algorithm generates a globally unique dynamic code containing rich context information for the final selected high-quality data resource.

9. The method of claim 8, wherein, The data resource encoding algorithm includes the following: Semantic analysis: extract business terms from natural language scenario description; Term to SPARQL conversion: build SPARQL query based on business terms; Distributed federated query: execute query in federated metadata network, aggregate candidate resources; Scenario quality evaluation and sorting: score and sort candidate resources according to scenario and data contract; Dynamic code generation: generate dynamic code for resources that meet the conditions.

Images