A public data availability optimization method and system

Abstract

This invention relates to the field of public data open technology, specifically to a method and system for optimizing public data availability. The method includes: converting raw data into machine-readable format files and generating format feature vectors; determining the optimal RDF mapping strategy for each machine-readable format file based on the format feature vectors; mapping according to the strategy to form an RDF knowledge graph; identifying entities during the mapping process; triggering key identifier field missing detection when a pre-labeled entity type is identified; scheduling an external API to complete missing key identifier fields when they are detected; directly injecting the completed values ​​of the missing identifier fields into the RDF knowledge graph; converting the evaluation index proportion of the data forest index into first-level weights; adjusting the second-level weights based on the completion results of the missing identifier fields; and then performing feedback optimization on the optimal RDF mapping strategy. This improves the availability of publicly available data and achieves a continuous, dynamic optimization process closely linked to business objectives.

Description

Technical Field

[0001] This invention relates to the field of public data open technology, and specifically to a method and system for optimizing public data availability. Background Technology

[0002] The "Data Forest Index" is a professional index deeply rooted in the field of public data openness, produced by the Digital and Mobile Governance Laboratory of Fudan University. Since its first release in 2017, the "Data Forest Index" has conducted a comprehensive evaluation of the level of local public data openness and utilization every year, carefully measuring the "open data forest" of various regions, thereby promoting the supply, circulation and value release of public data.

[0003] However, current practices in opening up public data still face severe systemic technical challenges in meeting the high standards of data usability advocated by the "Data Forest Index." The core issue lies in the difficulty of existing technical solutions to support dynamic, multi-dimensional, and quantifiable data governance requirements. Specifically, firstly, the usability of data formats is severely inadequate. A large amount of public data still exists in non-machine-readable formats such as PDF or proprietary formats without open APIs, forcing computers to perform inefficient manual conversions before processing public data, fundamentally violating the basic principles of machine readability and non-proprietary nature. Secondly, the semantic level of public data is extremely low. Only a very small proportion of open platforms use semantic web formats such as RDF to publish data, resulting in a lack of machine-understandable relationships between data. This makes it difficult to achieve automatic data linking and interoperability across domains and platforms, and the value of data cannot be amplified in circulation. Finally, current data evaluation technologies have certain limitations. Even the "Data Forest Index" itself mainly focuses on static evaluation and cannot achieve a continuous and dynamic optimization process that is closely linked to business objectives. This leads to a rigid optimization direction for the data platform, which cannot respond responsively to diverse actual needs. Consequently, the overall utilization rate of public data resources is low, making it difficult to realize their due socio-economic value. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a public data availability optimization method and system. The aim is to improve the availability of publicly available data in terms of machine readability, non-proprietary formats, RDF semantic publishing, and identification field coverage, while simultaneously achieving a continuous, dynamic optimization process closely linked to business objectives, supporting dynamic, multi-dimensional, and quantifiable data governance requirements.

[0005] Firstly, this application provides a method for optimizing the availability of public data, comprising the following steps:

[0006] Obtain raw data from the public data open platform;

[0007] Convert the raw data into machine-readable file format;

[0008] Generate format feature vectors for each machine-readable file format, wherein the format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty;

[0009] Based on the format feature vector, determine the optimal RDF mapping strategy for each machine-readable format file;

[0010] Each machine-readable file format is mapped into an RDF triple according to the corresponding optimal RDF mapping strategy to form an RDF knowledge graph. During the mapping process, each entity is identified to generate an entity URI. When a pre-labeled entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph.

[0011] The optimal RDF mapping strategy is optimized based on the mapping results.

[0012] Among them, the optimal RDF mapping strategy is optimized based on the mapping results, including:

[0013] Obtain the latest evaluation indicator proportions of the Shulin Index and convert the evaluation indicator proportions of the Shulin Index into first-level weights;

[0014] The first-level weight is dynamically adjusted based on the completion result of the missing identifier field to obtain the second-level weight;

[0015] Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback.

[0016] In some embodiments, converting the raw data into a machine-readable format file includes:

[0017] A pre-trained deep learning model is used to evaluate the format of the original data, and the format conversion priority score and expected conversion time are obtained.

[0018] Based on the format conversion priority score and expected conversion time obtained from the evaluation, the corresponding optimal processing path is selected for each original data.

[0019] Each raw data is converted into a machine-readable file according to its corresponding optimal processing path.

[0020] In some embodiments, based on the evaluated format conversion priority score and expected conversion time, the optimal processing path is selected for each original data, including:

[0021] When both the format conversion priority score and the expected conversion time are greater than or equal to the upper limit preset threshold, the deep OCR processing path is selected for the original data.

[0022] When the format conversion priority score is greater than or equal to the upper limit preset threshold, but the expected conversion time is less than the upper limit preset threshold, the parsing flattening path is selected for the original data;

[0023] When both the format conversion priority score and the expected conversion time are between the upper and lower preset thresholds, a dynamic SQL query path is selected for the original data.

[0024] When both the format conversion priority score and the expected conversion time are less than the lower preset threshold, the original data is directly recorded as a machine-readable format file.

[0025] In some embodiments, generating format feature vectors for each machine-readable format file includes:

[0026] Traverse all levels of the machine-readable file path to obtain the maximum nesting depth and the total number of fields in the machine-readable file.

[0027] Identify all array structures in machine-readable format files and calculate the array complexity of machine-readable format files based on the identification results;

[0028] The correlation between tables in a machine-readable file is calculated using a field semantic similarity algorithm.

[0029] The maximum nesting depth, total number of fields, array complexity, and inter-table relationships of machine-readable format files are weighted and integrated to calculate the field structure complexity score of machine-readable format files.

[0030] In some embodiments, determining the optimal RDF mapping strategy for each machine-readable format file based on the format feature vector includes:

[0031] Establish a candidate mapping strategy matrix, which has preference weights for different candidate mapping strategies relative to each dimension of the format feature vector;

[0032] The format feature vector of the machine-readable file is weighted and summed with the preference weights of the corresponding dimensions of each candidate mapping strategy to calculate the fit score between different candidate mapping strategies and the machine-readable file.

[0033] From all candidate mapping strategies, the strategy with the highest fit score is selected as the optimal RDF mapping strategy for this machine-readable format file.

[0034] In some embodiments, when a pre-labeled entity type is identified, a key identifier field missing detection is triggered, and when a key identifier field is detected to be missing, an external API is scheduled to complete it, including:

[0035] When a pre-labeled entity type is identified, a semantic event is created. The semantic event includes the corresponding entity URI and the identified entity type.

[0036] Based on semantic events, determine the key identifier fields required by the entity;

[0037] Verify whether the key identifier fields required for the entity exist in the corresponding RDF knowledge graph. When the verification result is that they do not exist, generate a completion task. The completion task includes the target entity URI, the missing field type, and existing query clues.

[0038] Based on the completion task, an external API is scheduled to query the missing identifier field of the target entity and obtain the completed value of the missing identifier field.

[0039] In some embodiments, the first-level weights include the enterprise identifier field coverage dimension and other dimensions, and the sum of the first-level weights of each dimension is 1. The step of dynamically adjusting the first-level weights based on the completion result of the missing identifier field to obtain the second-level weights includes:

[0040] The success rate and accuracy rate of completing missing identifier fields were statistically analyzed.

[0041] An adjustment factor is calculated based on the completion success rate and completion accuracy of the missing identifier field. The adjustment factor is positively correlated with the completion success rate and completion accuracy.

[0042] The adjustment factor is multiplied by the first-level weight of the enterprise identification field coverage dimension to obtain the second-level weight of the enterprise identification field coverage dimension;

[0043] Based on the second-level weight of the enterprise identifier field coverage dimension, the first-level weights of other dimensions are adjusted to obtain the second-level weights of other dimensions, so as to keep the sum of the second-level weights of each dimension equal to 1.

[0044] In some embodiments, the optimal RDF mapping strategy is optimized based on the second-level weights, including:

[0045] Each dimension of the second-level weights is associated with each candidate mapping strategy;

[0046] When calculating the fit score between different candidate mapping strategies and the machine-readable format file, the fit score is multiplied by the second-level weight of the corresponding associated dimension of the candidate mapping strategy, and the product is used as the comprehensive score of each candidate mapping strategy.

[0047] The strategy with the highest overall score among all candidate mapping strategies is selected as the optimal RDF mapping strategy for this machine-readable file format.

[0048] In some embodiments, selecting the optimal processing path for each original data based on the evaluated format conversion priority score and expected conversion time further includes:

[0049] The system monitors the amount of raw data to be converted into a machine-readable format in real time. When the amount of data to be converted exceeds a preset limit, the system uses an external API interface to convert the raw data that exceeds the preset limit as the optimal processing path.

[0050] Secondly, this application provides a public data availability optimization system, including:

[0051] The data acquisition module is used to acquire raw data from the public data open platform;

[0052] The machine-readable format conversion module is used to convert the raw data into machine-readable format files;

[0053] The RDF semantic enhancement module is used to generate format feature vectors for each machine-readable format file. The format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty. Based on the format feature vectors, the optimal RDF mapping strategy for each machine-readable format file is determined. Each machine-readable format file is mapped into an RDF triple according to the corresponding optimal RDF mapping strategy to form an RDF knowledge graph. This knowledge graph is used to identify each entity during the mapping process to generate entity URIs. When a pre-annotated entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph.

[0054] The feedback optimization module is used to optimize the optimal RDF mapping strategy based on the mapping results.

[0055] The feedback optimization module is used to perform the following steps:

[0056] Obtain the latest evaluation indicator proportions of the Shulin Index and convert them into first-level weights.

[0057] The first-level weight is dynamically adjusted based on the completion result of the missing identifier field to obtain the second-level weight;

[0058] Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback.

[0059] The beneficial technical effects of the present invention include at least the following:

[0060] 1. A public data availability optimization method and system are adopted, constructing a closed-loop system from data preprocessing and semantic enhancement to dynamic evaluation and optimization. By converting the raw data into a machine-readable format, a format feature vector (including field structure complexity, data type distribution, and semantic annotation difficulty) is generated. This allows for intelligent analysis of the data's inherent characteristics, and the optimal RDF mapping strategy is determined based on the format feature vector. This ensures that the semantic transformation is not a mechanical application but an adaptive process based on the data's own structure, significantly improving the semantic level and machine understandability. Simultaneously, when mapping to generate the RDF knowledge graph, entity recognition and key identifier field missing detection mechanisms are triggered, and external APIs are dynamically scheduled for completion. The completed values ​​are directly injected into the RDF knowledge graph. This collaboration not only enhances the data's relevance and completeness but also achieves automatic cross-source data linking, amplifying the data's interoperability value. Furthermore, the "Data Forest Index" evaluation indicator is introduced, and its proportion is converted into the first-level weight. However, it is not statically applied but dynamically adjusted based on the identifier field completion results to obtain the second-level weight, which is then used to optimize the RDF mapping strategy. This collaborative design deeply couples external evaluation metrics with internal data processing workflows, enabling semantic mapping strategies to evolve in tandem with actual data quality and business objectives (guided by the DataForest Index), thus transforming static evaluation into a continuous dynamic optimization process. Throughout the system, format conversion is fundamental, format feature vectors provide intelligent guidance for mapping strategies, entity completion during the mapping process improves data quality in real time, and the completion results serve as the basis for weight adjustments. The dynamic changes in weights then feed back into the mapping strategy optimization, forming a collaborative closed loop of "data conversion - semantic enhancement - quality completion - evaluation linkage - strategy iteration." This not only overcomes the technical challenges of low format usability, insufficient semanticization, and rigid evaluation in public data, but also generates an overall multiplier effect through positive feedback loops between processes—format feature vectors optimize mapping accuracy, completion during mapping enriches evaluation dimensions, and dynamic adjustments to evaluation weights further refine the application of format feature vectors, thus comprehensively supporting the requirements of dynamic, multidimensional, and quantifiable data governance. Ultimately, it achieves a comprehensive improvement in public data in terms of machine readability, non-proprietary format compliance, RDF semantic publishing, and identification field coverage. At the same time, it breaks the platform optimization stagnation through continuous dynamic optimization, which greatly promotes the supply and circulation of data and the release of its value.

[0061] 2. Existing data processing systems rely on external data labels and lack a fine-grained understanding of the data's inherent characteristics. They typically perform coarse-grained processing based on file extensions (e.g., .json) or simple rules (e.g., "call parser A if it's JSON"), failing to distinguish whether a JSON file is a simple list of key-value pairs or a deeply nested complex object. This leads to inefficiency and poor results when processing complex data, while wasting resources when processing simple data. Therefore, this application does not immediately begin RDF conversion after data format standardization. Instead, an RDF semantic enhancement module performs an analysis step beforehand, generating a multi-dimensional format feature vector that includes field structure complexity (revealing the data's internal hierarchy), data type distribution (reflecting the regularity of data content), and semantic annotation difficulty (evaluating the matching degree between field names and the standard vocabulary). This creates a precise digital profile for each machine-readable format file, thus finely distinguishing seemingly similar (e.g., both being JSON) data with vastly different internal complexities and qualities. This provides a reliable decision-making basis for the subsequent "dynamic adjustment of the optimal RDF mapping strategy," allowing for the selection of the optimal mapping strategy in advance based on the format feature vector, significantly improving the accuracy and processing efficiency of the RDF semanticization process.

[0062] 3. Existing RDF mapping tools (such as standard D2RQ) use static mapping configuration files, which cannot perceive the structural characteristics of the data to be transformed. Therefore, the same strategy may be used to process simple tables and complex nested JSON, resulting in either information loss or the generation of redundant and inaccurate RDF data. Moreover, RDF transformation is a "black box" operation, disconnected from the preceding and following processes, making the entire data processing pipeline unidirectional and rigid. To address this, this application elevates RDF semantic enhancement from a static, passive data format conversion process to an intelligent, proactive, and context-aware knowledge extraction process. By designing format feature vectors and intelligent mapping strategy generation algorithms, it achieves a shift from the existing "one-size-fits-all" static mapping to a dynamic mapping strategy adjustment "tailored to the data." The resulting RDF knowledge graph can more accurately express the semantics and hierarchical structure of the source data, thereby significantly improving data usability and greatly enhancing the accuracy of cross-domain queries and machine-automated reasoning. Its core is to enable the RDF semantic enhancement module to understand the characteristics of the data (through format feature vectors), and thus make dynamic decisions based on the characteristics (determining the optimal RDF mapping strategy). This breaks through the rigid technical limitations of traditional D2RQ mapping configuration and can work in collaboration with other modules (based on the second-level weights, to perform feedback optimization of the optimal RDF mapping strategy) to form a closed-loop adaptive system. This allows it to exhibit strong robustness and adaptability when processing massive heterogeneous data, thereby effectively reducing the overall processing time, increasing throughput during peak periods, and efficiently meeting the multi-dimensional and dynamic assessment requirements of the "Data Forest Index".

[0063] 4. In existing technologies, the strategy selection of mapping tools or scripts is not directly related to the final business value or assessment requirements (such as the "Data Forest Index" score). In other words, the mapping strategy selection is disconnected from business objectives, failing to guarantee that technological investment can be efficiently transformed into business results. Furthermore, once a mapping strategy is set, it does not automatically adjust with changes in execution performance, and it cannot perceive high-level decisions such as "increasing the RDF mapping ratio is more important than completing the identifier fields at this stage." Therefore, it cannot intelligently weigh and choose when resources are limited, resulting in low overall efficiency. To address this, this application redefines the selection of RDF mapping strategies from a simple technical matching problem into a resource optimization and allocation problem oriented towards business objectives. Its core idea is to introduce dynamic weights as a "command stick," aligning micro-level technical decisions with macro-level business objectives, and enabling the system to self-adjust based on the data availability optimization effect. The entire feedback optimization loop enables changes in business weights to precisely regulate the selection of technical strategies, allowing the entire system to dynamically adjust its work focus based on external assessment standards (Data Forest Index) and internal data status. This ensures that computing resources are always directed to the tasks that most effectively improve the overall Data Forest Index score, achieving intelligent and dynamic command of macro-strategic goals to micro-technical execution. It has evolved from a static tool into an intelligent solution that meets dynamic assessment requirements, ultimately solving the problem of low availability of public data caused by the fragmentation of "data format, semantics, and identification," and achieving a qualitative change from "data piling up" to "intelligent data organism."

[0064] 5. This application further proposes an API call strategy for massive data, freeing up limited local computing resources from heavy batch conversion tasks. This helps maximize quality inspection results under limited resources and cope with the complexity of real-world scenarios. For example, when the amount of raw data to be converted into machine-readable format exceeds 10,000 records, the system chooses to call external APIs instead of processing the raw data exceeding the preset limit. This allows the saved local computing power to be concentrated on subsequent core quality verification tasks (such as PDF semantic enhancement and integrity verification of identification fields), ensuring that limited resources are invested in the most critical quality dimensions. This is not an isolated mechanism, but rather an optimization and self-protection mechanism proactively triggered by the system when it senses that its processing capacity is approaching a bottleneck. This achieves a balance between cost, efficiency, security, and stability, and to some extent reflects intelligent collaboration between modules and between resources and tasks.

[0065] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0066] The invention will be further described below with reference to the accompanying drawings:

[0067] Figure 1 This is a flowchart of the public data availability optimization method according to an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the public data availability optimization system structure according to an embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0070] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to indicate orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0071] Example 1:

[0072] Please see the appendix Figure 1 , Figure 1 A flowchart illustrating a public data availability optimization method provided in one embodiment of this specification is shown.

[0073] like Figure 1 As shown, this public data availability optimization method may include at least the following steps:

[0074] S1, Obtain raw data from the public data open platform.

[0075] S2 converts the raw data into a machine-readable file format.

[0076] Machine-readable file formats include, but are not limited to, XLS, CSV, XML, RDF, API format files, as well as non-proprietary format files such as CSV and JSON.

[0077] Specifically, in this embodiment, converting the original data into a machine-readable format file includes:

[0078] S21 uses a pre-trained deep learning model to evaluate the format of the original data, obtaining a format conversion priority score and the expected conversion time.

[0079] The specific implementation method for using a pre-trained deep learning model to evaluate the format of the original data is as follows:

[0080] First, two types of features are extracted from the original data file to be evaluated to form a 128-dimensional comprehensive feature vector. The first is the binary feature of the file, which can be obtained by parsing the magic number signature, internal structure identifier and encoding mode of the file header. The second is the metadata feature, which can include descriptive attributes such as file size, creation format version, number of data tables contained, nesting depth of field structure and character set type.

[0081] Subsequently, the 128-dimensional feature vector is input into a pre-trained deep learning model. This deep learning model is trained with a large amount of sample data with known format conversion difficulty and can learn the non-linear mapping relationship between complex features and conversion cost. The specific model training operation method is similar to the operation method of training deep learning models in existing technologies, and will not be described in detail in this embodiment.

[0082] Finally, the deep learning model outputs two key quantitative indicators: one is a format conversion priority score between 0 and 1. The higher the score, the more unstructured the original data is, which means it hinders subsequent processing and therefore needs to be processed first; the other is the expected conversion time, which can be predicted based on regression analysis of historical conversion task data, providing a basis for system resource scheduling and thus enabling intelligent optimization and sorting of the conversion task queue.

[0083] S22. Based on the evaluation results of the format conversion priority score and the expected conversion time, select the corresponding optimal processing path for each original data.

[0084] Specifically, in this embodiment, based on the evaluated format conversion priority score and expected conversion time, the optimal processing path is selected for each original data, including:

[0085] S221, when both the format conversion priority score and the expected conversion time are greater than or equal to the upper limit preset threshold, the deep OCR processing path is selected for the original data.

[0086] OCR (Optical Character Recognition) refers to the technology of converting text in an image into editable and searchable text data.

[0087] It is understandable that in the aforementioned pre-trained deep learning model, when both the format conversion priority score and the expected conversion time are greater than or equal to the upper limit preset threshold, the type of the original data is usually unstructured data, such as PDF (scanned version or generated by Word), Word document, PPT, etc. This type of original data usually exists in the form of a mixture of text, images, and layout, especially scanned image documents. In this embodiment, the deep OCR processing path selected for the original data is executed sequentially: First, a convolutional neural network is used for image preprocessing, and an adaptive binarization algorithm is used to optimize image quality; then, a text recognition engine integrating LSTM and CTC loss is used to perform end-to-end recognition of Chinese and English and simplified and traditional Chinese mixed text; a layout analysis algorithm is run simultaneously to divide text regions, table structures, and title levels; after recognition, for the parsed content, especially complex tables, a cross-row and column detection algorithm is called to reconstruct cell merging relationships to ensure the integrity of the data structure; finally, machine-readable format files such as CSV, XML, and JSON-LD are generated in parallel through a multi-format output pipeline.

[0088] S222, when the format conversion priority score is greater than or equal to the upper limit preset threshold, but the expected conversion time is less than the upper limit preset threshold, the parsing flattening path is selected for the original data.

[0089] Understandably, in the aforementioned pre-trained deep learning model, if the format conversion priority score is greater than or equal to the upper preset threshold, but the expected conversion time is less than the upper preset threshold, the original data is usually semi-structured data. Specifically, a format conversion priority score greater than or equal to the upper preset threshold indicates that the original data does not meet the direct, standard machine-readable requirements. Reasons may include: the use of non-standard field names, excessively deep nesting levels, or non-standard syntax, leading to a high risk of "usability" and necessitating conversion. The expected conversion time being less than the upper preset threshold means that the system predicts a lower computational cost for processing this data. This is usually because semi-structured data has a certain structure, typically organized through labels, tags, or specific formatting rules, but not as structured as database tables. Examples include markup language files such as XML, HTML, and JSON, and spreadsheet files such as Excel. These do not require heavy processing like OCR; the difficulty in conversion lies mainly in the parsing and reconstruction of the logical structure, rather than the recognition of the physical format. To address this, this embodiment employs the following parsing and flattening path for the original data: First, it parses the semi-structured data (such as a deeply nested JSON file) to identify its overall structure, including nesting levels, array structure, key field nodes, and data types (strings, numbers, booleans, etc.). Next, for the identified complex nested structure, a flattening algorithm is executed. This algorithm expands deeply nested objects or arrays and generates new, unique column names by adding prefixes or delimiters, thus converting the tree-like or graph-like structure into a two-dimensional tabular structure. For example, {"user":{"name":"Alice","address":{"city":"Hangzhou"}}} is flattened into two columns: user_name and user_address_city. Finally, the flattened, streamlined data is converted in parallel to the target format, such as CSV (as a standard table format) and JSON-LD (as a semantic format), through a multi-format output pipeline. This provides a precise, lightweight, and seamless conversion process.

[0090] S223, When both the format conversion priority score and the expected conversion time are between the upper limit preset threshold and the lower limit threshold, a dynamic SQL query path is selected for the original data.

[0091] It is understandable that in the aforementioned pre-trained deep learning model, when both the format conversion priority score and the expected conversion time are between the upper and lower preset thresholds, the type of the original data is usually a proprietary format database within structured data. In response, this embodiment executes the dynamic SQL query path selected for the original data: analyzing the database schema and query logic, and accordingly creating a RESTful API description document conforming to the OpenAPI specification in real time. This encapsulates the data as a dynamically queried API service, rather than a static file. In this way, external users can obtain the data through common protocol requests (such as GET / api / v1 / companies) without needing to know what kind of "proprietary" database is being used in the backend. That is, for this type of original data, the goal of this embodiment is not to "parse" a structurally ambiguous file, but to "break down access barriers," transforming a originally closed structured data source into an open, standardized web service.

[0092] S224, when both the format conversion priority score and the expected conversion time are less than the lower preset threshold, the original data is directly recorded as a machine-readable format file.

[0093] It is understandable that in the aforementioned pre-trained deep learning model, when both the format conversion priority score and the expected conversion time are less than the lower preset threshold, the type of the original data is usually other structured data (such as standard CSV or JSON) besides the database. In this case, this embodiment directly records the original data as a machine-readable format file, or diverts the original data to a lightweight verification and standardization process and quickly outputs it as a target format file.

[0094] It is understandable that the technical solution provided in this embodiment clearly reveals the technical path to improve data availability. That is, in the process of converting machine-readable format files, the focus is on the identification and conversion of "unstructured data", the parsing of "semi-structured data" is standardized, and the semanticization and service-oriented nature of "structured data" is deeply optimized. Through the above decision logic scheme, the accurate matching of processing resources and data characteristics is achieved.

[0095] S23, convert each raw data into a machine-readable format file according to the corresponding optimal processing path.

[0096] It is understandable that the aforementioned S1 and S2 steps can effectively normalize the heterogeneous raw data, providing high-quality and structurally clear input for subsequent RDF mapping.

[0097] S3 generates format feature vectors for each machine-readable file format. The format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty.

[0098] It is understandable that, after the machine-readable format file is converted, this embodiment does not simply pass the data to the next module, but generates a format feature vector. This format feature vector (dimension ≥ 32) contains deep features of the data, such as the complexity of field structure, data type distribution and semantic annotation difficulty, providing preprocessing intelligence for subsequent RDF enhancement, making subsequent processing more predictable and targeted. Moreover, the subsequent intelligent mapping strategy generation algorithm does not consider these three dimensions of the format feature vector independently, but makes a comprehensive judgment to accurately identify the "type" of the data and trigger the best mapping strategy.

[0099] In this embodiment, field structure complexity is a quantitative value used to assess the complexity of data organization hierarchy and relationships; it focuses on the "shape" of the data. Machine-readable format files with multi-level nesting or complex hierarchical relationships have high field structure complexity values. For example, JSON data for a department with multiple employees, each with multiple project experiences and skill tags; or an Excel report with merged cell headers, the first row showing "2023 Sales Performance," followed by merged cells for "First Quarter," "Second Quarter," etc., and further down, data for different regions. Conversely, flat structures with simple relationships have low field structure complexity values. For example, a simple address book table with fixed columns: name, phone number, and department, with each row representing a single contact.

[0100] Specifically, in this embodiment, generating the format feature vectors of each machine-readable file includes:

[0101] S31: Traverse all levels of the machine-readable file path to obtain the maximum nesting depth and the total number of fields in the machine-readable file.

[0102] Specifically, for hierarchical data such as JSON / XML, starting from the root node, traversing all paths, and recording the number of levels required to reach the deepest leaf node, we obtain the maximum nesting depth of the machine-readable format file. For tabular data, this value is usually 1 (header + data rows) or 2 (grouped headers exist).

[0103] S32 identifies all array structures of a machine-readable file and calculates the array complexity of the machine-readable file based on the identification results.

[0104] Specifically, identify all array structures in machine-readable format files and calculate: array complexity = Σ(array length * array member structure complexity), where, for simple value arrays (such as [1,2,3]), its member structure complexity is 1; for object arrays (such as [{a:1},{a:2}]), its member structure complexity is the number of fields of the object.

[0105] S33 uses a field semantic similarity algorithm to calculate the correlation between tables in a machine-readable format file.

[0106] Specifically, the correlation between tables can be estimated based on the similarity of field names or the overlap of data values. For example, in a table, the strength of the relationship between tables can be evaluated by the similarity of foreign keys or field values ​​(such as the department ID of one table and the ID of another table may be related). The specific operation method is similar to the operation method of calculating semantic similarity in the prior art, and will not be described in detail in this embodiment.

[0107] S34 calculates the field structure complexity score of the machine-readable file by weighting and integrating the maximum nesting depth, total number of fields, array complexity, and inter-table relationships of the machine-readable file.

[0108] Specifically, the maximum nesting depth, total number of fields, array complexity, and inter-table correlation of machine-readable format files are all normalized, mapping each indicator to the [0,1] interval. Then, each indicator is multiplied by its corresponding weight coefficient, and the products are summed. The sum of the weight coefficients of each indicator is 1. This can be learned from the training data by a machine learning model or obtained according to the experimental settings, reflecting the contribution of each indicator to the overall complexity. Finally, the field structure complexity score of the machine-readable format file is a value between 0 and 1.

[0109] It is understood that this embodiment only uses the method of generating the field structure complexity of the format feature vector of a machine-readable file as an example for illustration. The method of generating the field structure complexity of the format feature vector of other machine-readable files can refer to this embodiment.

[0110] In this embodiment, the data type distribution describes the mixture and regularity of different data types in a dataset, focusing on the "properties" of the content within each field. If the data types within each field of a machine-readable file are highly consistent and pure (i.e., highly structured data), the data type distribution will exhibit strong regularity. For example, in a database's "User Table," the User ID field consists entirely of integers, the Registration Time field is entirely in standard date format, and the Email field consists entirely of strings conforming to email rules. However, if the same field contains multiple distinct data formats or types (i.e., semi-structured or dirty data), the data type distribution will appear chaotic or irregular. For instance, an "Address" field extracted from free text might have the following values: "Zhejiang Province, Hangzhou City, Xihu District" (pure string), {"Province": "Zhejiang", "City": "Hangzhou"} (JSON object), or even a string containing latitude and longitude "120.12, 30.16".

[0111] Specifically, in this embodiment, the data type distribution score for generating a machine-readable format file can be implemented as follows: First, for each field, scan all its data values ​​and determine the actual type of each value (e.g., integer, floating-point number, date / time, string, boolean, null / NULL). Calculate: Field type purity = (number of the most frequent data types in the field) / (total number of data types in the field). For example, if a field has 100 values, of which 98 are integers and 2 are strings, then the field type purity = 98 / 100 = 0.98. Then, calculate the average of the type purity of all fields in the machine-readable format file, and then calculate 1 - this average value to represent the overall impurity. Next, use the concept of information entropy to measure the disorder of the data type distribution of the entire dataset. Count the total frequency of each data type in all fields, and calculate the overall type distribution entropy based on the frequency ratio of each data type in the entire dataset. The higher the entropy value, the more uniform and disordered the data type distribution; the lower the entropy value, the more concentrated the data is on a few types, and the more regular the distribution. Finally, according to the preset weighting coefficients, the overall impurity and the normalized overall entropy are weighted and summed to obtain the data type distribution score of the machine-readable format file.

[0112] In this embodiment, the semantic annotation difficulty is used to evaluate the understandability of data field names and their matching degree with a general vocabulary. It focuses on the clarity of the data's "meaning." For machine-readable format files, if the field names are descriptive and conform to general conventions, the semantic annotation difficulty value is high. For example, field names like firstName, birthDate, and totalPrice are very intuitive and easily mapped to schema:givenName, schema:birthDate, and schema:totalPrice in the vocabulary. Conversely, if the field names are obscure and their meaning cannot be directly inferred—that is, semantically ambiguous or proprietary—the semantic annotation difficulty value is low. For example, field names like CUST_ABRV_NM, FLD_003, and data item A are abbreviations, codes, or internal terms, and their true meaning requires external data dictionaries or domain knowledge to interpret.

[0113] Specifically, in this embodiment, the semantic annotation difficulty score for generating machine-readable format files can be achieved as follows: For each field name, a pre-trained word vector model (such as BERT or Word2Vec) is used to convert it into a vector representation. Similarly, standard terms in a general vocabulary (such as Schema.org) are converted into vectors. Then, the maximum cosine similarity between the field name vector and all standard term vectors is calculated as the semantic annotation difficulty score. The closer the value is to 1, the closer the semantics are to the standard terms.

[0114] Existing data processing systems rely on external data labels, lacking a fine-grained understanding of the data's inherent characteristics. They typically perform coarse-grained processing based on file extensions (e.g., .json) or simple rules (e.g., "call parser A if it's JSON"), failing to distinguish whether a JSON file is a simple list of key-value pairs or a deeply nested complex object. This leads to inefficiency and poor results when processing complex data, while wasting resources when processing simple data. Therefore, this embodiment does not immediately begin RDF conversion after data format standardization. Instead, an RDF semantic enhancement module performs a pre-analysis step, generating a multi-dimensional format feature vector that includes field structure complexity (revealing the data's internal hierarchy), data type distribution (reflecting the regularity of data content), and semantic annotation difficulty (evaluating the matching degree between field names and the standard vocabulary). This creates a precise digital profile for each machine-readable file, thus finely distinguishing seemingly similar (e.g., both being JSON) data with vastly different internal complexities and qualities. This provides a reliable decision-making basis for the subsequent dynamic adjustment of the optimal RDF mapping strategy, allowing for the selection of the optimal mapping strategy in advance based on the format feature vector, significantly improving the accuracy and processing efficiency of the RDF semanticization process.

[0115] S4. Based on the format feature vector, determine the optimal RDF mapping strategy for each machine-readable format file.

[0116] RDF (Resource Description Frame) is a standard model for describing network resources. Its basic unit is a triple consisting of "subject-verb-object", which enables data to have clear semantics and is easy for machines to understand and associate.

[0117] Specifically, in this embodiment, the optimal RDF mapping strategy for each machine-readable format file is determined based on the format feature vector, including:

[0118] S41. Establish a candidate mapping strategy matrix, which has preference weights for different candidate mapping strategies relative to each dimension of the format feature vector.

[0119] The candidate mapping strategy matrix is ​​a two-dimensional table. The rows represent the available mapping strategies (such as strategy A: standard flat mapping, strategy B: row and column structure awareness, strategy C: depth-first traversal, etc.), and the columns represent the key evaluation dimensions extracted from the format feature vector (i.e., field structure complexity, data type distribution, and semantic annotation difficulty). The matrix elements are the preference weights of different candidate mapping strategies relative to each dimension of the format feature vector. These preference weights are preset through historical data training or expert experience, and this embodiment does not limit this.

[0120] S42, the format feature vector of the machine-readable format file is weighted and summed with the preference weights of the corresponding dimensions of each candidate mapping strategy to calculate the fit score between different candidate mapping strategies and the machine-readable format file.

[0121] It is understood that in this embodiment, the normalized values ​​of the current data in each dimension (i.e., the j-th component of the format feature vector of the machine-readable format file) are scored according to the preference weight of the candidate mapping strategy i relative to the dimension j of the format feature vector, and finally a comprehensive score representing the degree of matching between the candidate mapping strategy and the current data is obtained by summing them.

[0122] S43. Select the strategy with the highest fit score from all candidate mapping strategies as the optimal RDF mapping strategy for the machine-readable format file.

[0123] Furthermore, this embodiment can be fine-tuned: after selecting the optimal RDF mapping strategy for machine-readable format files, the algorithm enters the parameter fine-tuning stage. Based on the specific values ​​of the format feature vectors, the operating parameters of the strategy are determined through linear interpolation or table lookup.

[0124] For example, for the selected "depth-first traversal" strategy, the "maximum traversal depth" parameter MaxDepth is no longer a fixed value, but is linearly related to the field structure complexity S_c: MaxDepth = BaseDepth + k * S_c. Here, BaseDepth is the base traversal depth, k is the scaling factor, and S_c is the specific field structure complexity value. Thus, for a machine-readable file with S_c = 0.5, the system might set MaxDepth = 5; while for a machine-readable file with S_c = 0.9, it might set MaxDepth = 10 to handle deeper nesting.

[0125] Understandably, existing RDF mapping tools (such as standard D2RQ) use static mapping configuration files, which cannot perceive the structural characteristics of the data to be transformed. Therefore, they may use the same strategy to process simple tables and complex nested JSON, resulting in either information loss or the generation of redundant and inaccurate RDF data. Furthermore, RDF transformation is a "black box" operation, disconnected from the preceding and following processes, making the entire data processing pipeline unidirectional and rigid. To address this, this embodiment elevates RDF semantic enhancement from a static, passive data format conversion process to an intelligent, proactive, and context-aware knowledge extraction process. By designing format feature vectors and intelligent mapping strategy generation algorithms, it achieves a shift from the existing "one-size-fits-all" static mapping to a dynamic mapping strategy adjustment "tailored to the data." The resulting RDF knowledge graph can more accurately express the semantics and hierarchical structure of the source data, thereby significantly improving data usability and greatly enhancing the accuracy of cross-domain queries and automated machine reasoning. Its core is to enable the RDF semantic enhancement module to understand the characteristics of the data (through format feature vectors), thereby making dynamic decisions based on the characteristics (determining the optimal RDF mapping strategy). This breaks through the rigid technical limitations of traditional D2RQ mapping configuration and can work in collaboration with other modules (based on the second-level weights, to perform feedback optimization of the optimal RDF mapping strategy) to form a closed-loop adaptive system. This allows it to exhibit strong robustness and adaptability when processing massive amounts of heterogeneous data, thereby effectively reducing overall processing time, increasing throughput during peak periods, and efficiently meeting the multi-dimensional and dynamic assessment requirements of the "Data Forest Index".

[0126] S5 maps each machine-readable format file into an RDF triple according to the corresponding optimal RDF mapping strategy, forming an RDF knowledge graph. During the mapping process, each entity is identified to generate an entity URI. When a pre-labeled entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph.

[0127] Understandably, the intelligent mapping strategy generation algorithm transforms the results of optimal RDF mapping strategy selection and parameter fine-tuning into a set of executable, declarative mapping instructions. These instructions directly correspond to the configuration primitives of the underlying D2RQ engine. This instruction set precisely tells the D2RQ engine how to parse input data and how to construct RDF classes and attribute relationships, thereby transforming the abstract optimal RDF mapping strategy into concrete, repeatable RDF triple mapping actions.

[0128] A URI (Uniform Resource Identifier) ​​is a string used to uniquely identify an entity (such as a company, a person, or a location). For example, the rule for generating an entity URI could be "protocol: / / domain / dataset type / timestamp / field hash value", ensuring the global uniqueness and stability of the entity.

[0129] In this embodiment, the RDF knowledge graph consists of a series of RDF triples (subject-verb-object), characterized by precise mapping and unique entity identification. Each entity is uniquely identified by a global URI, and the entity's attributes are represented by a verb (i.e., attribute type) and an object (i.e., attribute value). Key identification fields (such as an enterprise's "Unified Social Credit Code") correspond to a specific verb in the RDF model (e.g., using a standard vocabulary such as schema:identifier or a custom attribute URI).

[0130] It is understandable that the process S6 in this embodiment is a highly automated closed-loop processing flow driven by semantic events. Its core is to transform data completion from an independent post-event step into a real-time, on-demand service embedded in the semantic process.

[0131] Specifically, in this embodiment, when a pre-labeled entity type is identified, a key identifier field missing detection is triggered, and when a key identifier field is detected to be missing, an external API is scheduled to complete it, including:

[0132] S51, when a pre-labeled entity type is identified, a semantic event is created. The semantic event includes the corresponding entity URI and the identified entity type.

[0133] Specifically, during the D2RQ mapping process performed by the RDF semantic enhancement module, the entity type (e.g., "enterprise" or "individual") represented by each data subject is identified based on the selected optimal RDF mapping strategy or semantic analysis based on field names (e.g., field names mapping to categories such as schema:Organization or schema:Person). Once the pre-labeled entity type is identified, a semantic event is immediately created and transmitted to the identification field completion module. This semantic event contains at least two key pieces of information:

[0134] 1. Entity URI: A globally unique identifier generated for this entity;

[0135] 2. Entity type: The identified entity type (e.g., schema:Organization).

[0136] S52, based on semantic events, determines the key identifier fields required by the entity.

[0137] Specifically, the identifier field completion module queries its internal pre-built "Entity Type - Key Identifier Fields" rule base to determine the key identifier fields required for the entity based on the identified entity type. For example, the rule base defines that for an entity of type schema:Organization, its key identifier fields are schema:identifier (with the specific value of "Unified Social Credit Code") and schema:name (enterprise name).

[0138] S53, verify whether the key identifier fields required for the entity exist in the corresponding RDF knowledge graph. When the verification result is that they do not exist, generate a completion task. The completion task includes the target entity URI, the missing field type, and existing query clues.

[0139] Specifically, the process checks the RDF knowledge graph currently being built regarding the entity's URI to verify that the required key identifier fields already exist. If a key field is found to be missing (e.g., a company name is present but the unified social credit code is missing), a precise completion task is automatically generated. For example, this completion task should include at least:

[0140] 1. Target entity URI: A globally unique identifier generated for this entity;

[0141] 2. Missing field type: schema:identifier (Unified Social Credit Code)

[0142] 3. Existing search clues: The currently known company name is schema:name "ABC Technology Co., Ltd."

[0143] S54, based on the completion task, schedule an external API to query the missing identifier field of the target entity and obtain the completed value of the missing identifier field.

[0144] In this embodiment, API (Application Programming Interface) refers to a standardized interface opened by an external data service provider for querying information, such as the "Tianyan Check API" for querying enterprise business information. This embodiment does not limit this.

[0145] Furthermore, this embodiment can also construct an intelligent scheduling system for external APIs, designing an API selection algorithm based on response time and success rate. Specifically, based on the completion task, multiple external APIs are scheduled to query the missing identifier field of the target entity. An API quality score (API_Score) is calculated based on the response time (T_r), historical success rate (R_s), and data freshness (D_f) of each external API, using the formula: API_Score = 0.3*T_r + 0.4*R_s + 0.3*D_f. When the completion results returned by multiple external APIs are inconsistent, the completion result of the external API with the higher API quality score is preferentially selected as the completion value of the missing identifier field.

[0146] Understandably, in this embodiment, during the RDF mapping process, when the "subject" (i.e., the entity URI) in the triple is identified as a pre-labeled entity type, but the description corresponding to that subject is missing a key identifier field (such as "Unified Social Credit Code"), this semantic event will automatically activate the query chain of the identifier field completion module. For example, even if the original data only contains the company name, it will trigger a real-time call to the business registration API to query and complete the key identifier field "Unified Social Credit Code". This means that the completion action is triggered by the semantic connotation of the data itself, rather than a simple polling scan. Furthermore, after obtaining the missing "Unified Social Credit Code", it is not stored independently, but the completion result is immediately integrated into the original RDF knowledge graph to form a closed loop.

[0147] It is understandable that this embodiment realizes an automated process from "semantic recognition" to "accurate completion" and then to "quality closed loop" through proactive data completion and closed-loop verification based on semantic events. This not only greatly enriches the content of the knowledge graph, but also ensures the accuracy and consistency of the supplemented data, thereby substantially improving the usability of the data in a machine-understandable way, and reflects a leap from "passive processing" to "proactive governance".

[0148] S6, perform feedback optimization on the optimal RDF mapping strategy based on the mapping results.

[0149] It is understandable that in this embodiment, a closed-loop feedback step after the mapping process requires adjusting the weights "based on the mapping results" and feeding them back to the front-end RDF mapping strategy determination process.

[0150] Specifically, the optimal RDF mapping strategy is optimized based on the mapping results, including:

[0151] S61, obtain the latest evaluation index proportions of the Shulin Index, and convert the evaluation index proportions of the Shulin Index into first-level weights.

[0152] For example, taking the latest evaluation indicators of the Data Forest Index as an example, the primary indicator "Data Quality" accounts for 20.5%, and the secondary indicator "Availability" accounts for 3.25%. The tertiary indicators under "Availability" include four core optimization dimensions: "Machine-readable format ratio," "Non-proprietary format ratio," "RDF format support," and "Enterprise identifier field coverage." Therefore, the first-level weight conversion is as follows: the secondary indicator "Availability" accounts for approximately 15.85% of the primary indicator "Data Quality" itself (3.25% / 20.5%). In this embodiment, the proportion of the tertiary indicators is converted into their relative importance in this optimization, i.e., the first-level weight. Currently, these four tertiary indicators have equal weight within "Availability" (this may be adjusted according to policy). Therefore, the first-level weight for each dimension is approximately 15.85% / 4≈3.96%. Then, these four weights are normalized to a sum of 1 (or 100%) for subsequent calculations. Here, after normalization, the first-level weight for each dimension is 0.25.

[0153] S62, dynamically adjust the first-level weights based on the completion results of the missing identifier field to obtain the second-level weights.

[0154] Understandably, this is the core of feedback optimization. In this embodiment, the first-level weights of all dimensions are dynamically adjusted based on the actual effect of the specific operation of "identifier field completion".

[0155] Specifically, in this embodiment, the first-level weights include the enterprise identifier field coverage dimension and other dimensions. The sum of the first-level weights for each dimension is 1. The first-level weights are dynamically adjusted based on the completion results of the missing identifier field to obtain the second-level weights, which include:

[0156] S621, Calculate the success rate and accuracy rate of completing missing identifier fields;

[0157] S622, an adjustment factor is calculated based on the success rate and accuracy rate of completion of the missing identifier field. The adjustment factor is positively correlated with the success rate and accuracy rate of completion.

[0158] It is understandable that the calculation logic of the adjustment factor in this embodiment is: if completion is difficult and the effect is poor, the priority of the target should be temporarily reduced to avoid wasting resources.

[0159] For example, the formula for calculating the adjustment factor α can be: α = (S * w_s + A * w_a) / (w_s + w_a). Here, w_s and w_a are the weights of the completion success rate and completion accuracy, respectively. When both the completion success rate S and the completion accuracy A are high, the adjustment factor α is close to 1, indicating that the task is progressing smoothly and the weights can be maintained; when both the completion success rate S and the completion accuracy A are low, the adjustment factor α is much less than 1.

[0160] S623, multiply the adjustment factor by the first-level weight of the enterprise identification field coverage dimension to obtain the second-level weight of the enterprise identification field coverage dimension.

[0161] S624, based on the second-level weight of the enterprise identifier field covering dimension, adjusts the first-level weight of other dimensions to obtain the second-level weight of other dimensions, so as to keep the sum of the second-level weights of each dimension equal to 1.

[0162] For example, if completion is difficult and ineffective (adjustment factor α is small), the second-level weight of the enterprise identification field coverage dimension is reduced, while the second-level weights of the other three dimensions are increased accordingly to keep the total weight at 1. The adjustment and increase can be achieved by proportionally redistributing the reduced weight of "enterprise identification field coverage", which will not be elaborated in this embodiment.

[0163] Understandably, step S8 ensures a dynamic balance in system resource allocation. When identifier field completion encounters a bottleneck, the system will automatically allocate more resources to other more effective dimensions (such as increasing the RDF conversion ratio).

[0164] S63, based on the second-level weights, performs feedback optimization on the optimal RDF mapping strategy.

[0165] Understandably, the feedback optimization module in this embodiment acts as the "brain" and works continuously. First, it obtains the latest evaluation index proportion of external business objectives (such as the "Shulin Index") and converts it into the first-level weights within the system. More importantly, it dynamically updates the first-level weights based on the completion results (such as completion success rate and accuracy). The adjusted second-level weights will be applied to the front end in real time, affecting the selection of the "optimal RDF mapping strategy", making the entire system an adaptive learning system that can continuously adjust itself according to business objectives and its own execution effect.

[0166] Specifically, in this embodiment, the optimal RDF mapping strategy is optimized based on the second-level weights, including:

[0167] S631 associates each dimension of the second-level weights with each candidate mapping strategy.

[0168] It is understood that this association in this embodiment is based on the core technical characteristics of each candidate mapping strategy and its contribution to the final data form and usability.

[0169] Specifically, for Strategy A: Standard Flat Mapping, the associated second-level weights are the proportion of machine-readable formats and the proportion of non-proprietary formats. This is because the core advantages of the Standard Flat Mapping strategy are fast processing speed, low resource consumption, and strong robustness. Its main contribution is the ability to efficiently and reliably convert data into the most basic RDF format that meets the requirements of machine readability and non-proprietary nature (such as a simple list of triples). Although the RDF semantic structure generated by the Standard Flat Mapping strategy is not rich, it ensures that the data can be processed by machines and is open to access, which is key to meeting the most basic threshold of the "Data Forest Index". Therefore, when the primary goal of the system is to quickly increase the "cardinality" of open data, the Standard Flat Mapping strategy is strongly correlated with these two dimensions.

[0170] For Strategy B: the row-column structure-aware algorithm, the associated second-level weight dimension is the enterprise identifier field coverage. This is because the row-column structure-aware strategy is specifically designed to process tabular data, and most entity data such as enterprises and legal persons in public data exist in tabular form (such as enterprise lists and taxpayer information tables). The row-column structure-aware strategy can accurately parse complex table headers, identify primary key columns and relationships, thereby ensuring that during the RDF conversion process, key identifier fields such as "enterprise name" and "unified social credit code" can be accurately extracted and correctly associated with the corresponding entity URIs. It directly determines whether the extraction of identifier fields is complete and accurate, and therefore is strongly correlated with the "enterprise identifier field coverage" dimension.

[0171] For Strategy C: Depth-First Search (DFS) algorithm, the associated second-level weight dimension is RDF format support. This is because the core value of DFS lies in semantic richness; for nested JSON or XML data, it can construct complex knowledge graphs with hierarchical classes, attributes, and relationships. This significantly improves the semantic clarity, interconnectivity, and reasoning ability of the generated RDF data, perfectly aligning with the higher-order requirements for data relevance and interoperability in the "RDF format support" dimension of the "DataForest Index." The DFS strategy is key to achieving the leap from "machine-readable" to "machine-understandable," and therefore strongly correlated with the "RDF format support" dimension.

[0172] S632, when calculating the fit score between different candidate mapping strategies and the machine-readable format file, the fit score is multiplied by the second-level weight of the corresponding associated dimension of the candidate mapping strategy, and the product is used as the comprehensive score of each candidate mapping strategy.

[0173] S633: Select the strategy with the highest overall score from all candidate mapping strategies as the optimal RDF mapping strategy for the machine-readable format file.

[0174] For example, suppose that due to external API instability, the completion effect of "Enterprise Identifier Field Coverage" has been poor recently, and the system has lowered the second-level weight of the "Enterprise Identifier Field Coverage" dimension. Then, when selecting a strategy for new data, even if Strategy B has a high technical fit score, its overall score will decrease because the second-level weight of the "Enterprise Identifier Field Coverage" dimension, which is strongly correlated with row-column structure awareness strategies, has been lowered. This might lead the system to choose a strategy with slightly lower technical fit but higher weight in its related dimensions (such as "RDF format support") (such as Strategy C) as the optimal RDF mapping strategy for the machine-readable file, thus allocating resources to the direction that is more likely to improve the overall "Data Forest Index". This means that even if a mapping strategy is not technically the most perfect match, it may still be selected if the business objective it serves is extremely important at present.

[0175] In existing technologies, the strategy selection of mapping tools or scripts is not directly related to the final business value or performance requirements (such as the "Data Forest Index" score). In other words, the mapping strategy selection is disconnected from business objectives, failing to guarantee that technological investment can be efficiently transformed into business results. Furthermore, once a mapping strategy is set, it does not automatically adjust with changes in execution performance, and it cannot perceive high-level decisions such as "increasing the RDF mapping ratio is more important than completing the identifier fields at this stage." Therefore, it cannot make intelligent trade-offs and choices when resources are limited, resulting in low overall efficiency. To address this, this embodiment redefines the selection of RDF mapping strategies from a simple technical matching problem into a resource optimization and allocation problem oriented towards business objectives. Its core idea is to introduce dynamic weights as a "command stick," aligning micro-level technical decisions with macro-level business objectives, and enabling the system to self-adjust based on the data availability optimization effect. The entire feedback optimization loop enables precise control of technical strategy selection based on changes in business weights. This allows the entire system to dynamically adjust its work focus based on external assessment standards (Data Forest Index) and internal data status, ensuring that computing resources are always directed to the tasks that most effectively improve the overall Data Forest Index score. This achieves intelligent and dynamic command of macro-strategic goals to micro-technical execution, evolving from a static tool into an intelligent solution that meets dynamic assessment requirements. Ultimately, it solves the problem of low availability of public data caused by the fragmentation of "data format, semantics, and identification," achieving a qualitative leap from "data piling up" to "intelligent data organism."

[0176] Example 2:

[0177] This embodiment only applies to comparisons with... Figure 1 The differences between the two embodiments will be described in the following descriptions. The technical concepts of the remaining designs are similar to those of the first embodiment, and will not be repeated here.

[0178] Specifically, in this embodiment, based on the evaluated format conversion priority score and expected conversion time, the optimal processing path is selected for each original data, which also includes:

[0179] The system monitors the amount of raw data to be converted into a machine-readable format in real time. When the amount of data to be converted exceeds a preset limit, the system uses an external API interface to convert the raw data that exceeds the preset limit as the optimal processing path.

[0180] Understandably, existing technologies typically rely on local computing power to directly convert the format of all public data. However, due to the massive volume, heterogeneity, and dynamic growth of public data, this approach faces significant performance bottlenecks and stability risks. It is highly susceptible to system processing latency spikes or even service crashes due to overloaded computing and memory requirements. This rigid processing model lacks elastic scalability, struggles to cope with fluctuations in data scale, and also results in continuous high consumption and wasted hardware resources. While directly relying on API calls to complete all conversion tasks may alleviate local pressure due to fully externalized processing, it introduces new problems: First, economic cost control is poor, especially for massive amounts of public data, where continuous calls to commercial APIs will generate incalculable costs. Second, data security and privacy risks exist; frequently sending sensitive government or corporate data to third-party services raises compliance concerns. Third, the reliability and consistency of external services are difficult to guarantee; API response delays, call frequency limits, or service interruptions can directly lead to the failure of the entire data processing flow, creating a single point of failure.

[0181] Therefore, this embodiment further proposes an API call strategy for massive data, freeing up limited local computing resources from heavy batch conversion tasks. This helps maximize quality inspection results under limited resources and cope with the complexity of real-world scenarios. For example, when the amount of raw data to be converted into machine-readable format exceeds 10,000 records, the system chooses to call external APIs instead of processing the raw data exceeding the preset limit. This allows the saved local computing power to be concentrated on subsequent core quality verification tasks (such as PDF semantic enhancement and identification field integrity verification), ensuring that limited resources are invested in the most critical quality dimensions. This is not an isolated mechanism, but rather an optimization and self-protection mechanism proactively triggered by the system when it senses that its processing capacity is approaching a bottleneck. This achieves a balance between cost, efficiency, security, and stability, and to a certain extent reflects the intelligent collaboration between modules and between resources and tasks. It is a typical example of the system solving complex problems as an organic whole.

[0182] Example 3:

[0183] Please see the appendix Figure 2 , Figure 2This is a schematic diagram of a public data availability optimization system architecture provided in one embodiment of this specification.

[0184] like Figure 2 As shown, the public data availability optimization system may include at least:

[0185] Data acquisition module 1 is used to acquire raw data from the public data open platform;

[0186] Machine-readable format conversion module 2 is used to convert the original data into machine-readable format files;

[0187] RDF semantic enhancement module 3 is used to generate format feature vectors for each machine-readable format file. The format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty. Based on the format feature vectors, the optimal RDF mapping strategy for each machine-readable format file is determined. Each machine-readable format file is mapped into RDF triples according to the corresponding optimal RDF mapping strategy to form an RDF knowledge graph. During the mapping process, each entity is identified to generate an entity URI. When a pre-annotated entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph.

[0188] Feedback optimization module 4 is used to perform feedback optimization on the optimal RDF mapping strategy based on the mapping results;

[0189] The feedback optimization module is used to perform the following steps:

[0190] Obtain the latest evaluation indicator proportions of the Shulin Index and convert them into first-level weights.

[0191] The first-level weight is dynamically adjusted based on the completion result of the missing identifier field to obtain the second-level weight;

[0192] Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback.

[0193] It is understood that the technical concept of the public data availability optimization system provided in this embodiment is similar to the technical concept of the aforementioned public data availability optimization method, and will not be repeated here.

[0194] The above description is merely a preferred embodiment disclosed in this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0195] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A method for optimizing the availability of public data, characterized in that, Includes the following steps: Obtain raw data from the public data open platform; Convert the raw data into machine-readable file format; Generate format feature vectors for each machine-readable file format, wherein the format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty; Based on the format feature vector, determine the optimal RDF mapping strategy for each machine-readable format file; Each machine-readable file format is mapped into an RDF triple according to the corresponding optimal RDF mapping strategy to form an RDF knowledge graph. During the mapping process, each entity is identified to generate an entity URI. When a pre-labeled entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph. The optimal RDF mapping strategy is optimized based on the mapping results. Among them, the optimal RDF mapping strategy is optimized based on the mapping results, including: Obtain the latest evaluation indicator proportions of the Shulin Index and convert the evaluation indicator proportions of the Shulin Index into first-level weights; The first-level weight is dynamically adjusted based on the completion result of the missing identifier field to obtain the second-level weight; Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback.

2. The public data availability optimization method as described in claim 1, characterized in that, Convert the raw data into machine-readable file formats, including: A pre-trained deep learning model is used to evaluate the format of the original data, and the format conversion priority score and expected conversion time are obtained. Based on the format conversion priority score and expected conversion time obtained from the evaluation, the corresponding optimal processing path is selected for each original data. Each raw data is converted into a machine-readable file according to its corresponding optimal processing path.

3. The public data availability optimization method as described in claim 2, characterized in that, Based on the format conversion priority score and expected conversion time obtained from the evaluation, the optimal processing path is selected for each original data, including: When both the format conversion priority score and the expected conversion time are greater than or equal to the upper limit preset threshold, the deep OCR processing path is selected for the original data. When the format conversion priority score is greater than or equal to the upper limit preset threshold, but the expected conversion time is less than the upper limit preset threshold, the parsing flattening path is selected for the original data; When both the format conversion priority score and the expected conversion time are between the upper and lower preset thresholds, a dynamic SQL query path is selected for the original data. When both the format conversion priority score and the expected conversion time are less than the lower preset threshold, the original data is directly recorded as a machine-readable format file.

4. The public data availability optimization method as described in claim 1, characterized in that, Generate format feature vectors for each machine-readable file format, including: Traverse all levels of the machine-readable file path to obtain the maximum nesting depth and the total number of fields in the machine-readable file. Identify all array structures in machine-readable format files and calculate the array complexity of machine-readable format files based on the identification results; The correlation between tables in a machine-readable file is calculated using a field semantic similarity algorithm. The maximum nesting depth, total number of fields, array complexity, and inter-table relationships of machine-readable format files are weighted and integrated to calculate the field structure complexity score of machine-readable format files.

5. The public data availability optimization method as described in claim 1, characterized in that, Based on format feature vectors, the optimal RDF mapping strategy for each machine-readable format file is determined, including: Establish a candidate mapping strategy matrix, which has preference weights for different candidate mapping strategies relative to each dimension of the format feature vector; The format feature vector of the machine-readable file is weighted and summed with the preference weights of the corresponding dimensions of each candidate mapping strategy to calculate the fit score between different candidate mapping strategies and the machine-readable file. From all candidate mapping strategies, the strategy with the highest fit score is selected as the optimal RDF mapping strategy for this machine-readable format file.

6. The public data availability optimization method as described in claim 1, characterized in that, When a pre-labeled entity type is identified, a key identifier field missing detection is triggered. If a key identifier field is detected to be missing, an external API is invoked to complete it, including: When a pre-labeled entity type is identified, a semantic event is created. The semantic event includes the corresponding entity URI and the identified entity type. Based on semantic events, determine the key identifier fields required by the entity; Verify whether the key identifier fields required for the entity exist in the corresponding RDF knowledge graph. When the verification result is that they do not exist, generate a completion task. The completion task includes the target entity URI, the missing field type, and existing query clues. Based on the completion task, an external API is scheduled to query the missing identifier field of the target entity and obtain the completed value of the missing identifier field.

7. The public data availability optimization method as described in claim 1, characterized in that, The first-level weights include the enterprise identifier field coverage dimension and other dimensions, and the sum of the first-level weights of each dimension is 1. The second-level weights are obtained by dynamically adjusting the first-level weights based on the completion results of the missing identifier field, including: The success rate and accuracy rate of completing missing identifier fields were statistically analyzed. An adjustment factor is calculated based on the completion success rate and completion accuracy of the missing identifier field. The adjustment factor is positively correlated with the completion success rate and completion accuracy. The adjustment factor is multiplied by the first-level weight of the enterprise identification field coverage dimension to obtain the second-level weight of the enterprise identification field coverage dimension; Based on the second-level weight of the enterprise identifier field coverage dimension, the first-level weights of other dimensions are adjusted to obtain the second-level weights of other dimensions, so as to keep the sum of the second-level weights of each dimension equal to 1.

8. The public data availability optimization method as described in claim 5, characterized in that, Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback, including: Each dimension of the second-level weights is associated with each candidate mapping strategy; When calculating the fit score between different candidate mapping strategies and the machine-readable format file, the fit score is multiplied by the second-level weight of the corresponding associated dimension of the candidate mapping strategy, and the product is used as the comprehensive score of each candidate mapping strategy. The strategy with the highest overall score among all candidate mapping strategies is selected as the optimal RDF mapping strategy for this machine-readable file format.

9. A public data availability optimization method as described in claim 2, characterized in that, Based on the format conversion priority score and expected conversion time obtained from the evaluation, the optimal processing path is selected for each original data, which also includes: The system monitors the amount of raw data to be converted into a machine-readable format in real time. When the amount of data to be converted exceeds a preset limit, the system uses an external API interface to convert the raw data that exceeds the preset limit as the optimal processing path.

10. A public data availability optimization system, characterized in that, include: The data acquisition module is used to acquire raw data from the public data open platform; The machine-readable format conversion module is used to convert the raw data into machine-readable format files; The RDF semantic enhancement module is used to generate format feature vectors for each machine-readable format file. The format feature vectors include field structure complexity, data type distribution, and semantic annotation difficulty. Based on the format feature vectors, the optimal RDF mapping strategy for each machine-readable format file is determined. Each machine-readable format file is mapped into RDF triples according to the corresponding optimal RDF mapping strategy to form an RDF knowledge graph. During the mapping process, each entity is identified to generate an entity URI. When a pre-annotated entity type is identified, a key identifier field missing detection is triggered. When a key identifier field is detected to be missing, an external API is scheduled to complete it. The completed value of the missing identifier field is directly injected into the RDF knowledge graph. The feedback optimization module is used to optimize the optimal RDF mapping strategy based on the mapping results. The feedback optimization module is used to perform the following steps: Obtain the latest evaluation indicator proportions of the Shulin Index and convert them into first-level weights. The first-level weight is dynamically adjusted based on the completion result of the missing identifier field to obtain the second-level weight; Based on the second-level weights, the optimal RDF mapping strategy is optimized through feedback.

Images