A data relationship modeling method and device for multi-source heterogeneous data sets and a storage medium
By automatically identifying and parsing the structure of multi-source heterogeneous datasets, and utilizing an extensible parsing strategy library and large model parsing strategies, the problems of high manual costs and storage difficulties in the management of multi-source heterogeneous datasets are solved, achieving efficient and accurate data pairing and traceability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 江苏量界数据科技有限公司
- Filing Date
- 2026-03-17
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies struggle to effectively manage multi-source heterogeneous datasets, especially when datasets are complex and diverse in format. This results in high manual costs and a lack of a unified storage structure for paired results, making subsequent tracing and reuse difficult.
By automatically identifying the dataset structure and selecting a parsing strategy, and leveraging the synergistic effect of an extensible parsing strategy library and large model parsing strategies, a set of pairing results is generated. The results are then uniformly stored and verified for reliability through a relational model, ensuring the accuracy and traceability of the pairing relationships.
It reduces manual configuration costs, improves the automation level and universal adaptability of data parsing, ensures the accuracy and traceability of the data pairing process, and provides a reliable data foundation.
Smart Images
Figure CN121858784B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data management technology, and in particular relates to a data relationship modeling method, device and storage medium for multi-source heterogeneous datasets. Background Technology
[0002] In dataset management, evaluation, and annotation governance scenarios, datasets come from complex sources and have diverse formats. Samples may be tables and text, or they may be images, audio, or video files; annotation information may be provided in the form of index files, structured files, or scattered throughout directory structures or file naming rules. In more complex cases, the same dataset may contain multiple annotation methods, such as both index files and structured annotation files, and the final pairing method needs to be determined based on the combination of modality, task, and annotation format.
[0003] Current practices typically rely on manual judgment of dataset types, manual selection of parsing methods, and manual configuration of mapping rules or script writing. As the number of data sources increases and format differences widen, manual and adaptation costs rise significantly. Simultaneously, the lack of a unified storage structure for pairing results makes it difficult to save pairing relationships, location information, version information, and strategy configurations, leading to difficulties in subsequent traceability, auditing, and reuse. Existing data management methods for multi-source heterogeneous datasets often only cover a single annotation organization method, making it difficult to form a closed-loop chain of data integration, relationship modeling, and data reuse, and failing to adapt to multimodal, multi-task, and complex annotation formats. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data relationship modeling method, device and storage medium for multi-source heterogeneous datasets. It can automatically identify the dataset structure and select the parsing strategy, and at the same time save the pairing relationship and location information through a unified data structure, thereby realizing the full-link management of multi-source heterogeneous datasets.
[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:
[0006] In a first aspect, the present invention provides a data relationship modeling method for multi-source heterogeneous datasets, comprising:
[0007] Obtain the dataset to be parsed and obtain the corresponding dataset structure summary based on the dataset; wherein, the dataset structure summary is a structured numerical description of the dataset's organizational structure and content features;
[0008] The parsing path of the dataset to be parsed is determined based on the dataset structure summary; wherein, the parsing path includes modality type, task type and annotation form;
[0009] Based on the parsing path, a parsing strategy is selected from a pre-built extensible parsing strategy library, and a corresponding strategy configuration is generated; wherein, the extensible parsing strategy library includes large model parsing strategies and various rule-based parsing strategies; the strategy configuration includes field table configuration and strategy parameters;
[0010] The selected parsing strategy is executed, and the association inference between the original data and labeled data in the dataset to be parsed is performed according to the corresponding strategy configuration, generating a set of paired results containing the original data locators and labeled data locators;
[0011] The reliability of each pairing relationship in the pairing result set is verified; if the verification fails, the parsing strategy is switched or the strategy configuration is adjusted until the verification passes; if the verification passes, the pairing result set is stored using a pre-built relational model, and the strategy configuration and strategy execution information are recorded for backtracking; wherein, the relational model represents the mapping relationship between the pairing relationship and the location information.
[0012] Optionally, obtaining the corresponding dataset structure summary based on the dataset to be parsed includes:
[0013] Perform directory scanning and file exploration on the dataset to be parsed, count the total number of files, total number of directories and maximum directory depth in the dataset, and count the file type distribution by extension;
[0014] Identify candidate index files and candidate annotation files in the dataset to be parsed, and record their file identifiers, relative paths, file types, and naming pattern matching status; wherein, the candidate index files include table-type files that record the correspondence between the original data and the annotation data, and the candidate annotation files include structured files containing annotation content;
[0015] Determine whether a training / test / validation split directory exists in the dataset to be parsed, and identify the original data directory list and the labeled data directory list in the dataset to be parsed to obtain the directory structure features;
[0016] Determine whether the file names in the dataset to be parsed contain numeric identifiers and whether there is a sequential naming pattern, and extract the common prefix and common suffix information in the file names to obtain naming pattern features;
[0017] Sampling and parsing of the parsable files in the candidate index files or candidate annotation files to extract the set of parsable field names;
[0018] By integrating the file type distribution, candidate index files, candidate labeled files, directory structure features, naming pattern features, and set of resolvable field names, a dataset structure summary is generated.
[0019] Optionally, determining the parsing path of the dataset to be parsed based on the dataset structure summary includes:
[0020] Modal types are determined based on the file type distribution; wherein, the modal types include text, images, audio, and video;
[0021] The task type is determined based on the organizational clues and field name set of the candidate labeled files; wherein, the task type includes classification, detection, segmentation, recognition and extraction;
[0022] The annotation format is determined based on the candidate index file, candidate annotation file, and directory structure characteristics; wherein, the annotation format includes directory structure type, index file type, structured file type, and naming rule type.
[0023] Optionally, the rule-based parsing strategy includes at least an index file parsing strategy, a structured file extraction strategy, and a directory mapping strategy;
[0024] The step of selecting a parsing strategy from a pre-built extensible parsing strategy library based on the parsing path and generating the corresponding strategy configuration includes:
[0025] When a candidate index file exists and meets the preset pattern, the index file parsing strategy is selected first.
[0026] When a structured candidate labeled file exists and the set of field names matches the preset set of key fields, the structured file extraction strategy is selected.
[0027] When the directory structure characteristics indicate that both the original data directory and the labeled data directory exist, select the directory mapping strategy;
[0028] When none of the above rules are met, the large model parsing strategy is triggered;
[0029] The corresponding field table configuration and strategy parameters are generated based on the selected strategy type; wherein, the field table configuration is used to describe the fields for locating the original data, the fields for extracting the labeled content, and the field extraction path, and the strategy parameters are used to describe the scanning range, the separation format, the label extraction method, and the filtering conditions.
[0030] Optionally, the step of executing the selected parsing strategy and performing association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration includes:
[0031] When executing the index file parsing strategy, the original data field and annotation field are extracted from the index file according to the preset delimiter format. When the original data field or annotation field is a file path type, the identifier is extracted from the file path and the matching file is searched within the preset range according to the identifier to generate a pairing relationship between the original data locator and the annotation data locator.
[0032] When executing the structured file extraction strategy, locate the set of data items in the structured file, extract the original data field and annotation field from each data item according to the field table configuration, and when the original data field or annotation field is a file path type, extract the identifier from the file path and search for matching files within a preset range based on the identifier to generate a pairing relationship between the original data locator and the annotation data locator.
[0033] When executing the directory mapping strategy, the original data directory and the annotation data directory are scanned respectively. Identifiers are extracted from the file names according to the matching rules. The correspondence between the original files and the annotation files is established based on the identifiers and filtered according to the extension filtering conditions to generate the pairing relationship between the original data locators and the annotation data locators.
[0034] When executing the large model parsing strategy, the structural summary, candidate file fragments and field name set are input into the pre-trained strategy generation model. The strategy generation model generates the field table configuration and strategy parameters and outputs structured pairing suggestions. Pairing is performed based on the model output.
[0035] Optionally, verifying the reliability of each pairing relationship in the pairing result set includes:
[0036] Randomly select pairing samples from the pairing result set;
[0037] Determine the validity of the location fields, the completeness of the labeled fields, and the consistency between the original data and the labeled data in the paired data samples, respectively.
[0038] The verification pass rate is calculated based on the judgment results. When the pass rate is not lower than the preset threshold, the verification is considered successful.
[0039] Optionally, storing the pairing result set using a pre-built relational model includes:
[0040] Store the pairing results as pairing relationships, recording the unique identifier of the pairing relationship, the original data type, the original data locator, the labeled data type, the labeled data locator, and the creation time;
[0041] Store the file location information as location information, recording the dataset version identifier, file unique identifier, file name, full path relative to the dataset root directory, whether it is a directory, file size, extension, directory depth, parent directory identifier, and modification time;
[0042] Establish a mapping between pairing relationships and location information, so that the original data locators and labeled data locators can be mapped to the corresponding location information.
[0043] Optionally, the modeling method further includes:
[0044] When the parsing path is a combination or multiple candidate parsing strategies meet the rule conditions at the same time, the multiple parsing strategies are executed in order of priority.
[0045] The pairing results from each parsing strategy are merged, and conflict marking is performed for cases where the same original data corresponds to multiple labeled locators, which are then used as input for subsequent conflict resolution.
[0046] Secondly, the present invention provides a data relationship modeling apparatus for multi-source heterogeneous datasets, comprising:
[0047] Dataset structure summary acquisition module: used to acquire the dataset to be parsed and obtain the corresponding dataset structure summary based on the dataset to be parsed; wherein, the dataset structure summary is a structured numerical description of the dataset's organizational structure and content features;
[0048] Parsing path acquisition module: used to determine the parsing path of the dataset to be parsed based on the dataset structure summary; wherein, the parsing path includes modality type, task type and annotation form;
[0049] Parsing strategy acquisition module: used to select a parsing strategy from a pre-built extensible parsing strategy library according to the parsing path and generate a corresponding strategy configuration; wherein, the extensible parsing strategy library includes large model parsing strategies and various rule-based parsing strategies; the strategy configuration includes field table configuration and strategy parameters;
[0050] The pairing result acquisition module is used to execute the selected parsing strategy, perform association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration, and generate a pairing result set containing the original data locators and labeled data locators.
[0051] Relationship modeling module: used to verify the reliability of each pairing relationship in the pairing result set; if the verification fails, the parsing strategy is switched or the strategy configuration is adjusted until the verification passes; if the verification passes, the pairing result set is stored using a pre-built relationship model, and the strategy configuration and strategy execution information are recorded for backtracking; wherein, the relationship model represents the mapping relationship between pairing relationships and location information.
[0052] Thirdly, the present invention provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the data relationship modeling method for multi-source heterogeneous datasets as described in any of the first aspects.
[0053] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: By automatically generating dataset structure summaries and automatically determining parsing paths, the appropriate parsing strategy can be selected without manual judgment of dataset type, significantly reducing the manual configuration cost of multi-source heterogeneous datasets. Through the synergistic cooperation of rule-based strategies and large-model parsing strategies in the scalable parsing strategy library, efficient processing of common scenarios is ensured, while large models cover complex scenarios that are difficult for rules to handle, improving the automation level and general adaptability of data parsing. Furthermore, through the reliability verification and closed-loop optimization mechanism of pairing results, automatic switching or adjustment is performed when the parsing strategy execution effect is poor, ensuring the accuracy of the pairing relationship between the original data and the labeled data. Combined with the unified storage of pairing results and location information by the relation model, and the recording of strategy configuration and execution information for backtracking, the data pairing process is traceable and auditable, and the pairing results can be repeatedly called, providing a reliable data foundation for subsequent dataset governance and quality improvement, thereby reducing manual configuration costs and improving the versatility of data parsing and relation pairing. Attached Figure Description
[0054] Figure 1 The diagram shown is a flowchart of a data relationship modeling method for multi-source heterogeneous datasets in one embodiment of the present invention. Detailed Implementation
[0055] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0056] Example 1
[0057] like Figure 1 As shown, this embodiment provides a data relationship modeling method for multi-source heterogeneous datasets. It generates a structure summary from the multi-source heterogeneous data received from the database system, and uses this structure summary to drive the determination of the parsing path, thereby driving the selection and execution of the parsing strategy and the unified storage of pairing relationships. This achieves automatic pairing and traceable management of raw data and labeled data in multi-source heterogeneous datasets. The specific steps are as follows:
[0058] S1: Dataset Access and Version Initialization
[0059] The database system receives dataset version information submitted by users and creates a file view corresponding to that version. Each file or directory in the dataset version is abstracted as a version file item, and information such as the file's unique identifier, the full path relative to the dataset root directory, whether it is a directory, the directory depth, the parent directory identifier, the file size, the extension, and the modification time are recorded to enable precise location of any file later.
[0060] S2: Dataset Structure Summary Acquisition
[0061] The system recursively scans the dataset starting from the root directory, retrieves the dataset to be parsed, and performs directory scanning and file probing. It then counts the total number of files, directories, and maximum directory depth, and analyzes the file type distribution by extension. For example, audio extensions are categorized as audio types, image extensions as image types, and text, tables, or structured text as index or annotation types.
[0062] Based on the file type distribution, candidate original files, candidate index files, and candidate annotation files are distinguished and identified from the dataset. The file identifier, relative path, file type, and naming pattern matching of candidate index files and candidate annotation files are recorded. Candidate original files include audio files, image files, video files, or text files. Candidate index files include tabular files recording the correspondence between original data and annotation data, such as CSV, TSV, and TXT transcription files, which can be obtained by matching extensions and naming patterns. For example, common naming patterns or common extensions such as transcription, list, and index can be used as clues. Candidate annotation files include structured files containing annotation content, such as JSON, XML, and YAML, which can be obtained based on directory location or extension characteristics.
[0063] Further analysis of directory structure features determines whether common split directories such as training / testing / validation exist in the dataset, and identifies the original data directory list and the labeled data directory list. For example, when some directories mainly contain audio or image files while others mainly contain text or labeled files, they can be categorized into the original data directory and the labeled data directory, respectively. Simultaneously, the naming pattern features of the files are analyzed to determine whether filenames contain numerical identifiers, whether there are sequential naming rules, and common prefixes and suffixes are extracted to assist in strategy selection, extract identifiers during the pairing process, and determine the matching range.
[0064] In addition, parsable files in the candidate index files or candidate annotation files are sampled and parsed to extract a set of parsable field names. For example, header fields are extracted from tables, and common key names are extracted from structured annotation files. In this embodiment, the set of parsable field names is used for key field hit determination during strategy selection, and can also be used for field table configuration initialization during strategy configuration generation.
[0065] By integrating the above-mentioned file statistics and type distribution, candidate index files, candidate labeled files, directory structure features, naming pattern features, and set of resolvable field names, a structured numerical description of the dataset's organizational structure and content features is generated, producing a dataset structure summary for automatic determination of subsequent parsing paths.
[0066] S3: Resolution Path Determination
[0067] The modality type is determined based on the file type distribution in the dataset structure summary, including text, image, audio, and video. The task type is determined based on the organizational clues and the set of resolvable field names of the candidate labeled files, including classification, detection, segmentation, recognition, and extraction. The annotation form is determined based on the candidate index files, candidate labeled files, and directory structure features, including directory structure type, index file type, structured file type, naming rule type, and combinations thereof. In this embodiment, the parsing path can be any one or more of the modality type, task type, and annotation form, used to guide strategy selection, and is not limited to a specific file format. When complex situations occur, the system allows for the determination of a combined form, and multiple parsing strategies can be selected for joint execution in subsequent steps.
[0068] S4: Resolution Strategy Selection and Configuration
[0069] The system selects a parsing strategy from a pre-built, extensible parsing strategy library based on the parsing path and generates the corresponding strategy configuration. In this embodiment, the extensible parsing strategy library includes a large model parsing strategy and various rule-based parsing strategies. The rule-based parsing strategies include at least directory mapping strategies, index file parsing strategies, structured file extraction strategies, and naming rule inference strategies, and support the expansion to cover new annotation formats or annotation organization methods by adding new parsing strategies. Different parsing strategies use a unified output structure to output pairing sets and diagnostic information. The diagnostic information includes at least the number of successful matches and a list of unmatched matches.
[0070] The strategy selection adopts a rule priority mechanism: when a candidate index file exists and meets the preset pattern, the index file parsing strategy is selected first; when a structured candidate annotation file exists and the set of parsable field names matches the preset key field set, the structured file extraction strategy is selected; when the directory structure features indicate that both the original data directory and the annotation data directory exist, the directory mapping strategy is selected; when none of the above rule conditions are met, the large model parsing strategy is triggered as a fallback strategy.
[0071] The selected strategy type generates a corresponding strategy configuration, including field table configuration and strategy parameters. The field table configuration describes the key fields used to locate the original data, the key fields used to extract the labeled content, and the field extraction path or column index information in the labeled data. It is also used to drive field extraction and association inference during the parsing strategy execution process. The strategy parameters describe the scan range, separation format, identifier extraction method, and filtering conditions, etc.
[0072] S5: Pairing Relationship Generation
[0073] The system executes the selected parsing strategy, performs association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration, generates a set of pairing results containing original data locators and labeled data locators, and generates a confidence level for each pairing relationship.
[0074] When executing the index file parsing strategy, the candidate index file content is parsed according to the preset delimiter format and delimiter number in the strategy parameters, extracting the original data column and annotation column data from each record. When the original data or annotation data is a file path type, the identifier is extracted from the path or file name, and a matching file is searched within the preset range determined by the strategy parameters. A pairing relationship between the original data locator and the annotation data locator is generated, and the confidence level is output. For example, if a line of content in the index file is "19-198-0000 CHAPTER ONE", the system can generate the pair "19-198-0000.flac ←→ CHAPTER ONE".
[0075] When executing the structured file extraction strategy, candidate structured labeled files are parsed and the set of data items is located. Based on the field table configuration, the original data fields and label fields are extracted from each data item. When the original data is a file path, matching files are searched within a preset range, pairing relationships are generated, and the confidence level is output. For example, when a structured data item contains an original image path and a category label, a pairing "images / 001.jpg ←→ cat" can be generated.
[0076] When executing the directory mapping strategy, the original data directory list and the labeled data directory list are scanned to obtain the file set. Identifiers are extracted from the file names according to the matching rules in the strategy parameters. The correspondence between the original files and the labeled files is established according to the identifiers. The files are filtered according to the extension filter conditions to generate the pairing relationship and output the confidence level.
[0077] When executing the large model parsing strategy, the dataset structure summary, candidate file fragments and set of parsable field names are input into the pre-trained strategy generation model. The strategy generation model generates field table configuration and strategy parameters and outputs structured pairing suggestions. Pairing is performed based on the model output and diagnostic information is generated.
[0078] S6: Pairing Verification
[0079] The system performs sampling verification on the pairing result set, judging the validity of the positioning fields, the completeness of the annotation fields, and the consistency of the pairing. In this embodiment, the specific judgment logic is as follows: randomly sample pairing relationships from the pairing result set, use a pre-trained pairing verification model to judge the validity of the positioning fields, the completeness of the annotation fields, and the consistency of the pairing between the original data and the annotation data in the pairing relationship samples, and calculate the verification pass rate based on the judgment results. When the pass rate is not lower than a preset threshold, the verification is judged to be successful.
[0080] When the verification fails, a policy configuration adjustment or policy switch is triggered, and execution returns to S4 and S5. If necessary, the parsing path is redefined, and execution returns to S3, S4 and S5 to ensure the reliability of the pairing results.
[0081] S7: Relational Modeling and Storage
[0082] After the verification is passed, the pairing result set is stored using a pre-built relational model. The relational model represents the mapping relationship between the pairing relationship and the location information, and adopts a two-layer structure of pairing relationship-location information.
[0083] The pairing relationship is used to record the core fields of the pairing relationship, including at least the unique identifier of the pairing relationship, the original data type, the original data locator, the labeled data type, the labeled data locator, and the creation time. The location information is used to record file location information and version information, including at least the dataset version identifier, the file unique identifier, the file name, the complete path relative to the dataset root directory, whether it is a directory, the file size, the extension, the directory depth, the parent directory identifier, and the modification time. An association mapping is established between the pairing relationship and the location information, so that the original data locator and the labeled data locator can be mapped to the corresponding location information. In this embodiment, the original data locator and the labeled data locator can be taken as the file unique identifier, or as a normalized relative path that can be uniquely mapped to the file unique identifier.
[0084] The system also records pairing configuration items, which include at least a unique configuration identifier, configuration name, policy type, policy configuration content, and execution statistics such as the total number of pairings generated, the number of successful pairings, the number of warnings, and the number of errors. These statistics are used for pairing process backtracking and auditing. In this embodiment, the system can also mark this configuration as the effective pairing scheme of the current version for subsequent reuse.
[0085] S8: Multi-strategy combination execution
[0086] When the parsing path is a combination or multiple candidate parsing strategies meet the rule conditions simultaneously, the system executes multiple parsing strategies in order of priority, merges the pairing results output by each parsing strategy, and marks the cases where the same original data corresponds to multiple labeled locators as input for subsequent conflict resolution.
[0087] S9: Output and Interconnection
[0088] The system outputs a set of pairing relationships and diagnostic information, including the number of successful matches and the list of unmatched matches, and uses the pairing results as the basic input for subsequent evaluation, annotation management, or unified database entry.
[0089] Example 2
[0090] This embodiment provides a data relationship modeling device for multi-source heterogeneous datasets, including:
[0091] Dataset structure summary acquisition module: used to acquire the dataset to be parsed and obtain the corresponding dataset structure summary based on the dataset to be parsed; wherein, the dataset structure summary is a structured numerical description of the dataset's organizational structure and content features;
[0092] Parsing path acquisition module: used to determine the parsing path of the dataset to be parsed based on the dataset structure summary; wherein, the parsing path includes modality type, task type and annotation form;
[0093] Parsing strategy acquisition module: used to select a parsing strategy from a pre-built extensible parsing strategy library according to the parsing path and generate a corresponding strategy configuration; wherein, the extensible parsing strategy library includes large model parsing strategies and various rule-based parsing strategies; the strategy configuration includes field table configuration and strategy parameters;
[0094] The pairing result acquisition module is used to execute the selected parsing strategy, perform association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration, and generate a pairing result set containing the original data locators and labeled data locators.
[0095] Relationship modeling module: used to verify the reliability of each pairing relationship in the pairing result set; if the verification fails, the parsing strategy is switched or the strategy configuration is adjusted until the verification passes; if the verification passes, the pairing result set is stored using a pre-built relationship model, and the strategy configuration and strategy execution information are recorded for backtracking; wherein, the relationship model represents the mapping relationship between pairing relationships and location information.
[0096] The apparatus provided in this embodiment can execute the data relationship modeling method for multi-source heterogeneous datasets provided in any step of Embodiment 1, and has the corresponding functional modules and beneficial effects of the execution method.
[0097] Example 3
[0098] This embodiment provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the data relationship modeling method for multi-source heterogeneous datasets provided in any step of Embodiment 1.
[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0103] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A data relationship modeling method for multi-source heterogeneous datasets, characterized in that, include: Obtain the dataset to be parsed and obtain the corresponding dataset structure summary based on the dataset; wherein, the dataset structure summary is a structured numerical description of the dataset's organizational structure and content features; The parsing path of the dataset to be parsed is determined based on the dataset structure summary; wherein, the parsing path includes modality type, task type and annotation form; Based on the parsing path, a parsing strategy is selected from a pre-built extensible parsing strategy library, and a corresponding strategy configuration is generated; wherein, the extensible parsing strategy library includes large model parsing strategies and various rule-based parsing strategies; the strategy configuration includes field table configuration and strategy parameters; The selected parsing strategy is executed, and the association inference between the original data and labeled data in the dataset to be parsed is performed according to the corresponding strategy configuration, generating a set of paired results containing the original data locators and labeled data locators; The reliability of each pairing relationship in the pairing result set is verified; if the verification fails, the parsing strategy is switched or the strategy configuration is adjusted until the verification passes; if the verification passes, the pairing result set is stored using a pre-built relational model, and the strategy configuration and strategy execution information are recorded for backtracking; wherein, the relational model represents the mapping relationship between pairing relationships and location information; The step of obtaining the corresponding dataset structure summary based on the dataset to be parsed includes: Perform directory scanning and file exploration on the dataset to be parsed, count the total number of files, total number of directories and maximum directory depth in the dataset, and count the file type distribution by extension; Identify candidate index files and candidate annotation files in the dataset to be parsed, and record their file identifiers, relative paths, file types, and naming pattern matching status; wherein, the candidate index files include table-type files that record the correspondence between the original data and the annotation data, and the candidate annotation files include structured files containing annotation content; Determine whether a training / test / validation split directory exists in the dataset to be parsed, and identify the original data directory list and the labeled data directory list in the dataset to be parsed to obtain the directory structure features; Determine whether the file names in the dataset to be parsed contain numeric identifiers and whether there is a sequential naming pattern, and extract the common prefix and common suffix information in the file names to obtain naming pattern features; Sampling and parsing of the parsable files in the candidate index files or candidate annotation files to extract the set of parsable field names; By integrating the file type distribution, candidate index files, candidate labeled files, directory structure features, naming pattern features, and set of resolvable field names, a dataset structure summary is generated.
2. The data relationship modeling method for multi-source heterogeneous datasets according to claim 1, characterized in that, The step of determining the parsing path of the dataset to be parsed based on the dataset structure summary includes: Modal types are determined based on the file type distribution; wherein, the modal types include text, images, audio, and video; The task type is determined based on the organizational clues and field name set of the candidate labeled files; wherein, the task type includes classification, detection, segmentation, recognition and extraction; The annotation format is determined based on the candidate index file, candidate annotation file, and directory structure characteristics; wherein, the annotation format includes directory structure type, index file type, structured file type, and naming rule type.
3. The data relationship modeling method for multi-source heterogeneous datasets according to claim 2, characterized in that, The rule-based parsing strategy includes at least an index file parsing strategy, a structured file extraction strategy, and a directory mapping strategy. The step of selecting a parsing strategy from a pre-built extensible parsing strategy library based on the parsing path and generating the corresponding strategy configuration includes: When a candidate index file exists and meets the preset pattern, the index file parsing strategy is selected. When a structured candidate labeled file exists and the set of field names matches the preset set of key fields, the structured file extraction strategy is selected. When the directory structure characteristics indicate that both the original data directory and the labeled data directory exist, select the directory mapping strategy; When none of the above rules are met, the large model parsing strategy is triggered; The corresponding field table configuration and strategy parameters are generated based on the selected strategy type; wherein, the field table configuration is used to describe the fields for locating the original data, the fields for extracting the labeled content, and the field extraction path, and the strategy parameters are used to describe the scanning range, the separation format, the label extraction method, and the filtering conditions.
4. The data relationship modeling method for multi-source heterogeneous datasets according to claim 3, characterized in that, The execution of the selected parsing strategy, and the association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration, includes: When executing the index file parsing strategy, the original data field and annotation field are extracted from the index file according to the preset delimiter format. When the original data field or annotation field is a file path type, the identifier is extracted from the file path and the matching file is searched within the preset range according to the identifier to generate a pairing relationship between the original data locator and the annotation data locator. When executing the structured file extraction strategy, locate the set of data items in the structured file, extract the original data field and annotation field from each data item according to the field table configuration, and when the original data field or annotation field is a file path type, extract the identifier from the file path and search for matching files within a preset range based on the identifier to generate a pairing relationship between the original data locator and the annotation data locator. When executing the directory mapping strategy, the original data directory and the annotation data directory are scanned respectively. Identifiers are extracted from the file names according to the matching rules. The correspondence between the original files and the annotation files is established based on the identifiers and filtered according to the extension filtering conditions to generate the pairing relationship between the original data locators and the annotation data locators. When executing the large model parsing strategy, the structural summary, candidate file fragments and field name set are input into the pre-trained strategy generation model. The strategy generation model generates the field table configuration and strategy parameters and outputs structured pairing suggestions. Pairing is performed based on the model output.
5. The data relationship modeling method for multi-source heterogeneous datasets according to claim 1, characterized in that, The verification of the reliability of each pairing relationship in the pairing result set includes: Randomly select pairing samples from the pairing result set; Determine the validity of the location fields, the completeness of the labeled fields, and the consistency between the original data and the labeled data in the paired data samples, respectively. The verification pass rate is calculated based on the judgment results. When the pass rate is not lower than the preset threshold, the verification is considered successful.
6. The data relationship modeling method for multi-source heterogeneous datasets according to claim 1, characterized in that, The method of storing the pairing result set using a pre-built relational model includes: Store the pairing results as pairing relationships, recording the unique identifier of the pairing relationship, the original data type, the original data locator, the labeled data type, the labeled data locator, and the creation time; Store the file location information as location information, recording the dataset version identifier, file unique identifier, file name, full path relative to the dataset root directory, whether it is a directory, file size, extension, directory depth, parent directory identifier, and modification time; Establish a mapping between pairing relationships and location information, so that the original data locators and labeled data locators can be mapped to the corresponding location information.
7. The data relationship modeling method for multi-source heterogeneous datasets according to claim 1, characterized in that, The modeling method also includes: When the parsing path is a combination or multiple candidate parsing strategies meet the rule conditions at the same time, the multiple parsing strategies are executed in order of priority. The pairing results output by each parsing strategy are merged, and conflict marking is performed for cases where the same original data corresponds to multiple labeled locators, which are then used as input for subsequent conflict resolution.
8. A data relationship modeling device for multi-source heterogeneous datasets, characterized in that, include: Dataset structure summary acquisition module: used to acquire the dataset to be parsed and obtain the corresponding dataset structure summary based on the dataset to be parsed; wherein, the dataset structure summary is a structured numerical description of the dataset's organizational structure and content features; Parsing path acquisition module: used to determine the parsing path of the dataset to be parsed based on the dataset structure summary; wherein, the parsing path includes modality type, task type and annotation form; Parsing strategy acquisition module: used to select a parsing strategy from a pre-built extensible parsing strategy library according to the parsing path and generate a corresponding strategy configuration; wherein, the extensible parsing strategy library includes large model parsing strategies and various rule-based parsing strategies; the strategy configuration includes field table configuration and strategy parameters; The pairing result acquisition module is used to execute the selected parsing strategy, perform association inference between the original data and labeled data in the dataset to be parsed according to the corresponding strategy configuration, and generate a pairing result set containing the original data locators and labeled data locators. Relationship modeling module: used to verify the reliability of each pairing relationship in the pairing result set; if the verification fails, the parsing strategy is switched or the strategy configuration is adjusted until the verification passes; if the verification passes, the pairing result set is stored using a pre-built relationship model, and the strategy configuration and strategy execution information are recorded for backtracking; wherein, the relationship model represents the mapping relationship between pairing relationships and location information; The step of obtaining the corresponding dataset structure summary based on the dataset to be parsed includes: Perform directory scanning and file exploration on the dataset to be parsed, count the total number of files, total number of directories and maximum directory depth in the dataset, and count the file type distribution by extension; Identify candidate index files and candidate annotation files in the dataset to be parsed, and record their file identifiers, relative paths, file types, and naming pattern matching status; wherein, the candidate index files include table-type files that record the correspondence between the original data and the annotation data, and the candidate annotation files include structured files containing annotation content; Determine whether a training / test / validation split directory exists in the dataset to be parsed, and identify the original data directory list and the labeled data directory list in the dataset to be parsed to obtain the directory structure features; Determine whether the file names in the dataset to be parsed contain numeric identifiers and whether there is a sequential naming pattern, and extract the common prefix and common suffix information in the file names to obtain naming pattern features; Sampling and parsing of the parsable files in the candidate index files or candidate annotation files to extract the set of parsable field names; By integrating the file type distribution, candidate index files, candidate labeled files, directory structure features, naming pattern features, and set of resolvable field names, a dataset structure summary is generated.
9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data relationship modeling method for multi-source heterogeneous datasets as described in any one of claims 1-7.