Multi-source data resource directory management method and system, electronic equipment and medium
By dividing data type and converting formats to multi-source data, combined with the data label generation model, the problem of low accuracy in the management of multi-source data resource catalogs in the existing technology is solved, and efficient integration and accurate management of multi-source data is achieved.
Patent Information
- Application Number
- CN202411211532.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
AI Technical Summary
Existing data resource directory management methods are difficult to accurately identify different types of data in multiple data sources, resulting in low accuracy in multi-source data resource directory management.
By obtaining the data to be managed by multiple data sources, dividing it into data sample sets of different data types, and format conversion is performed based on the preset data format, inputting the data label generation model to generate accurate data labels, and finally generating data resources with the label Table of contents.
It realizes the effective organization and classification of multi-source heterogeneous data, improves the consistency and efficiency of data processing, enhances the accuracy and relevance of data labels, and thus improves the accuracy of multi-source data resource directory management.
Smart Images

Figure CN119025921A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a multi-source data resource directory management method, system, electronic device and medium. Background Art
[0002] With the rapid development of information technology, data has become an important resource in modern society. In various business scenarios, such as Internet services and military-civilian integration, it is necessary to process and analyze huge data sets from different data sources. In order to effectively manage this data, the establishment and maintenance of data resource directories has become particularly important.
[0003] At present, existing data resource directory management methods usually identify specific types of data to generate data resource directories. However, in practical applications, when processing data from multiple data sources, it is difficult to accurately identify different types of data by only identifying specific types of data. There are errors in the identified data content, which leads to low accuracy of multi-source data resource directory management when identifying multi-source data. Summary of the invention
[0004] The present application provides a multi-source data resource directory management method, system, electronic device and medium, which have the effect of improving the accuracy of multi-source data resource directory management.
[0005] In a first aspect, the present application provides a multi-source data resource directory management method, comprising: Obtain the data to be managed collected from multiple data sources; According to the data type of each of the to-be-managed data, each of the to-be-managed data is divided into a plurality of data sample sets corresponding to different data types; Based on a preset data format, format conversion processing is performed on the data to be managed in each of the data sample sets to obtain a target data sample set corresponding to each of the data sample sets; Inputting each of the target data sample sets into a corresponding data label generation model to obtain a data label for each of the target data sample sets; Combine the data tags to generate a data resource directory.
[0006] In a second aspect of the present application, a system for a multi-source data resource directory management method is provided.
[0007] A data acquisition module is used to acquire the data to be managed collected from multiple data sources; A data sample set determination module, used to determine the data type of each of the to-be-managed data, and divide each of the to-be-managed data into a plurality of data sample sets corresponding to different data types; A data label determination module is used to perform format conversion processing on the data to be managed in each of the data sample sets based on a preset data format to obtain a target data sample set corresponding to each of the data sample sets; input each of the target data sample sets into a corresponding data label generation model to obtain a data label for each of the target data sample sets; The directory generation module is used to generate a data resource directory by combining the data tags.
[0008] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can implement a multi-source data resource directory management method when loaded and executed by the processor.
[0009] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements a multi-source data resource directory management method.
[0010] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By adopting the above technical solution, the data to be managed collected from multiple data sources is obtained, and it is divided into different data sample sets according to the data type, so as to realize the effective organization and classification of multi-source heterogeneous data. Based on the preset data format, each data sample set is format converted to obtain the target data sample set. This step unifies the format of data from different sources and improves the consistency and efficiency of subsequent processing. The target data sample set is input into the corresponding data label generation model to generate accurate data labels. This method of using models for different data types improves the accuracy and relevance of the labels. Finally, by combining various data labels to generate a data resource directory, it can better adapt to different types of data, realize the integration of multi-source data, and thus improve the accuracy of multi-source data resource directory management. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a flowchart of a multi-source data resource directory management method provided by an embodiment of the present application; Figure 2 It is a structural diagram of a multi-source data resource directory management system provided by an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.
[0012] Description of reference numerals: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION
[0013] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.
[0014] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.
[0015] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.
[0016] The present application embodiment provides a multi-source data resource directory management method. In one embodiment, please refer to Figure 1 , Figure 1 This is a flow chart of a multi-source data resource directory management method provided by an embodiment of the present application. The method can be implemented by a computer program, which can be integrated into an application or run as an independent tool application. The method can also be implemented by a single-chip microcomputer or run on a multi-source data resource directory management system based on the von Neumann system. Specifically, the method can include the following steps: Step 101: Acquire data to be managed collected from multiple data sources.
[0017] Among them, data source refers to various sources that can generate or provide data. In the embodiments of the present application, data source can be understood as different channels that can provide raw data, such as internal enterprise databases, Web logs, various sensors, and user uploaded data. These data sources are used to provide raw materials for the subsequent generation of data resource directories, that is, multi-source heterogeneous raw data in various formats and with different semantics.
[0018] Data to be managed refers to raw data obtained from various data sources that need to be managed and utilized. In the embodiments of the present application, data to be managed can be understood as raw data in multi-source heterogeneous formats collected from channels such as databases, logs, and sensors. These data to be managed are used to be input into the data resource directory management process for subsequent processing to achieve formatting, labeling, classification and organization, and finally generate a data resource directory.
[0019] Specifically, in order to establish a complete and accurate multi-source data resource directory, the first step is to collect as comprehensive raw data as possible. This is achieved by building a heterogeneous data acquisition system based on a distributed acquisition framework. The system deploys data acquisition agents to each data source node, such as relational databases, non-relational databases, log file directories, streaming data access terminals, etc., and uses incremental or full methods to obtain new data from the data source, and performs data filtering locally to retain only valuable raw data. At the same time, the system calls different data connectors, and extracts data from various structured, semi-structured, and unstructured data sources according to predefined data extraction logic, using SQL queries or file reading, etc. The collected heterogeneous data is first temporarily stored in the distributed data lake, and then desensitized and cleaned by the data conversion component to remove duplicate and invalid data. The cleaned data is output to the processing module as multi-source heterogeneous data to be managed, and submitted to the subsequent data resource directory generation process for use.
[0020] Step 102: Determine the data type of each to-be-managed data, and divide each to-be-managed data into a plurality of data sample sets corresponding to different data types.
[0021] Among them, data type refers to a collection of features that describe and distinguish data structure, organization form and meaning. In the embodiments of the present application, data type can be understood as an abstract classification of data with different levels of structure, including structured data, semi-structured data and unstructured data. These data types are used to distinguish data to be managed in different formats such as text, image, audio, video, etc. in heterogeneous data, and then divide them into different data sample sets according to the degree of structure of the data, so as to facilitate subsequent customized processing according to the characteristics of the data format.
[0022] A data sample set refers to a collection of data of the same type. In the embodiments of the present application, a data sample set can be understood as a collection of data of the same type obtained by dividing the data to be managed according to its data type, such as a text data sample set, a picture data sample set, etc. These data sample sets are used to extract data of the same type from the original multi-source heterogeneous data, and then corresponding subsequent processing can be performed according to the characteristics of a certain type of data, such as image recognition for image data and natural language processing for text data.
[0023] Specifically, in order to be able to perform customized processing on heterogeneous data in the future, it is necessary to first identify the data type and divide it. This embodiment constructs a data sample library and a data type classification model based on data header information, structural features, value features, etc. Extract data headers, structures and other features from the collected original data to be managed, and calculate the vector representation of each dimensional feature. Then, these vectorized features are matched and calculated with the feature templates of various types of data in the sample library. The sample library contains typical feature templates of various types of data such as structured data, semi-structured data, and unstructured data. Finally, the classification model integrates the multi-dimensional feature similarity and outputs the data type judgment result. After obtaining the data type judgment result, the embodiment uses a derived classification component to divide the data to be managed with the same data type into sample sets of corresponding types according to the results, forming a relational data sample set, an XML data sample set, a text data sample set, an image data sample set, etc. The sample set management module will monitor and maintain the data size of each sample set.
[0024] Based on the above embodiment, as an optional embodiment, in step 102: determining the data type of each data to be managed, and dividing each data to be managed into a plurality of data sample sets corresponding to different data types, this step may also include the following steps: Step 201: Acquire data features of each data to be managed; match the data features with a preset data type feature library to determine the data type of each data to be managed.
[0025] Among them, data features refer to a collection of various attributes that reflect and characterize the structure, composition and meaning of data. In the embodiments of the present application, data features can be understood as various features extracted from heterogeneous data to be managed to describe and distinguish different data types, such as data header information, data structure, content statistical features, etc. These data features are used to help determine which type of data to be managed belongs to by matching with preset data type features, and provide a basis for subsequently dividing different types of data into corresponding sample sets.
[0026] The preset data type feature library refers to a collection of standard feature templates for data type identification. In the embodiments of the present application, the preset data type feature library can be understood as a library of structured data, semi-structured data, and various unstructured data feature templates obtained by statistical analysis of various types of data. These feature templates are used to match and compare with the features of the data to be managed to determine which data type the data to be managed belongs to, so as to classify it into the corresponding sample set.
[0027] Specifically, in order to accurately and efficiently determine the type of multi-source heterogeneous data, it is necessary to first extract the features of the data and then match and determine it with the preset type feature library. Using a custom data feature extractor, multi-dimensional features such as data header content, structural information, field name distribution, word vectors, etc. can be extracted for different source data to form feature vectors. For example, information is extracted from structured data, and unstructured data is converted into word vector representation. Then, a heterogeneous data feature library is pre-built. The library is constructed by sampling and statistical analysis of various types of data, and contains typical feature templates of structured data, semi-structured data, and unstructured data such as images, audio, and text. Then, using the feature matching model of machine learning, the feature vector of the data to be managed is calculated for similarity with various feature templates in the feature library, and the most likely matching feature template and data type judgment result are output. The model can also continuously optimize feature weights through incremental learning. This method makes full use of machine learning technology to automatically determine the type of heterogeneous data to be managed, avoiding the complex process of designing multiple rules, and is more efficient.
[0028] Step 202: According to each data type, the to-be-managed data of the same data type are divided into the same data sample set to obtain a plurality of data sample sets corresponding to different data types.
[0029] Specifically, in order to process different types of data in a targeted manner in the future, it is necessary to further divide the data of the same type into corresponding sample sets. Perform feature extraction and matching on each data to be managed to determine which data type each data belongs to. Then, the heterogeneous data partitioning module uses a type-based derived classification algorithm to automatically divide the data of the same type into corresponding sample sets according to the data type judgment result. For example, all data to be managed that are judged to be structured data enter the structured data sample set, and text data enters the text sample set. The sample set management component continuously monitors the data volume scale of each sample set and maintains a reasonable data volume by cutting or augmenting. Finally, a sample set of structured data, a semi-structured data sample set, and unstructured data such as text, audio, and video is formed. This method of dividing sample sets by data type decouples different types of data, which facilitates subsequent customized processing. At the same time, the reasonable scale of the sample set is maintained, which provides support for training subsequent intelligent data processing models, and finally a data resource directory can be constructed completely and accurately.
[0030] Step 103: Based on a preset data format, format conversion is performed on the data to be managed in each data sample set to obtain a target data sample set corresponding to each data sample set.
[0031] The preset data format refers to the target data representation format used for normalized conversion of heterogeneous data. In the embodiments of the present application, the preset data format can be understood as a unified data format standard predefined for different types of data sample sets, including Schema for structured data, JSON Schema for semi-structured data, etc.
[0032] Specifically, standardized format conversion is required to reduce format differences between heterogeneous data and improve subsequent processing efficiency. For different types of data sample sets, corresponding preset data format standards are constructed, such as unified schema for structured data, JSON Schema for JSON data, etc. These standard formats are pre-designed according to the characteristics of the sample sets. Then, the data of each type of sample set is formatted. The specific approach is: for structured data, fields and types are mapped to the predefined Schema. For JSON data, the structure is adjusted and the information is completed to comply with the JSON Schema. The log format is extracted through the parser. The encoding format of pictures, audio and video is optimized. Through conversion, a target data sample set with unified format is finally obtained, including standardized structured data, standard JSON, optimized logs and multimedia data, etc. Through data format conversion, the format differences between heterogeneous data are eliminated, and the subsequent model can be built on the universality of standardized format data.
[0033] Based on the above embodiment, as an optional embodiment, in step 103: based on the preset data format, the data to be managed in each data sample set is format converted to obtain the target data sample set corresponding to each data sample set. This step may also include the following steps: Step 301: Obtain format information of the data to be managed in each data sample set; compare each format information with a preset data format to determine the matching degree of each data sample set with the preset data format.
[0034] Among them, format information refers to a set of features that describe and reflect format attributes such as data organization structure, storage layout, etc. In the embodiments of the present application, format information can be understood as various information extracted from a heterogeneous data sample set to characterize the data format structure, such as the schema of structured data, encoding parameters of unstructured data, etc.
[0035] The matching degree refers to the degree of conformity between the data format information and the preset format standard. In the embodiment of the present application, the matching degree can be understood as a similarity evaluation result obtained by comparing and analyzing the consistency between the format information of the data sample set and the corresponding preset format standard.
[0036] Specifically, in order to evaluate the difference between the heterogeneous data to be managed and the standard format and guide the subsequent conversion, it is necessary to first extract the format information and calculate the matching degree. By using a custom format analysis component, format composition information can be extracted from various data sample sets. For example, schema information is obtained from structured data, and encoding parameters are obtained from unstructured data. Then, standard formats for various types of data have been preset, such as a unified database schema, optimized multimedia encoding format, etc. The extracted format information is compared with the corresponding standard format item by item. After comparison, the format difference score of each dimension is calculated through a matching algorithm, and the scores of each dimension are combined to evaluate the matching degree of each sample set with the standard format.
[0037] Step 302: Filter out the data to be managed whose matching degree is less than the matching degree threshold as data that does not conform to the format, and convert the data that does not conform to the format into data that conforms to the preset data format according to the preset format conversion rule, to obtain the target data sample set after the format conversion processing of each data sample set.
[0038] The preset format conversion rules refer to a set of preset conversion logic or processes for guiding data format conversion. In the embodiments of the present application, the preset format conversion rules can be understood as format conversion schemes pre-designed for different types of data sample sets, which are used to guide the conversion of data with irregular formats to meet the requirements of the standard format.
[0039] Specifically, in order to eliminate data format differences, it is necessary to convert data with non-standard formats in a targeted manner. According to the format matching results of the previous step, the data to be managed with a matching degree lower than the set threshold, such as 90%, is screened out. This part of the data is determined as data whose format does not meet the standards. Then, for different data types, the system calls the corresponding format converter and converts the data that does not meet the format standards according to the preset conversion rules. For example, the database table structure is adjusted according to the Schema specification, and the non-standard video file is re-encoded using the optimizer. After conversion, all types of data that do not meet the format standards are converted into standardized formats, such as structured data that conforms to the Schema, audio and video files with uniform encoding, etc., and finally a sample set of various target data with unified formats is formed. In this way, by screening data with non-standard formats and performing customized conversions, the differences in data formats can be effectively eliminated so that they meet the preset standard format requirements. This provides standardized data input for subsequent algorithm applications such as content annotation and knowledge extraction.
[0040] Based on the above embodiment, as an optional embodiment, in step 102: converting the data that does not conform to the format into data that conforms to the preset data format according to the preset format conversion rule, this step may also include the following steps: Step 312: Determine the field mapping relationship and conversion operation sequence in the preset format conversion rule; for multiple fields in the data that do not conform to the format, extract the field value of each field.
[0041] The field mapping relationship refers to the correspondence between the fields in the source data format and the fields in the target format. In the embodiments of the present application, the field mapping relationship can be understood as the mapping pairing association between the field name, type, etc. in the structured data in the non-standard format and the fields in the standardized Schema.
[0042] The conversion operation sequence refers to a series of conversion actions and their order for format conversion. In the embodiment of the present application, the conversion operation sequence can be understood as various operation steps and their execution flow for format adjustment and conversion of source data fields according to field mapping relationships.
[0043] A field refers to a specific data item that constitutes an attribute or feature of a data record. In the embodiments of the present application, a field can be understood as a unit that constitutes a structured data record, is used to describe a specific attribute, and is a basic component of a data unit.
[0044] The field value refers to the specific data item content corresponding to the field. In the embodiment of the present application, the field value can be understood as the specific value contained in a field in a specific data record, which is a quantitative expression of the attribute described by the field of the record.
[0045] Specifically, in order to perform format conversion of structured data, it is necessary to first extract the fields and their value information. Based on the preset database Schema mapping rules, determine the correspondence between the source table fields and the target table fields, as well as the conversion operations that need to be performed, such as type conversion, column splitting, etc. Then, the system calls a custom field parser to scan the structured source data that does not conform to the format, and accurately extracts the name and value of each field according to the preset mapping relationship, and stores it as a mapping table of the original field-new field. In this way, by parsing the data of the non-standard table, the complete field and its value information are obtained, and the mapping correspondence between the source field and the target field is established accordingly. This mapping relationship provides input for the subsequent table structure adjustment and data type conversion according to the Schema specification, which can guide the subsequent format conversion process and convert the structured data into a standardized format to facilitate unified storage and management.
[0046] Step 322: Process each field value according to the conversion operation sequence to obtain the target field value; determine the target field in each field according to the field mapping relationship.
[0047] Specifically, in order to realize the conversion of structured data to a standardized format, it is necessary to convert and map the values of each field in turn. First, according to the preset conversion operation sequence, the values of each source field are processed step by step, such as type conversion, default value filling, splitting or merging fields, etc., and finally the converted target field value is obtained. Then, based on the field mapping relationship extracted in the early stage, the system determines which standardized target fields in the converted data format these target field values belong to, and forms a mapping table of target field-target value. Finally, the system uses the generated mapping relationship to construct the converted data record, in which each target field has been converted in content and corresponds to the field in the standard Schema, and the format is unified. Through this process, the source structured data is gradually converted by field, which not only changes the field name and structure, but also adjusts the representation of the field value, and finally realizes the conversion of non-standard data to standard format, providing standardized data input for subsequent centralized management and algorithm processing.
[0048] Step 332: Add the target field value to the target field to obtain data that complies with the preset data format.
[0049] Specifically, in order to form standardized structured data, it is necessary to construct data records in the target format based on the converted fields and values. The system has completed the field value conversion of non-standard data according to the predetermined rules, and generated a mapping relationship between the target field and the target value. The system reads this mapping relationship, and then for each data record, adds its target field values to the corresponding target field according to the mapping relationship to form a standardized record whose field name and structure meet the preset Schema requirements. The process is repeated to finally construct a converted data set containing multiple data records, which realize the conversion of the source non-standard data to the target format in terms of fields and values.
[0050] Step 104: Input each target data sample set into the corresponding data label generation model to obtain the data label of each target data sample set.
[0051] Among them, the data label generation model refers to a pre-trained machine learning model that can automatically analyze data content and generate corresponding semantic labels. In the embodiments of the present application, the data label generation model can be understood as a deep learning model pre-built for different types of data for semantic understanding and annotation, which is used to analyze the semantics of heterogeneous data and generate text labels representing the data content.
[0052] Data tags refer to text identifiers that reflect the key semantics of data content. In the embodiments of the present application, data tags can be understood as keywords or phrases generated by semantic analysis models for heterogeneous data to summarize data content, and are used to describe and understand the semantic connotations of different types of data from a semantic level.
[0053] Specifically, in order to deeply understand the semantics of different types of data, it is necessary to generate data labels in a targeted manner. The system has pre-trained corresponding label generation models for different data types, such as a text annotation model based on BERT and an image classification model based on ResNet. For each target data sample set with a unified format, the system inputs it into the corresponding model, such as text data into the text annotation model and image data into the image classification model. These customized models can analyze the semantics of different types of data and output data labels for each sample, such as text samples are labeled with the label "military-civilian integration" and images are labeled as categories such as "portrait". In this way, each data instance in the data sample set is labeled with a semantic label, which provides input information at the semantic level for subsequent directory construction. At the same time, the use of targeted model annotation enables the semantics of different types of data to be effectively extracted. Through data label generation, heterogeneous data can be described and understood from a semantic perspective, enriching the directory content and laying the foundation for knowledge base construction.
[0054] Based on the above embodiment, as an optional embodiment, in step 104: inputting each target data sample set into the corresponding data label generation model, before this step, the following steps may also be included: Step 401: Obtain historical data sample sets corresponding to multiple types of data and pre-labeled labels corresponding to each historical data sample.
[0055] Specifically, in order to train the data label generation model in the target domain, it is necessary to obtain pre-labeled historical data samples. The system collects various types of historical data related to the enterprise business, such as text data such as past sales records, product descriptions, user feedback, and multimedia data such as pictures and videos. At the same time, these historical data samples have been pre-labeled by domain experts, with text labels reflecting the data content. In this way, a pre-labeled dataset containing different data types is formed, which provides high-quality sample input for subsequent label generation model training. The text labels in the pre-labeled dataset ensure that the samples cover the semantics of the target domain. Model training can obtain key domain information through these samples, thereby generating high-quality semantic labels.
[0056] Step 402: Perform data cleaning on each historical data sample set to obtain a training set corresponding to each historical data sample set.
[0057] Specifically, in order to improve the quality of model training, it is necessary to clean the collected historical samples and construct an optimized training set. The system cleans various historical data samples, such as filtering out invalid samples, deleting duplicate samples, and correcting incorrect annotations. For text data, NLP technology is used to clean samples, filter meaningless samples, and correct semantic tagging errors. For image data, low-quality images are deleted to ensure that the samples are clear and recognizable. For video data, irrelevant content is trimmed and valid clips are retained. Through data cleaning, the quality of the historical sample set is improved and interference with model training is eliminated. Then, the system constructs clean training data sets for different data types, which contain high-quality and reasonably annotated samples. These optimized data sets can better train the data labeling model in the target field, enabling it to capture key semantic information and generate high-quality label outputs.
[0058] Step 403: Perform model training on the preset initial model in combination with each training set to obtain a data label generation model after model training.
[0059] Specifically, in order to obtain a data label model that is adapted to the target domain, it is necessary to use a pre-labeled sample Fine-tune preset model. The system selects pre-trained deep learning models according to different data types, such as a text labeling model based on BERT and an image classification model based on ResNet. Then, the system uses the constructed text, image and other data sets to Fine-tune the above model and adjust the model parameters through sample training. During the training process, the model gradually learns the association between sample data and labeled labels, capturing the key semantic information contained in the data. Ultimately, the label generation models for various types of data are adapted to the target domain through labeled sample training and can accurately parse the data semantics. The trained model can be applied to actual business data as a customized data label generation model to automatically generate text labels representing the data content.
[0060] Step 105: Generate a data resource directory by combining various data tags.
[0061] The data resource directory refers to a directory structure that uniformly organizes, describes, and manages data in multiple heterogeneous data sets. In the embodiments of the present application, the data resource directory can be understood as a tree-like directory structure that establishes semantic associations for different types of data sets such as text, images, and videos based on semantic tag associations of data content, and is used to uniformly summarize and organize multi-source heterogeneous data resources, making it easier for users to fully query and discover related data sets.
[0062] Specifically, in order to uniformly organize and manage heterogeneous data resources, it is necessary to build a data resource directory. The system has generated corresponding semantic labels for each type of data sample set by calling a customized model. The system reads these label data and performs association analysis on the labels of different types of data, and finds that there is a strong association between the "sales" label of text data and the "goods" label of image data. Based on the label association relationship, the system builds a directory structure to describe data resources, and establishes connections between semantically related data sets in the directory, such as placing text sales records and product images in the "sales goods" directory. At the same time, the system generates a directory query interface to receive keyword query requests such as "sales" and return the matching related data set information in the directory. In this way, the directory organizes heterogeneous data resources based on label association, supports rapid retrieval of related data sets, and effectively supports business analysis, decision-making and other needs. By building a data resource directory, unified organization and management of complex and scattered data resources is achieved, and users can quickly locate relevant data content.
[0063] Based on the above embodiment, as an optional embodiment, in step 105: combining various data tags to generate a data resource directory, this step may also include the following steps: Step 501: Create a data resource directory template; format each data tag to obtain a target data tag corresponding to each data tag.
[0064] The data resource directory template refers to a fixed mode that specifies the framework structure, organization form and field representation of the data resource directory. In the embodiment of the present application, the data resource directory template can be understood as a pre-set data directory organization framework for guiding the construction of a standardized and normalized data resource directory.
[0065] Specifically, in order to build a standardized data resource directory, it is necessary to define a directory template and format labels. The system first designs a framework template for the directory based on standardization requirements, including directory field formats, organizational structures, etc. Then, the system reads the various types of raw data labels generated and performs formatting processing, including refining label keywords, deleting invalid content, unifying label styles, etc. After processing, target data labels with unified formats are obtained, which can correspond to the directory template fields. In this way, the directory template determines the standard framework for directory construction, and the formatted labels can be filled into the directory in compliance with regulations. By defining the directory template and adjusting the label format, the standardized construction of the directory can be guided, making it suitable for organizing and managing complex heterogeneous data resources within the enterprise, and better serving data query, analysis, and decision-making needs.
[0066] Based on the above embodiment, as an optional embodiment, in step 501: formatting each data tag to obtain a target data tag corresponding to each data tag, this step may also include the following steps: Step 511: Obtain metadata information corresponding to each data tag, where the metadata information includes data source, data type, and data volume.
[0067] Among them, metadata information refers to structured information that describes the attributes of the data set itself. In the embodiments of the present application, metadata information can be understood as a structured description of the data source, storage format, data volume, etc. that reflects the characteristics of the data set, which is used to supplement the key attributes of the data set itself and help users understand the data set information organized in the directory more comprehensively.
[0068] Specifically, in order to enrich the descriptive information of the data resource directory, it is necessary to obtain the metadata of the data set. The system has generated tags that represent the content of each data set. The system correspondingly obtains the metadata of each data set from the data storage layer, including data source department information, storage type, data volume, etc. For example, the metadata of a text data set includes that it comes from the sales department, is stored in a relational database, and contains 10,000 records. The system associates and binds these metadata with the corresponding data set tags. In this way, each data set in the directory is marked with a tag representing its content, and metadata describing the characteristics of the data set itself is attached. By associating tags and metadata, the directory integrates and organizes data content information and data set attribute information, and can comprehensively describe enterprise data resources.
[0069] Step 521: Add each metadata information to the corresponding data tag to obtain the target data tag corresponding to each data tag.
[0070] Specifically, in order to build a complete data resource directory, metadata needs to be integrated into the label representation. The system has obtained metadata information for each data set, such as data volume, storage format, and other attribute data. The system reads these metadata and associates them with the corresponding labels, such as associating the metadata of the product image dataset with the "product image" label. Then, the system adds metadata fields to each label representation, fills in the metadata of the corresponding dataset, and forms information-rich target data labels.
[0071] Step 502: Fill each target data tag into the data resource directory template, and generate an index corresponding to each target data tag to obtain a data resource directory.
[0072] Specifically, in order to build a standardized data resource directory, it is necessary to fill the processed labels into the preset template and establish an index. The system has obtained various target data labels after formatting. The system reads these labels and fills the key information of the labels into the corresponding directory node fields according to the organizational structure requirements of the directory template to form standardized directory content. At the same time, the system uses the label text to generate an inverted index to support subsequent directory query needs. In this way, the directory template guides the standardized organization of the directory content, the directory nodes and data labels are bound, and the established index supports fast query. The final standardized data resource directory can uniformly organize and manage the distributed, multi-source heterogeneous data resources within the enterprise, and users can quickly locate related data sets through the directory.
[0073] Reference Figure 2 , is a multi-source data resource directory management system provided by an embodiment of the present application, the system includes: a data acquisition module, a data sample set determination module, a data label determination module, and a directory generation module, wherein: A data acquisition module is used to acquire the data to be managed collected from multiple data sources; A data sample set determination module is used to determine the data type of each to-be-managed data, and divide each to-be-managed data into a plurality of data sample sets corresponding to different data types; The data label determination module is used to perform format conversion processing on the data to be managed in each data sample set based on a preset data format to obtain a target data sample set corresponding to each data sample set; input each target data sample set into a corresponding data label generation model to obtain a data label for each target data sample set; The directory generation module is used to generate a data resource directory by combining various data tags.
[0074] Based on the above embodiments, the data sample set determination module is also used to obtain data features of each data to be managed; match the data features with a preset data type feature library to determine the data type of each data to be managed; and divide the data to be managed of the same data type into the same data sample set according to each data type, to obtain data sample sets corresponding to multiple different data types.
[0075] On the basis of the above embodiments, the data sample set determination module is also used to obtain the format information of the data to be managed in each data sample set; compare each format information with the preset data format to determine the matching degree of each data sample set with the preset data format; screen out the data to be managed whose matching degree is less than the matching degree threshold as data that does not conform to the format, and convert the data that does not conform to the format into data that conforms to the preset data format according to the preset format conversion rules, so as to obtain the target data sample set after the format conversion processing of each data sample set.
[0076] Based on the above embodiments, the data sample set determination module is also used to determine the field mapping relationship and conversion operation sequence in the preset format conversion rules; for multiple fields in the data that do not conform to the format, extract the field value of each field; process each field value according to the conversion operation sequence to obtain the target field value; determine the target field in each field according to the field mapping relationship; add the target field value to the target field to obtain data that conforms to the preset data format.
[0077] Based on the above embodiments, the data sample set determination module is also used to obtain historical data sample sets corresponding to multiple types of data and pre-labeled labels corresponding to each historical data sample; perform data cleaning on each historical data sample set to obtain a training set corresponding to each historical data sample set; and perform model training on a preset initial model in combination with each training set to obtain a data label generation model after model training.
[0078] Based on the above embodiment, the directory generation module is also used to establish a data resource directory template; format each data tag to obtain a target data tag corresponding to each data tag; fill each target data tag into the data resource directory template, and generate an index corresponding to each target data tag to obtain a data resource directory.
[0079] Based on the above embodiment, the directory generation module is also used to obtain metadata information corresponding to each data tag, the metadata information includes data source, data type and data volume; each metadata information is added to the corresponding data tag to obtain the target data tag corresponding to each data tag.
[0080] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0081] The present application also discloses an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .
[0082] The communication bus 302 is used to realize the connection and communication between these components.
[0083] The user interface 303 may include a display interface and a camera interface. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.
[0084] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0085] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface diagrams and applications, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.
[0086] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a multi-source data resource directory management method.
[0087] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a multi-source data resource directory management method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.
[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0089] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0091] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.
[0093] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.
[0094] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not recorded in the present disclosure. The description and examples are to be regarded as exemplary only.
Claims
1. A multi-source data resource directory management method, characterized in that: include: Obtain the data to be managed collected from multiple data sources; Determine the data type of each of the to-be-managed data, and divide each of the to-be-managed data into a plurality of data sample sets corresponding to different data types; Based on a preset data format, format conversion processing is performed on the data to be managed in each of the data sample sets to obtain a target data sample set corresponding to each of the data sample sets; Inputting each of the target data sample sets into a corresponding data label generation model to obtain a data label for each of the target data sample sets; Combine the data tags to generate a data resource directory.
2. The multi-source data resource directory management method according to claim 1, characterized in that: The determining of the data type of each of the to-be-managed data and dividing each of the to-be-managed data into a plurality of data sample sets corresponding to different data types includes: Acquiring data characteristics of each of the data to be managed; Matching the data features with a preset data type feature library to determine the data type of each of the data to be managed; According to each of the data types, the to-be-managed data of the same data type are divided into the same data sample set to obtain a plurality of data sample sets corresponding to different data types.
3. The multi-source data resource directory management method according to claim 1, characterized in that: The method of performing format conversion processing on the data to be managed in each of the data sample sets based on the preset data format to obtain a target data sample set corresponding to each of the data sample sets includes: Acquire format information of the data to be managed in each of the data sample sets; Comparing each of the format information with the preset data format to determine the matching degree between each of the data sample sets and the preset data format; The data to be managed whose matching degree is less than the matching degree threshold is screened out as data that does not conform to the format, and according to the preset format conversion rule, the data that does not conform to the format is converted into data that conforms to the preset data format, to obtain the target data sample set after the format conversion processing of each data sample set.
4. The multi-source data resource directory management method according to claim 3, characterized in that: The step of converting the data that does not conform to the format into data that conforms to the preset data format according to the preset format conversion rule includes: Determine the field mapping relationship and conversion operation sequence in the preset format conversion rule; For multiple fields in the data that do not conform to the format, extract the field value of each of the fields; Processing each of the field values according to the conversion operation sequence to obtain a target field value; Determine the target field in each of the fields according to the field mapping relationship; The target field value is added to the target field to obtain the data that conforms to the preset data format.
5. The multi-source data resource directory management method according to claim 1, characterized in that: Before inputting each of the target data sample sets into the corresponding data label generation model, the method further includes: Obtaining historical data sample sets corresponding to multiple types of data and pre-labeled labels corresponding to each of the historical data samples; Performing data cleaning on each of the historical data sample sets to obtain a training set corresponding to each of the historical data sample sets; The preset initial model is trained in combination with each of the training sets to obtain the data label generation model after model training.
6. The multi-source data resource directory management method according to claim 1, characterized in that: The step of combining the data tags to generate a data resource directory includes: Create a data resource directory template; Formatting each of the data tags to obtain a target data tag corresponding to each of the data tags; Fill each of the target data tags into the data resource directory template, and generate an index corresponding to each of the target data tags to obtain the data resource directory.
7. The multi-source data resource directory management method according to claim 6, characterized in that: The formatting of each of the data tags to obtain a target data tag corresponding to each of the data tags includes: Obtaining metadata information corresponding to each of the data tags, the metadata information including data source, data type, and data volume; Each metadata information is added to a corresponding data tag to obtain a target data tag corresponding to each data tag.
8. A multi-source data resource directory management system, characterized in that: The system comprises: A data acquisition module is used to acquire the data to be managed collected from multiple data sources; A data sample set determination module, used to determine the data type of each of the to-be-managed data, and divide each of the to-be-managed data into a plurality of data sample sets corresponding to different data types; A data label determination module is used to perform format conversion processing on the data to be managed in each of the data sample sets based on a preset data format to obtain a target data sample set corresponding to each of the data sample sets; input each of the target data sample sets into a corresponding data label generation model to obtain a data label for each of the target data sample sets; The directory generation module is used to generate a data resource directory by combining the data tags.
9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the multi-source data resource directory management method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the multi-source data resource directory management method according to any one of claims 1 to 7 is executed.
Citation Information
Cited By
Enterprise data asset directory management method and system based on multi-source fusion
CN121349970A
Enterprise data asset directory management method and system based on multi-source fusion
CN121349970B
