Intelligent metadata management method and device, equipment and medium

Abstract

The invention relates to the field of big data, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses an intelligent metadata management method, device and equipment and a medium, and the method comprises the following steps: collecting original data of a target business scene from a preset data source; performing label prediction processing on the original data through a pre-trained label generation model to obtain a descriptive label of the original data; generating metadata for describing the original data according to the descriptive tag; calling a pre-constructed adaptive configuration engine, and performing adaptive parameter adjustment on the metadata through the adaptive configuration engine to obtain management configuration of the metadata in the target business scene; and executing a data processing task in the target business scene based on the management configuration of the metadata. The metadata is obtained by intelligently generating the descriptive tag, the integrity and the reliability are ensured, different business requirements can be responded in time by adaptively adjusting the management configuration of the metadata, and the flexibility of metadata management is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an intelligent metadata management method, device, equipment and medium. Background Art

[0002] With the increasing popularity of the internet and the development of IoT technology, big data applications are becoming increasingly common, and the amount of data generated by enterprises every day is exploding. With the prevalence of big data applications, metadata management has become a key step in ensuring data quality and improving data analysis efficiency.

[0003] In the financial sector, banking transaction systems are the core systems used by financial institutions to process various financial transactions, including deposits, withdrawals, transfers, loan issuance, and credit card transactions. This transaction data requires strict management to ensure accuracy and security. Metadata can record transaction sources, times, and types, and identify relationships between different transaction data. Therefore, metadata management can help banks monitor transaction risks. For example, they can monitor the source, amount, and frequency of transactions in real time, identify abnormal transaction behavior, and ensure the compliance of transaction data.

[0004] Electronic health record (EHR) systems are widely used in the healthcare sector for storing and managing patient medical information. This information includes medical records, diagnosis results, treatment plans, medication usage records, and examination reports. Metadata management allows healthcare professionals to quickly retrieve specific patient records and other information. For example, a doctor can quickly find the most recent X-ray report by using the patient ID and examination type, enabling rapid retrieval and sharing of patient information.

[0005] However, existing traditional metadata management systems usually require manual configuration, which not only increases maintenance costs, is prone to errors and makes it difficult to respond to new business needs in a timely manner, but also reduces the reliability and flexibility of metadata management. Summary of the Invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an intelligent metadata management method, device, equipment and medium that can be applied to the medical field, financial technology or other related fields. Its main purpose is to achieve adaptive metadata management configuration and improve the reliability and flexibility of metadata management.

[0007] The technical solutions of the present invention are as follows:

[0008] A first aspect of the present invention provides an intelligent metadata management method, comprising:

[0009] Collect raw data of the target business scenario from the preset data source;

[0010] Performing label prediction processing on the raw data using a pre-trained label generation model to obtain descriptive labels for the raw data;

[0011] generating metadata for describing the original data according to the descriptive tag;

[0012] Invoking a pre-built adaptive configuration engine, and adaptively adjusting parameters of the metadata through the adaptive configuration engine to obtain the management configuration of the metadata under the target business scenario;

[0013] The data processing task in the target business scenario is executed based on the management configuration of the metadata.

[0014] A second aspect of the present invention provides an intelligent metadata management device, comprising:

[0015] The data collection module is used to collect the original data of the target business scenario from the preset data source;

[0016] A label prediction module is used to perform label prediction processing on the raw data using a pre-trained label generation model to obtain descriptive labels for the raw data;

[0017] A metadata generation module, configured to generate metadata for describing the original data based on the descriptive tags;

[0018] An adaptive configuration module, configured to call a pre-built adaptive configuration engine, and adaptively adjust parameters of the metadata through the adaptive configuration engine to obtain a management configuration of the metadata under the target business scenario;

[0019] A task execution module is used to execute the data processing task in the target business scenario based on the management configuration of the metadata.

[0020] A third aspect of the present invention provides a computer device comprising at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned intelligent metadata management method.

[0023] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned intelligent metadata management method.

[0024] Beneficial effects: The present invention discloses an intelligent metadata management method, apparatus, device and medium. Compared with the prior art, the embodiments of the present invention collect the original data of the target business scenario from a preset data source; perform label prediction processing on the original data through a pre-trained label generation model to obtain descriptive labels for the original data; generate metadata for describing the original data based on the descriptive labels; call a pre-built adaptive configuration engine, and perform adaptive parameter adjustment on the metadata through the adaptive configuration engine to obtain the management configuration of the metadata under the target business scenario; and execute data processing tasks in the target business scenario based on the management configuration of the metadata. Obtaining metadata through intelligently generated descriptive labels ensures integrity and reliability, and by adaptively adjusting the management configuration of metadata, it is possible to respond to different business needs in a timely manner, thereby improving the flexibility of metadata management. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A schematic diagram of an application environment of the intelligent metadata management method provided by an embodiment of the present invention;

[0027] Figure 2 A flow chart of the intelligent metadata management method provided by an embodiment of the present invention;

[0028] Figure 3 A flow chart of step S202 in the intelligent metadata management method provided by an embodiment of the present invention;

[0029] Figure 4 A flowchart of step S203 in the intelligent metadata management method provided in an embodiment of the present invention;

[0030] Figure 5 A flow chart of step S204 in the intelligent metadata management method provided by an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of the functional modules of an intelligent metadata management device provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and effects of the present invention more clear and distinct, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below with reference to the accompanying drawings.

[0034] The intelligent metadata management method provided by the embodiment of the present invention can be applied in the following Figure 1 In an application environment, the system includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0036] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a backend server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The backend server may analyze and process the received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0038] It should be noted that the intelligent metadata management method provided in the embodiments of the present application can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the intelligent metadata management apparatus provided in the embodiments of the present invention can also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103. Alternatively, the intelligent metadata management method provided in the embodiments of the present invention can generally be executed by the server 105. Accordingly, the intelligent metadata management apparatus provided in the embodiments of the present invention can generally be provided in the server 105.

[0039] It should be understood that the numbers of the above terminal devices, networks and servers are merely illustrative and any number of terminal devices, networks and servers may be provided as required.

[0040] like Figure 2 As shown, the intelligent metadata management method provided by the embodiment of the present invention specifically includes the following steps:

[0041] S201: Collect original data of the target business scenario from a preset data source.

[0042] In this embodiment, metadata management can be applied in different business scenarios. For example, in the financial sector, metadata management can be used to monitor bank credit card transactions and detect suspicious transaction behavior. In the healthcare sector, it can be used to manage hospital electronic health records (EHRs) to support clinical research and patient care. Therefore, based on different target business scenarios, the preset data source is identified and connected to, and the corresponding raw data is collected from the preset data source. Raw data refers to unprocessed data. Raw data can be unstructured data (such as text, images, audio, etc.) or semi-structured data (such as log files, JSON format data, etc.), thereby improving the data foundation required for metadata management in business scenarios and ensuring the comprehensiveness and integrity of metadata management.

[0043] For example, in the financial field, customer transaction records can be collected from the bank's transaction system, including transaction time, amount, transaction type, etc., and the amount, time, location, merchant type and other information of each transaction can be recorded to support the target business scenario of financial transaction monitoring.

[0044] In the field of healthcare, patients' medical records, examination reports, diagnosis results, etc. can be collected from the hospital's EHR system to record patients' personal information, medical history, diagnosis results, treatment plans, etc. to support the target business scenarios of medical and health record management.

[0045] S202: Perform label prediction processing on the original data using a pre-trained label generation model to obtain descriptive labels for the original data.

[0046] In this embodiment, the collected raw data is intelligently labeled using a pre-trained label generation model. The specific label generation model can be a machine learning model (such as a decision tree, support vector machine, random forest, etc.), or a deep learning model (such as RNN, LSTM, CNN, BERT, etc.). The label generation model is trained with a large amount of labeled data to learn the relationship between data features and labels. The trained label generation model can perform inference and prediction based on the current input raw data, and add corresponding descriptive labels to the raw data. The descriptive label refers to a label used to describe the content of the raw data, such as "financial news", "heart disease patient", etc., so that unstructured data can also be effectively managed and used, so as to understand and manage the raw data more completely and efficiently.

[0047] For example, in the financial field, the collected transaction record data is input into a label generation model, such as generating the labels "large transaction" and "cross-border transaction" for a cross-border transaction with an amount of 100,000 yuan.

[0048] In the field of medical health, the collected medical record text is input into the label generation model. For example, the label "heart disease patient" is generated for a medical record containing keywords such as "angina pectoris" and "abnormal electrocardiogram".

[0049] S203: Generate metadata for describing the original data according to the descriptive tag.

[0050] In this embodiment, metadata is data that describes data and can provide contextual information about the data, such as its source, type, format, creation time, and tags. Therefore, descriptive tags generated through intelligent tagging can provide rich metadata information. By integrating the descriptive tags generated by intelligent tagging with other descriptive data, metadata describing the original data can be generated, providing a richer data description and facilitating data management and analysis.

[0051] For example, in the financial field, the intelligently generated tags "large-value transactions" and "cross-border transactions" are integrated with other transaction description data, including data source, transaction time, data format, etc., to generate complete metadata {"source":"bank transaction system","format":"JSON","creation time":"2025-04-30 10:00","tags":["large-value transactions","cross-border transactions"]}.

[0052] In the healthcare field, the intelligently generated label "heart disease patient" is integrated with other medical record description data, including data source, medical record time, data format, etc., to generate complete metadata {"source":"hospital EHR system","format":"PDF","creation time":"2025-04-30 10:00","label":["heart disease patient","hypertension"]}.

[0053] S204: Invoke a pre-built adaptive configuration engine, and adjust adaptive parameters of the metadata through the adaptive configuration engine to obtain management configuration of the metadata in the target business scenario.

[0054] In this embodiment, a pre-built adaptive configuration engine is called. This adaptive configuration engine is an automated configuration tool based on rules and pattern recognition. By calling this adaptive configuration engine to read the corresponding metadata, adaptive parameters are adjusted to meet the needs of different scenarios. For example, if the tag contains "suspicious transaction", the index depth is increased. When a large amount of new data is detected, the index depth can be dynamically increased to ensure that the system can quickly retrieve and process records marked as "suspicious transaction"; if the tag contains "heart disease patient", the data update frequency is increased, etc., to ensure that the system can promptly update and process medical records marked as "heart disease patient", ensuring the timeliness of the data.

[0055] S205: Execute the data processing task in the target business scenario based on the management configuration of the metadata.

[0056] In this embodiment, based on the adaptively adjusted metadata management configuration, data processing tasks in the target business scenario are executed to ensure efficient and accurate completion of business requirements. Specific data processing tasks can include data retrieval, data analysis, data sharing, and data update tasks. Data retrieval tasks involve quickly retrieving relevant data based on metadata tags and indexes; data analysis tasks utilize metadata information for data analysis and mining; data sharing tasks involve sharing data with other departments or systems based on metadata descriptive tags; and data update tasks involve updating data based on metadata descriptive tags.

[0057] For example, in the financial sector, metadata-based management configuration allows for in-depth analysis of records marked as "suspicious transactions" to detect potential fraud, enabling rapid retrieval and analysis of suspicious transaction records and supporting anti-fraud monitoring. In the healthcare sector, metadata-based management configuration allows for data sharing of medical records marked as "heart disease patients," meeting cross-departmental collaboration needs.

[0058] In the above embodiments, the present invention discloses an intelligent metadata management method, which includes collecting the original data of the target business scenario from a preset data source; performing label prediction processing on the original data through a pre-trained label generation model to obtain descriptive labels of the original data; generating metadata for describing the original data according to the descriptive labels; invoking a pre-constructed adaptive configuration engine to perform adaptive parameter adjustment on the metadata through the adaptive configuration engine to obtain the management configuration of the metadata in the target business scenario; and executing the data processing task in the target business scenario based on the management configuration of the metadata. By obtaining metadata through intelligently generated descriptive labels, integrity and reliability are ensured, and by adaptively adjusting the management configuration of the metadata, different business requirements can be responded to in a timely manner, improving the flexibility of metadata management.

[0059] In one embodiment, as Figure 3 shown, step S202 includes:

[0060] S301. Perform corresponding data preprocessing on the original data according to the data type to obtain the standard data to be predicted;

[0061] S302. Extract features from the standard data to obtain the key features of the standard data;

[0062] S303. Input the key features into a pre-trained label generation model, and perform label prediction on the key features according to the mapping relationship between the features and labels learned in advance, and output the corresponding descriptive labels.

[0063] In this embodiment, when performing intelligent label generation on the original data, first perform corresponding data preprocessing on the original data according to the data type. Since the original data may include multiple data types, such as text, images, numerical values, and structured data, etc., perform corresponding data preprocessing for different data types to obtain the standard data to be predicted. For text data, preprocessing such as word segmentation (dividing the text into words or phrases), removing stop words (removing common meaningless words such as "of", "is", "in", etc.), stemming or lemmatization (restoring the word to its basic form) can be performed; for image data, preprocessing such as image size (adjusting the image to the size required by the model), normalization (normalizing the pixel values to a specific range such as 0 to 1) can be performed; for numerical data, preprocessing such as standardization (standardizing the numerical data to a specific range such as 0 to 1), filling missing values (filling missing values with the mean, median or mode) can be performed; for structured data, key fields can be extracted and encoded. By performing corresponding data preprocessing for different data types, the data quality is improved, and noise and redundancy are reduced.

[0064] After that, feature extraction is performed on the standard data, extracting key features that represent the data content. For example, for text data, text feature vectors can be extracted using technologies such as TF-IDF, Word2Vec, and BERT. For image data, high-level image features can be extracted using convolutional neural networks. For numerical data and structured data, corresponding numerical features and field features can be extracted, etc. By extracting key features that are helpful for model prediction, the data dimension is reduced, and the efficiency and accuracy of model prediction are improved. The extracted key features are input into a pre-trained label generation model. The model predicts labels for the key features based on the mapping relationship between the learned features and labels, outputs the probability distribution of the labels, and selects the label with the highest probability as the final descriptive label. By intelligently generating labels for raw data, the searchability and comprehensibility of the raw data are effectively improved, allowing all types of raw data to be effectively managed and used.

[0065] In one embodiment, Figure 4 As shown, step S203 includes:

[0066] S401: Acquire a predefined metadata structure, where the metadata structure includes a plurality of preset fields;

[0067] S402: Match the descriptive tag with the preset field to confirm the target field that matches the descriptive tag;

[0068] S403: Fill the descriptive tag into the corresponding target field to generate metadata for describing the original data.

[0069] In this embodiment, when generating metadata, a predefined metadata structure is first obtained, that is, the fields that the metadata needs to include are predetermined. These fields can comprehensively describe the characteristics and uses of the original data, such as data source, data type, creation time, data format, label, data quality, etc. The defined metadata structure is stored in a configuration file or database, providing a standardized framework for storing and managing metadata for subsequent use, ensuring the consistency and integrity of the metadata.

[0070] After generating descriptive tags, traverse the preset fields in the metadata structure and match the descriptive tags with the preset fields. For example, the fields can be automatically matched through a rule engine, or the semantic relationship between fields and tags can be automatically identified using natural language processing technology, thereby confirming the target field that matches the descriptive tag. For example, descriptive tags are usually stored in the "tag" field to ensure that the tags are stored correctly, thereby improving the accuracy and usability of metadata.

[0071] Fill the target field with a descriptive label, and combine the filled field with other metadata fields to generate a complete metadata record, which contains information such as the source, type, format, and label of the original data. Store the complete metadata record in the metadata management system for subsequent use.

[0072] For example, metadata for generating transaction records in the financial field:

[0073] {

[0074] "Data Source":"Bank Transaction System",

[0075] "Data Type":"Transaction Record",

[0076] "Creation Time": "2025-04-30 10:00",

[0077] "Data format": "JSON",

[0078] "Tag": ["Large-value transaction", "Cross-border transaction"],

[0079] "Data quality": "High"

[0080] }

[0081] The above metadata is stored in the bank's metadata management system to support subsequent data retrieval and analysis.

[0082] Metadata for generating medical records in the healthcare field:

[0083] {

[0084] "Data Source":"Hospital EHR System",

[0085] "Data format":"PDF",

[0086] "Creation Time": "2025-04-30 10:00",

[0087] "label":["heart disease patients","hypertension"],

[0088] "Data quality": "High"

[0089] }

[0090] The above metadata is stored in the hospital's metadata management system to support clinical research and patient care.

[0091] In one embodiment, Figure 5 As shown, step S204 includes:

[0092] S501: Calling a pre-built adaptive configuration engine, wherein the adaptive configuration engine includes pre-defined configuration rules and pattern recognition models;

[0093] S502: Match the metadata with the predefined configuration rules to obtain a rule matching result;

[0094] S503: Perform data operation pattern recognition on the metadata using the pattern recognition model to obtain a target operation pattern that matches the metadata;

[0095] S504: Adaptively configure the resource parameters and task processing parameters of the metadata according to the rule matching result and the target operation mode, and generate a management configuration of the metadata under the target business scenario.

[0096] In this embodiment, an adaptive configuration engine is pre-built, which includes pre-defined configuration rules and a pattern recognition model. That is, users can define a series of rules according to business needs. These rules can be conditional statements based on data characteristics, operation frequency, data type, etc., for example, "If the data type is text and the keyword contains 'finance', then increase the index depth"; and a pattern recognition model is also built, such as a machine learning-based classifier or clustering model to learn historical operation records and system operation data, so as to identify common data operation patterns. The pre-defined configuration rules and pattern recognition model are combined to build an adaptive configuration engine, which can automatically adjust the configuration parameters of metadata management according to the pre-defined rules and recognized patterns.

[0097] The pre-built adaptive configuration engine is called and initialized, loading pre-defined configuration rules and pattern recognition models. The generated metadata is passed to the adaptive configuration engine, which then matches the metadata against the configuration rules to obtain the corresponding rule matching results. This involves traversing the configuration rules and checking whether the metadata meets the rule conditions, such as "If the medical record tag contains 'heart disease patient,' then increase the index depth." For each rule, the presence or absence of a match and the specific content of the match are recorded. For example, if the tag field in the metadata contains "heart disease patient," the rule is recorded as a successful match.

[0098] The adaptive configuration engine also uses a pattern recognition model to identify data operation patterns in metadata. This model uses historical operation patterns and metadata to learn pattern recognition logic and analyze metadata features, thereby deriving a target operation pattern that matches the metadata. For example, for metadata related to transaction records in the financial sector, the pattern recognition model identifies that it conforms to a "high-frequency query mode" and outputs the target operation mode of "high-frequency query." Alternatively, for metadata related to medical records in the healthcare sector, the pattern recognition model identifies that it conforms to a "low-frequency update mode" and outputs the target operation mode of "low-frequency update." This provides a more flexible and adaptable basis for configuration adjustments.

[0099] Based on rule matching results and the target operation mode, the adaptive configuration engine adaptively adjusts metadata resource parameters and task processing parameters. Resource parameters are system resource configuration parameters, such as index depth and resource allocation, while task processing parameters are task processing configuration parameters, such as data update frequency and query optimization. For example, for transaction record metadata, based on the rule matching result "suspicious transaction" and the target operation mode "high-frequency query," the adaptive configuration engine increases index depth to optimize query efficiency, generating management configurations for "index depth": "increase," "resource allocation": "increase," and "data update frequency": "remain unchanged." For another example, for medical record metadata, based on the rule matching result "heart disease patients" and the target operation mode "low-frequency update," the adaptive configuration engine reduces data update frequency to conserve resources, generating management configurations for "index depth": "remain unchanged," "resource allocation": "decrease," and "data update frequency": "decrease." This system automatically adjusts metadata management system parameter settings based on metadata information, effectively optimizing system performance and resource utilization.

[0100] In one embodiment, after step S205, the method further includes:

[0101] Monitoring resource consumption data when executing the data processing task, and performing statistical analysis on the resource consumption data within a preset time period to obtain corresponding resource consumption distribution information;

[0102] Computing resources are dynamically allocated according to the resource consumption distribution information.

[0103] In this embodiment, during the execution of data processing tasks, the usage of system resources is monitored in real time, including CPU usage, memory usage, disk I / O, network bandwidth, etc. These resource consumption data can be collected through monitoring tools, and statistical analysis can be performed on the resource consumption data within a preset time period, such as calculating statistical indicators such as average value, peak value, and fluctuation range. Based on the statistical analysis results, resource consumption distribution information is generated, specifically, the time distribution of resource usage, task type distribution, etc. can be generated. For example, in a bank's transaction monitoring system, the resource consumption data of transaction processing tasks is monitored, and statistical analysis reveals that CPU and memory usage are higher during peak transaction periods (such as 9:00 a.m. to 5:00 p.m. on weekdays); or in a hospital's electronic health record (EHR) system, the resource consumption data of medical record retrieval tasks is monitored, and statistical analysis reveals that disk I / O is higher during peak medical record retrieval periods (such as 8:00 a.m. to 12:00 p.m.).

[0104] Computing resources are dynamically allocated based on the resource consumption distribution information obtained from statistical analysis. For example, resource allocation can be dynamically adjusted according to resource usage during peak and trough periods, or according to different data processing tasks. This allows computing resources to be dynamically adjusted according to the system load through elastic scaling to meet the performance requirements of different time periods or different data processing tasks, ensuring stable system operation.

[0105] In one embodiment, after step S203, the method further includes:

[0106] extracting data features associated with data quality from the metadata;

[0107] Data quality prediction is performed based on the data characteristics, and corresponding data maintenance operations are automatically triggered based on the quality prediction results.

[0108] In this embodiment, features related to data quality are extracted from metadata, namely, attributes related to data quality, such as completeness, consistency, timeliness, and accuracy. Specifically, data integrity can be obtained by checking whether the data has missing fields or values; data consistency can be obtained by checking whether the data conforms to predefined formats and rules; data timeliness can be obtained by checking the creation and update times of the data; and data accuracy can be obtained by checking whether the data conforms to business logic and domain knowledge. Extracting data features associated with data quality provides quantitative indicators of data quality, helping to assess data reliability.

[0109] Data quality predictions can be made based on the extracted data features. Specifically, a data quality prediction model (such as a machine learning model or a deep learning model) can be used to train by inputting past data features and data quality results. The data quality prediction model learns the relationship between data features and data quality issues. The trained data quality prediction model then predicts data quality for the extracted data features and outputs data quality prediction results, such as "high," "medium," or "low." Based on the data quality prediction results, corresponding data maintenance operations are triggered. For example, if the data quality prediction is "low," a data cleansing operation is triggered; if the data quality prediction is "medium," a data verification operation is triggered; and if the data quality prediction is "high," the data remains unchanged. After metadata is generated, predictive maintenance is performed to predict potential data quality issues and perform data maintenance, ensuring the accuracy and reliability of metadata management.

[0110] It should be noted that there is not necessarily a certain order between the above steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or may be executed interchangeably, etc.

[0111] Further references Figure 6 , as a response to the above Figure 2 The present invention provides an embodiment of an intelligent metadata management device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Figure 6 As shown, the intelligent metadata management device 60 of this embodiment includes:

[0113] The data collection module 601 is used to collect the original data of the target business scenario from a preset data source;

[0114] The label prediction module 602 is used to perform label prediction processing on the original data using a pre-trained label generation model to obtain descriptive labels for the original data;

[0115] A metadata generation module 603 is configured to generate metadata for describing the original data according to the descriptive tags;

[0116] An adaptive configuration module 604 is configured to call a pre-built adaptive configuration engine, and adaptively adjust parameters of the metadata through the adaptive configuration engine to obtain a management configuration of the metadata under the target business scenario;

[0117] The task execution module 605 is used to execute the data processing task in the target business scenario based on the management configuration of the metadata.

[0118] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions. It is more suitable for describing the intelligent metadata management execution process than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0119] In one embodiment, the tag prediction module 602 includes:

[0120] A preprocessing unit, configured to perform corresponding data preprocessing on the raw data according to data type to obtain standard data to be predicted;

[0121] A feature extraction unit, configured to extract features from the standard data to obtain key features of the standard data;

[0122] The label prediction unit is used to input the key features into a pre-trained label generation model, perform label prediction on the key features based on the mapping relationship between the pre-learned features and labels, and output corresponding descriptive labels.

[0123] In one embodiment, the metadata generation module 603 includes:

[0124] A structure acquisition unit, configured to acquire a predefined metadata structure, wherein the metadata structure includes a plurality of preset fields;

[0125] A field matching unit, configured to match the descriptive tag with the preset field and identify a target field that matches the descriptive tag;

[0126] The filling generation unit is used to fill the descriptive tag into the corresponding target field and then generate metadata for describing the original data.

[0127] In one embodiment, the adaptive configuration module 604 includes:

[0128] A calling unit, configured to call a pre-built adaptive configuration engine, wherein the adaptive configuration engine includes pre-defined configuration rules and a pattern recognition model;

[0129] A rule matching unit, configured to perform rule matching on the metadata and the predefined configuration rules to obtain a rule matching result;

[0130] a pattern recognition unit, configured to perform data operation pattern recognition on the metadata using the pattern recognition model to obtain a target operation pattern that matches the metadata;

[0131] An adaptive configuration unit is used to adaptively configure resource parameters and task processing parameters of the metadata according to the rule matching result and the target operation mode, and generate a management configuration of the metadata under the target business scenario.

[0132] In one embodiment, the apparatus 60 further includes:

[0133] A monitoring module is used to monitor resource consumption data when executing the data processing task, and perform statistical analysis on the resource consumption data within a preset time period to obtain corresponding resource consumption distribution information;

[0134] The resource allocation module is used to dynamically allocate computing resources according to the resource consumption distribution information.

[0135] In one embodiment, the device 60 comprises:

[0136] A feature extraction module, configured to extract data features associated with data quality from the metadata;

[0137] The quality prediction module is used to predict data quality based on the data characteristics and automatically trigger corresponding data maintenance operations based on the quality prediction results.

[0138] In one embodiment, the data processing tasks include data retrieval tasks, data analysis tasks, data sharing tasks, and data update tasks.

[0139] In the above embodiment, the present invention discloses an intelligent metadata management device, which collects the original data of the target business scenario from a preset data source; performs label prediction processing on the original data through a pre-trained label generation model to obtain descriptive labels for the original data; generates metadata for describing the original data based on the descriptive labels; calls a pre-built adaptive configuration engine, and uses the adaptive configuration engine to adaptively adjust the parameters of the metadata to obtain the management configuration of the metadata under the target business scenario; and executes data processing tasks in the target business scenario based on the management configuration of the metadata. Obtaining metadata through intelligently generated descriptive labels ensures integrity and reliability, and by adaptively adjusting the management configuration of metadata, it is possible to respond to different business needs in a timely manner, thereby improving the flexibility of metadata management.

[0140] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0141] One or more processors 701 and memory 702, Figure 7In the description, a processor 701 is used as an example. The processor 701 and the memory 702 can be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0142] The processor 701 is used to complete various control logics of the computer device 70. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination of these components. In addition, the processor 701 can also be any traditional processor, microprocessor or state machine. The processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0143] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions corresponding to the intelligent metadata management method in the embodiments of the present invention. Processor 701 executes the non-volatile software programs, instructions, and modules stored in memory 702 to execute various functional applications and data processing functions of computer device 70, thereby implementing the intelligent metadata management method in the above-mentioned method embodiments.

[0144] The memory 702 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device 70, etc. Furthermore, the memory 702 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely located relative to the processor 701, and such remote memory may be connected to the computer device 70 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the intelligent metadata management method in any of the above-described method embodiments are performed.

[0145] In the above embodiment, the present invention discloses a computer device that collects raw data of a target business scenario from a preset data source; performs label prediction processing on the raw data through a pre-trained label generation model to obtain descriptive labels for the raw data; generates metadata for describing the raw data based on the descriptive labels; calls a pre-built adaptive configuration engine, and adaptively adjusts the parameters of the metadata through the adaptive configuration engine to obtain the management configuration of the metadata under the target business scenario; and executes data processing tasks in the target business scenario based on the management configuration of the metadata. Obtaining metadata through intelligently generated descriptive labels ensures integrity and reliability, and by adaptively adjusting the management configuration of metadata, it is possible to respond to different business needs in a timely manner, thereby improving the flexibility of metadata management.

[0146] An embodiment of the present invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the intelligent metadata management method in any of the above method embodiments are performed.

[0147] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which collects raw data of a target business scenario from a preset data source; performs label prediction processing on the raw data through a pre-trained label generation model to obtain descriptive labels for the raw data; generates metadata for describing the raw data based on the descriptive labels; calls a pre-built adaptive configuration engine, and uses the adaptive configuration engine to adaptively adjust parameters of the metadata to obtain the management configuration of the metadata under the target business scenario; and executes data processing tasks in the target business scenario based on the management configuration of the metadata. Obtaining metadata through intelligently generated descriptive labels ensures integrity and reliability, and by adaptively adjusting the management configuration of metadata, it is possible to respond to different business needs in a timely manner, thereby improving the flexibility of metadata management.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0149] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0150] In summary, the present invention discloses an intelligent metadata management method, device, equipment and medium, the method including: collecting raw data of a target business scenario from a preset data source; performing label prediction processing on the raw data through a pre-trained label generation model to obtain descriptive labels for the raw data; generating metadata for describing the raw data based on the descriptive labels; calling a pre-built adaptive configuration engine, and adaptively adjusting the parameters of the metadata through the adaptive configuration engine to obtain the management configuration of the metadata under the target business scenario; and executing data processing tasks in the target business scenario based on the management configuration of the metadata. Obtaining metadata through intelligently generated descriptive labels ensures integrity and reliability, and by adaptively adjusting the management configuration of metadata, it is possible to respond to different business needs in a timely manner, thereby improving the flexibility of metadata management.

[0151] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. The storage medium can be a memory, a magnetic disk, a floppy disk, a flash memory, an optical storage device, etc.

[0152] It should be noted that if any software tools or components not developed by our company appear in the examples of this application, they are for illustration purposes only and do not represent actual use. It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications shall fall within the scope of protection of the appended claims.

Claims

1. An intelligent metadata management method, characterized in that: include: Collect raw data of the target business scenario from the preset data source; Performing label prediction processing on the raw data using a pre-trained label generation model to obtain descriptive labels for the raw data; generating metadata for describing the original data according to the descriptive tag; Invoking a pre-built adaptive configuration engine, and adaptively adjusting parameters of the metadata through the adaptive configuration engine to obtain the management configuration of the metadata under the target business scenario; The data processing task in the target business scenario is executed based on the management configuration of the metadata.

2. The intelligent metadata management method according to claim 1, characterized in that: The label generation model completed by pre-training performs label prediction processing on the original data to obtain descriptive labels for the original data, including: Performing corresponding data preprocessing on the original data according to the data type to obtain standard data to be predicted; Performing feature extraction on the standard data to obtain key features of the standard data; The key features are input into a pre-trained label generation model, and labels are predicted for the key features based on the mapping relationship between the pre-learned features and labels, and corresponding descriptive labels are output.

3. The intelligent metadata management method according to claim 1, characterized in that: Generating metadata for describing the original data according to the descriptive tag includes: Obtaining a predefined metadata structure, wherein the metadata structure includes a plurality of preset fields; Matching the descriptive tag with the preset field to determine the target field that matches the descriptive tag; After the descriptive tag is filled into the corresponding target field, metadata for describing the original data is generated.

4. The intelligent metadata management method according to claim 1, characterized in that: The calling of a pre-built adaptive configuration engine, and adaptively adjusting parameters of the metadata by the adaptive configuration engine to obtain the management configuration of the metadata in the target business scenario, includes: Invoking a pre-built adaptive configuration engine, the adaptive configuration engine including pre-defined configuration rules and pattern recognition models; Matching the metadata with the predefined configuration rules to obtain a rule matching result; Performing data operation pattern recognition on the metadata using the pattern recognition model to obtain a target operation pattern that matches the metadata; The resource parameters and task processing parameters of the metadata are adaptively configured according to the rule matching result and the target operation mode, and the management configuration of the metadata under the target business scenario is generated.

5. The intelligent metadata management method according to claim 4, characterized in that: After executing the data processing task in the target business scenario based on the metadata management configuration, the method further includes: Monitoring resource consumption data when executing the data processing task, and performing statistical analysis on the resource consumption data within a preset time period to obtain corresponding resource consumption distribution information; Computing resources are dynamically allocated according to the resource consumption distribution information.

6. The intelligent metadata management method according to claim 1, characterized in that: After generating metadata for describing the original data according to the descriptive tag, the method further includes: extracting data features associated with data quality from the metadata; Data quality prediction is performed based on the data characteristics, and corresponding data maintenance operations are automatically triggered based on the quality prediction results.

7. The intelligent metadata management method according to any one of claims 1 to 6, characterized in that: The data processing tasks include data retrieval tasks, data analysis tasks, data sharing tasks and data update tasks.

8. An intelligent metadata management device, characterized in that: include: The data collection module is used to collect the original data of the target business scenario from the preset data source; A label prediction module is used to perform label prediction processing on the raw data using a pre-trained label generation model to obtain descriptive labels for the raw data; A metadata generation module, configured to generate metadata for describing the original data based on the descriptive tags; An adaptive configuration module, configured to call a pre-built adaptive configuration engine, and adaptively adjust parameters of the metadata through the adaptive configuration engine to obtain a management configuration of the metadata under the target business scenario; A task execution module is used to execute the data processing task in the target business scenario based on the management configuration of the metadata.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the intelligent metadata management method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the intelligent metadata management method according to any one of claims 1 to 7.

Images