Dynamic data management method and device, computer equipment and storage medium
By combining distributed crawler technology with adaptive data models, the problem of low data collection efficiency in the fields of financial technology, healthcare and elderly care has been solved, efficient management and anomaly detection of dynamic data have been achieved, and data timeliness and reliability have been improved.
Patent Information
- Application Number
- CN202510687006.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-26
- Publication Date
- 2025-09-19
AI Technical Summary
In the fields of financial technology, healthcare and elderly care, existing technologies have low data collection efficiency and difficulty in capturing dynamic data updates in a timely manner, resulting in poor data timeliness and an inability to meet the real-time needs of rapid decision-making.
Distributed crawler technology is used to collect dynamic data, and pre-trained adaptive data models are used for classification and anomaly detection, to establish a dynamic database and conduct multi-dimensional analysis and management.
It improves the efficiency and speed of data collection, enhances the searchability and maintainability of data, timely detects and handles data anomalies, and improves the analysis and management efficiency of the data management system.
Smart Images
Figure CN120670482A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a dynamic data management method, apparatus, computer equipment, and storage medium. Background Art
[0002] With the rapid development of the big data era, data has become a vital asset for all businesses and institutions. Taking financial data as an example, existing methods rely primarily on manual operations or small-scale crawling, which not only consumes significant manpower and time, but is also prone to incomplete and inaccurate data collection due to human negligence. Furthermore, the lack of efficient automated mechanisms makes it difficult to capture data updates in a timely manner, significantly compromising the timeliness of data and failing to meet the real-time demands of rapid decision-making. For example, updates to certain key economic indicators may directly impact a company's investment decisions or financial institutions' policy adjustments. However, delayed data acquisition can lead to misguided decisions, resulting in serious consequences.
[0003] The same is true in the medical field. Medical data covers a wide range of aspects, including basic patient information, medical records, test results, and medical imaging. It is of irreplaceable value to the provision of medical services, the improvement of medical quality, the development of medical research, and the formulation of medical policies. However, small-scale data crawling methods can only obtain data within a limited range, making it difficult to fully cover all types of information in the medical field. Furthermore, the timeliness of the data cannot be guaranteed, and data updates cannot be captured in a timely manner, which greatly reduces the timeliness of the data and cannot meet the real-time needs of rapid decision-making. However, the current connection and governance of these public data generally face a series of difficult problems, which seriously restrict the efficient mining and full utilization of data value. Therefore, how to improve the efficiency of data management systems in business fields such as financial technology, medical health, and elderly care for analyzing and managing dynamic data has become a technical problem that needs to be solved urgently. Summary of the Invention
[0004] This application provides a dynamic data management method, device, computer equipment and storage medium to improve the efficiency of data management systems in analyzing and managing dynamic data in business fields such as financial technology, medical health and elderly care.
[0005] In a first aspect, the present application provides a dynamic data management method, the method comprising:
[0006] Collecting various dynamic data through distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database;
[0007] Based on the exception handling mechanism, dynamically detecting each of the dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result;
[0008] Perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results.
[0009] In a second aspect, the present application further provides a dynamic data management device, the device comprising:
[0010] A dynamic database establishment module is used to collect various dynamic data through distributed crawler technology, and classify all the dynamic data according to the type of each dynamic data to establish a dynamic database;
[0011] A data detection result generation module is used to dynamically detect each dynamic data in the dynamic database through a pre-trained adaptive data model based on an exception handling mechanism to generate a data detection result;
[0012] The multi-dimensional analysis management module is used to perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results.
[0013] In a third aspect, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the dynamic data management method as described above when executing the computer program.
[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the dynamic data management method as described above.
[0015] The present application discloses a dynamic data management method, device, computer equipment and storage medium, the dynamic data management method includes collecting various dynamic data through distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database; based on the exception handling mechanism, dynamically detecting each dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result; and performing multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection result. Through the above-mentioned method, the present application collects dynamic data from multiple data sources at the same time through distributed crawler technology, comprehensively obtains dynamic data, greatly improves the efficiency and speed of data collection, classifies and establishes a dynamic database according to the type of dynamic data, facilitates storage and management operations, improves the retrievability and maintainability of data, uses adaptive data models to dynamically detect dynamic data, promptly discovers and handles data anomalies, and improves the efficiency of data management systems in business fields such as financial technology, medical health and elderly care to analyze and manage dynamic data. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0017] Figure 1 is a schematic flow chart of a dynamic data management method provided by the first embodiment of the present application;
[0018] Figure 2 is a schematic flow chart of a dynamic data management method provided in the second embodiment of the present application;
[0019] Figure 3 A schematic block diagram of a dynamic data management device provided in an embodiment of the present application;
[0020] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0022] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.
[0023] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0024] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0025] The embodiments of the present application provide a dynamic data management method, apparatus, computer equipment and storage medium. Among them, the dynamic data management method can be applied to a server, and the distributed crawler technology is used to collect dynamic data from multiple data sources at the same time, and the dynamic data is comprehensively acquired, which greatly improves the efficiency and speed of data collection. The dynamic data is classified according to its type and a dynamic database is established to facilitate storage and management operations, thereby improving the retrievability and maintainability of the data. The adaptive data model is used to dynamically detect dynamic data, and data anomalies are discovered and processed in a timely manner, thereby improving the efficiency of data management systems in business fields such as financial technology, medical health and elderly care in analyzing and managing dynamic data. Among them, the server can be an independent server or a server cluster.
[0026] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.
[0027] See also Figure 1 , Figure 1 This is a schematic flow chart of a dynamic data management method provided in the first embodiment of the present application. The dynamic data management method can be applied to a server and is used to simultaneously collect dynamic data from multiple data sources through distributed crawler technology, comprehensively obtain dynamic data, greatly improve the efficiency and speed of data collection, classify dynamic data according to its type and establish a dynamic database to facilitate storage and management operations, improve the retrievability and maintainability of data, use an adaptive data model to dynamically detect dynamic data, promptly discover and handle data anomalies, and improve the efficiency of data management systems in business fields such as financial technology, medical health and elderly care for analyzing and managing dynamic data.
[0028] like Figure 1 As shown, the dynamic data management method specifically includes steps S10 to S30.
[0029] Step S10: collecting various dynamic data by distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database;
[0030] Specifically, various dynamic data are collected through distributed crawler technology, and all dynamic data are classified according to the type of each dynamic data to establish a dynamic database. Specifically, first configure the distributed crawler, set the target website list, access frequency and data collection rules to ensure the comprehensive and efficient collection of the required dynamic data. Then, perform preliminary preprocessing operations on the collected data to remove invalid data and duplicate data and perform preliminary data format conversion to improve the quality and availability of the data. Then, identify the data type of the preprocessed data and determine the type of each dynamic data by analyzing the data format, content characteristics and metadata information. Finally, all dynamic data are classified according to the identified dynamic data type to form classified data sets of different data types, which are stored in the dynamic database according to the preset database architecture and storage rules.
[0031] Step S20: Based on the exception handling mechanism, dynamically detect each dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result;
[0032] Specifically, a pre-trained adaptive data model is used to extract key features of each dynamic data set, such as statistical and time series characteristics. These key features are then compared with the normal data patterns preset in the model. When data deviates from the normal pattern, an exception handling mechanism is triggered, recording relevant information about the abnormal data, such as the location and severity of the abnormality. Finally, this abnormal data information is integrated to generate data detection results, which are presented to users in an intuitive manner, helping them to promptly understand the quality status of the dynamic data.
[0033] In one embodiment, in combination with a financial business scenario, the real-time monitoring of dynamic data by the pre-trained adaptive data model in this embodiment is described.
[0034] Dynamic data, such as transaction data, customer data, and market data, is collected in real time from multiple financial data sources. The collected raw data is cleaned to remove obvious errors, duplicates, and missing data records, and standardized, converting data in different formats into a unified format for model processing.
[0035] Extract key features of dynamic data such as statistical features, time series features, trading behavior features, and market indicator features.
[0036] Load a pre-trained adaptive data model. The adaptive data model in this embodiment has been trained using a large amount of historical financial data and can identify normal and abnormal trading patterns, market behaviors, etc., and input the pre-processed dynamic data into the adaptive data model in real time.
[0037] The model analyzes key features of the input data and compares them to pre-defined normal data patterns. These patterns are derived through learning and training from historical data and reflect the characteristics of normal financial transactions and market behavior. If data features differ significantly from these patterns and exceed a pre-defined anomaly threshold, the data is considered abnormal. Examples include sudden and significant fluctuations in transaction amounts, unusual increases in transaction frequency, or market volatility exceeding normal ranges.
[0038] Record relevant information of abnormal data and generate data detection results, such as the identification of abnormal data, the degree of abnormality, and abnormality description.
[0039] Step S30: Perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection result.
[0040] Specifically, first, the data in the dynamic database is evaluated for quality by calculating indicators such as the completeness, accuracy, consistency and timeliness of each data item, comprehensively evaluating the overall quality of the data and marking low-quality data; then trend analysis is performed, the data is decomposed into time series, and the moving average method or exponential smoothing method is used to identify the long-term trend of the data, and the cyclical fluctuations in the data are analyzed; then association analysis is carried out, and the association rule mining algorithm is used to mine the association between different data items, and the correlation coefficient is calculated to evaluate the strength of the linear relationship between them; then distribution analysis is performed, and geographic information system technology is used to analyze the geographical distribution characteristics of the data, and the distribution of data in different business fields is analyzed; finally, abnormal data processing is performed, abnormal data is identified according to the data detection results, the causes of its occurrence are analyzed and corresponding treatment measures are taken, such as correcting or deleting abnormal data, in order to have a deeper understanding of the data and provide support for decision-making.
[0041] In one embodiment, this embodiment is described in conjunction with an application scenario in the medical business field.
[0042] Hospital information systems generate a vast amount of dynamic data daily, and these data undergo quality assessments. These assessments examine accuracy (e.g., whether patient test results are within reasonable ranges), completeness (whether there are missing medical records), consistency (whether data across different systems is consistent), and timeliness (whether data is updated promptly). Data that does not meet quality requirements is flagged, and trend analysis is performed on all data, plotting changes in patients' vital signs over time to identify unusual fluctuations. Correlation analysis is then performed to uncover correlations between different test indicators, providing clues for disease diagnosis.
[0043] Based on these analysis results, data can be classified and managed, low-quality data can be corrected or supplemented, abnormal trends and correlations can be investigated in depth, and targeted management strategies can be formulated for data with different dimensional characteristics. This will help improve the quality of medical data, optimize the allocation of medical resources, and enhance the efficiency and quality of medical services.
[0044] The present embodiment discloses a dynamic data management method, device, computer equipment and storage medium, the dynamic data management method includes collecting various dynamic data through distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database; based on the exception handling mechanism, dynamically detecting each dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result; and performing multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection result. Through the above-mentioned method, the present application collects dynamic data from multiple data sources at the same time through distributed crawler technology, comprehensively obtains dynamic data, greatly improves the efficiency and speed of data collection, classifies and establishes a dynamic database according to the type of dynamic data, facilitates storage and management operations, improves the retrievability and maintainability of data, uses adaptive data models to dynamically detect dynamic data, promptly discovers and handles data anomalies, and improves the efficiency of data management systems in business fields such as financial technology, medical health and elderly care to analyze and manage dynamic data.
[0045] based on Figure 1 In the embodiment shown, in this embodiment, step S20 includes:
[0046] Extracting a current data feature vector and a current data label of each of the dynamic data through the pre-trained adaptive data model;
[0047] Specifically, a pre-trained adaptive data model is used to extract features from each dynamic data point in the dynamic database. The adaptive data model analyzes each data sample, extracting a set of numerical values that characterize its characteristics, known as the current data feature vector, and simultaneously determines the corresponding data category label, also known as the current data label. For example, in financial transaction data monitoring, the model might extract features such as transaction amount, transaction time, and transaction frequency to form a feature vector, and label the transaction type (e.g., normal consumption, suspected cash withdrawal, etc.) as the data label.
[0048] Determining a preset data tag that matches the current data tag from the dynamic database;
[0049] Specifically, the current data label is compared with pre-set data labels stored in the dynamic database. Pre-set data labels are pre-defined categories with clear classification significance, based on business needs and historical experience. Through a matching operation, a pre-set data label that matches the current data label is found. For example, if the current data label is "suspected cash-out," a label similarly marked "suspected cash-out" is found from the pre-set data labels.
[0050] The pre-trained adaptive data model is used to dynamically detect each of the dynamic data according to the preset data feature vector corresponding to the preset data label and the current data feature vector to generate the data detection result.
[0051] Specifically, with the help of a pre-trained adaptive data model, a comparative analysis is performed with the current data feature vector based on the preset data feature vector corresponding to the preset data label. The model calculates the similarity or distance between the two and determines whether the current data conforms to the preset pattern based on the preset threshold. If the similarity is within the threshold range, the data is considered normal; otherwise, it is determined to be abnormal data. Ultimately, based on this series of judgments, a data detection result is generated containing information such as the detection status of each dynamic data (normal or abnormal), the degree of abnormality (if applicable), etc. For example, the model will output whether a financial transaction sample is a normal transaction or an abnormal transaction such as suspected cash-out, fraud, etc., as well as the corresponding confidence level or anomaly score.
[0052] See also Figure 2 , Figure 2 This is a schematic flow chart of a dynamic data management method provided in the second embodiment of the present application. This dynamic data management method can be applied to a server and is used to accurately classify different types of dynamic data and establish a clearly structured dynamic database by identifying and analyzing the data format information, content feature information, and metadata information of the dynamic data. The identification and storage of metadata information provides rich background and descriptive information for the data, helping to improve data retrieval efficiency and management convenience, and enhancing the efficiency of dynamic data analysis and management in data management systems in business fields such as financial technology, medical health, and elderly care.
[0053] based on Figure 1 The embodiment shown, this embodiment Figure 2 As shown, step S10 specifically includes steps S101 to S103.
[0054] Step S101: collecting initial data using the distributed crawler technology, and performing data screening and data cleaning operations on the initial data to generate the dynamic data;
[0055] Specifically, each crawler node simultaneously begins data collection on the target website based on a pre-set task allocation and scheduling mechanism. Web page data is obtained according to collection rules and temporarily stored in the distributed crawler system's local storage or distributed storage system. Predefined data filtering rules are used to filter out content unrelated to the target data, such as web page headers, footers, and navigation bars. Parts containing the required data, such as table data and text paragraphs, are retained as valid initial data for subsequent processing.
[0056] Check data records, delete duplicate data items, ensure data uniqueness, identify and correct data errors such as format errors and numerical errors, and supplement or mark missing data fields according to business rules or statistical methods to ensure data integrity. Integrate the cleaned data and convert it into a unified data format and structure to form dynamic data.
[0057] Step S102: Identify the data format information, content feature information and metadata information of the dynamic data;
[0058] Specifically, dynamic data is formatted and identified, such as tables and text. Related features, such as encoding methods and delimiters, are recorded, and content characteristics, including data type, data range, key entities, and data distribution, are extracted. Next, metadata is collected, such as data source (specific website, institution, etc.), generation time, update frequency, data description (indicator meaning, statistical caliber, etc.), and relationships (such as fields associated with other data). This information is integrated to form a comprehensive description of the dynamic data, providing a basis for subsequent data classification and management.
[0059] Step S103: Classify the dynamic data according to the data format information, the content feature information and the metadata information, and establish the dynamic database.
[0060] Specifically, detailed classification standards are developed based on data format information, content feature information, and metadata information. For example, data can be divided into categories such as numerical data, text data, time series data, and geospatial data.
[0061] Perform format scanning on dynamic data to identify its specific data format. According to the data format, store the data into corresponding format categories. For example, store table data into the table data area.
[0062] Analyze the content feature information of the data, such as data type, range, key entities, etc., and further subdivide the data into more specific categories based on the content features. For example, numerical data can be divided into economic indicator data, population indicator data, etc. based on the business indicators it represents.
[0063] Combined with metadata information such as data source, generation time, update frequency, etc., the data can be classified. For example, data from the same website can be grouped into one category, or data updated monthly can be grouped into one category.
[0064] Based on the classification results, design the architecture of the dynamic database, including database table structure, field definition, index settings, etc. According to the designed architecture, create corresponding database tables to prepare for data storage.
[0065] Store categorized dynamic data in a dynamic database according to pre-set storage rules. For example, data in different formats can be stored in corresponding tables or collections, and data with the same metadata characteristics can be stored in the same partition or table. To improve data query efficiency, appropriate indexes are created for database tables based on data classification and query requirements.
[0066] The present embodiment discloses a dynamic data management method, device, computer equipment and storage medium, the dynamic data management method includes collecting initial data through the distributed crawler technology, and performing data screening operations and data cleaning operations on the initial data to generate the dynamic data; identifying the data format information, content feature information and metadata information of the dynamic data; classifying the dynamic data according to the data format information, the content feature information and the metadata information to establish the dynamic database; based on the exception handling mechanism, dynamically detecting each of the dynamic data in the dynamic database through a pre-trained adaptive data model to generate data detection results; performing multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results. Through the above-mentioned method, the present application can accurately classify different types of dynamic data and establish a dynamic database with a clear structure by identifying and analyzing the data format information, content feature information and metadata information of the dynamic data. The identification and storage of metadata information provides rich background and description information for the data, which helps to improve the retrieval efficiency and management convenience of the data, and improves the efficiency of the data management system in the business fields of financial technology, medical health and elderly care to analyze and manage dynamic data.
[0067] based on Figure 2 In the embodiment shown, in this embodiment, step S103 includes:
[0068] Extracting data format keywords, content feature keywords, and metadata keywords of the dynamic data respectively, wherein the data format keywords are keywords of the data format information, the content feature keywords are keywords of the content feature information, and the metadata keywords are keywords of the metadata information;
[0069] Classifying the data format keywords, the content feature keywords, and the metadata keywords to determine at least one group of keyword types;
[0070] The dynamic data are classified according to the keyword types, and the dynamic database is established based on the classified dynamic data.
[0071] Specifically, we analyze the data format of dynamic data, extract keywords that can describe the data format, analyze the content characteristics of dynamic data, identify and extract keywords that reflect the characteristics of the data content, and sort out the metadata information of dynamic data to select key metadata as keywords.
[0072] Predefine keyword types based on business needs and data characteristics. For example, you could categorize keyword types into "data format," "data content," "data source," "update frequency," etc.
[0073] Each extracted keyword is matched against predefined keyword types. Through methods such as text comparison and semantic analysis, each keyword is classified into a corresponding type. For example, keywords such as "table" and "JSON (JavaScript Object Notation)" are classified as "data format"; keywords such as "GDP (Gross Domestic Product)" and "CPI (Consumer Price Index)" are classified as "data content"; and keywords such as "financial institution" and "monthly" are classified as "data source" and "update frequency" respectively.
[0074] Develop classification rules based on keyword types. For example, all data containing "economic indicator" keywords such as "GDP" and "CPI" could be classified as "economic data"; all data with "monthly" update frequency keywords could be classified as "highly updated data."
[0075] According to the classification rules, dynamic data is divided into different categories, each data object is associated with the extracted keywords and their types, and the data is placed in the corresponding category according to the rules.
[0076] In a specific embodiment, step S30 includes:
[0077] Analyzing the data detection results by the pre-trained adaptive data model to determine the quality assessment results, trend analysis results, correlation analysis results and / or distribution analysis results of each of the dynamic data;
[0078] Performing quantitative evaluations on the quality assessment results, the trend analysis results, the correlation analysis results, and / or the distribution analysis results based on a preset multi-dimensional analysis system to determine quantitative scores for each dimension of the dynamic data;
[0079] Perform multi-dimensional analysis and management on each of the dynamic data based on the quantitative scores of each dimension.
[0080] Specifically, a pre-trained adaptive data model is used to analyze data detection results and determine the quality assessment, trend analysis, correlation analysis, and distribution analysis results of dynamic data. Based on a pre-set multi-dimensional analysis system, the quality, trend, correlation, and distribution analysis results are quantitatively evaluated to determine the quantitative score of each dynamic data in each dimension.
[0081] Based on quantitative scores across various dimensions, dynamic data can be analyzed and managed across multiple dimensions. Quality scores can be used to screen high-quality data, trend scores can be used to predict the future, correlation scores can be used to explore connections between data, and distribution scores can be used to understand data coverage. For example, in the financial sector, quantitative scores can be used to identify high-risk transaction data, allowing proactive mitigation measures.
[0082] based on Figure 1 In the embodiment shown, in this embodiment, after step S30, the following steps are further included:
[0083] determining an update frequency according to the amount of the dynamic data in the dynamic database;
[0084] performing data update detection on each of the dynamic data in the dynamic database according to the update frequency;
[0085] When it is detected that there is data update in the dynamic data, an update operation is performed on the dynamic data through the pre-trained adaptive data model, wherein the update operation includes a modification operation, a supplement operation and / or a deletion operation.
[0086] Specifically, the update frequency is set based on the amount of dynamic data in the dynamic database. Data with large volumes and high real-time requirements can be updated in real time or at a high frequency, while data with small volumes and slow changes can be updated in a low frequency. Based on the set update frequency, each dynamic data in the dynamic database is monitored regularly or in real time, and the timestamps, version numbers, or hash values of the new and old data are compared to determine whether the data has been updated.
[0087] When an update is detected for dynamic data, the pre-trained adaptive data model is used to perform update operations. For example, the modification operation corrects errors or outdated information in the data; the supplement operation adds new fields or records; and the deletion operation removes duplicate, invalid, or irrelevant data.
[0088] Based on any of the above embodiments, in this embodiment, before step S20, the following steps are included:
[0089] Acquire a historical data set, determine the data type of each historical data in the historical data set using an original adaptive data model, and generate an initial data label for each historical data according to the data type;
[0090] Extracting data features and data structures of each of the historical data, and generating target data labels based on the data features and the data structures;
[0091] Comparing each of the initial data labels with each of the target data labels using a preset deep learning algorithm of the original adaptive data model to generate a historical detection result of the historical data;
[0092] The model internal parameters of the original adaptive data model are adjusted according to the historical detection results to determine the pre-trained adaptive data model.
[0093] See also Figure 3 , Figure 3 This is a schematic block diagram of a dynamic data management device provided by an embodiment of the present application, wherein the dynamic data management device is used to execute the aforementioned dynamic data management method.
[0094] like Figure 3 As shown, the dynamic data management device 400 includes:
[0095] A dynamic database establishment module 410 is used to collect various dynamic data using a distributed crawler technology, and classify all the dynamic data according to the type of each dynamic data to establish a dynamic database;
[0096] The data detection result generating module 420 is configured to dynamically detect each dynamic data in the dynamic database through a pre-trained adaptive data model based on an exception handling mechanism, and generate a data detection result;
[0097] The multi-dimensional analysis management module 430 is used to perform multi-dimensional analysis management on all the dynamic data in the dynamic database according to the data detection results.
[0098] Furthermore, the data detection result generating module 420 includes:
[0099] A vector and label extraction unit, configured to extract a current data feature vector and a current data label of each of the dynamic data using the pre-trained adaptive data model;
[0100] a preset data tag determining unit, configured to determine a preset data tag matching the current data tag from the dynamic database;
[0101] A data detection result generating unit is used to dynamically detect each of the dynamic data according to the preset data feature vector corresponding to the preset data label and the current data feature vector through the pre-trained adaptive data model to generate the data detection result.
[0102] Furthermore, the dynamic database establishment module 410 includes:
[0103] A dynamic data generating unit, configured to collect initial data using the distributed crawler technology, and perform data screening and data cleaning operations on the initial data to generate the dynamic data;
[0104] an information identification unit, configured to identify data format information, content feature information, and metadata information of the dynamic data;
[0105] The dynamic database establishing unit is used to classify the dynamic data according to the data format information, the content feature information and the metadata information, and establish the dynamic database.
[0106] Furthermore, the dynamic database establishment unit includes:
[0107] a keyword extraction subunit, configured to extract data format keywords, content feature keywords, and metadata keywords of the dynamic data, respectively, wherein the data format keywords are keywords of the data format information, the content feature keywords are keywords of the content feature information, and the metadata keywords are keywords of the metadata information;
[0108] a keyword type determination subunit, configured to classify the data format keywords, the content feature keywords, and the metadata keywords to determine at least one group of keyword types;
[0109] The dynamic database establishing subunit is used to classify the dynamic data according to each keyword type and establish the dynamic database according to each classified dynamic data.
[0110] Furthermore, the multi-dimensional analysis management module 430 includes:
[0111] a result parsing unit, configured to parse the data detection results using the pre-trained adaptive data model to determine a quality assessment result, a trend analysis result, a correlation analysis result, and / or a distribution analysis result of each of the dynamic data;
[0112] a dimensional quantitative score determination unit, configured to perform quantitative evaluation on the quality assessment result, the trend analysis result, the correlation analysis result, and / or the distribution analysis result based on a preset multi-dimensional analysis system, and determine a quantitative score for each dimension of each dynamic data;
[0113] The multi-dimensional analysis management unit is used to perform multi-dimensional analysis and management on each of the dynamic data according to the quantitative scores of each dimension.
[0114] Furthermore, the dynamic data management device 400 includes:
[0115] An update frequency determination module, configured to determine an update frequency according to the amount of the dynamic data in the dynamic database;
[0116] A data update detection module, configured to perform data update detection on each of the dynamic data in the dynamic database according to the update frequency;
[0117] The update operation module is used to perform an update operation on each dynamic data through the pre-trained adaptive data model when it is detected that there is a data update in the dynamic data, wherein the update operation includes a modification operation, a supplement operation and / or a deletion operation.
[0118] Furthermore, the dynamic data management device 400 includes:
[0119] An initial data label generation module is used to obtain a historical data set, determine the data type of each historical data in the historical data set through an original adaptive data model, and generate an initial data label for each historical data according to the data type;
[0120] a target data label generation module, configured to extract data features and data structures of each of the historical data, and generate target data labels based on the data features and data structures;
[0121] A historical detection result generating module, configured to compare each of the initial data labels with each of the target data labels using a preset deep learning algorithm of the original adaptive data model to generate a historical detection result of the historical data;
[0122] The adaptive data model training module is used to adjust the model internal parameters of the original adaptive data model according to the historical detection results to determine the pre-trained adaptive data model.
[0123] It should be noted that those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0124] The above-mentioned device can be realized in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.
[0125] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0126] See Figure 4 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0127] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any dynamic data management method.
[0128] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0129] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any dynamic data management method.
[0130] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0131] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0132] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:
[0133] Collecting various dynamic data through distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database;
[0134] Based on the exception handling mechanism, dynamically detecting each of the dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result;
[0135] Perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results.
[0136] In one embodiment, based on the exception handling mechanism, each dynamic data in the dynamic database is dynamically detected by a pre-trained adaptive data model to generate a data detection result for achieving:
[0137] Extracting a current data feature vector and a current data label of each of the dynamic data through the pre-trained adaptive data model;
[0138] Determining a preset data tag that matches the current data tag from the dynamic database;
[0139] The pre-trained adaptive data model is used to dynamically detect each of the dynamic data according to the preset data feature vector corresponding to the preset data label and the current data feature vector to generate the data detection result.
[0140] In one embodiment, the dynamic data are collected by distributed crawler technology, and all the dynamic data are classified according to the type of each dynamic data to establish a dynamic database for achieving:
[0141] Collecting initial data through the distributed crawler technology, and performing data screening and data cleaning operations on the initial data to generate the dynamic data;
[0142] Identifying data format information, content feature information, and metadata information of the dynamic data;
[0143] The dynamic data is classified according to the data format information, the content feature information and the metadata information to establish the dynamic database.
[0144] In one embodiment, the dynamic data is classified according to the data format information, the content feature information, and the metadata information to establish the dynamic database for implementing:
[0145] Extracting data format keywords, content feature keywords, and metadata keywords of the dynamic data respectively, wherein the data format keywords are keywords of the data format information, the content feature keywords are keywords of the content feature information, and the metadata keywords are keywords of the metadata information;
[0146] Classifying the data format keywords, the content feature keywords, and the metadata keywords to determine at least one group of keyword types;
[0147] The dynamic data are classified according to the keyword types, and the dynamic database is established based on the classified dynamic data.
[0148] In one embodiment, multi-dimensional analysis and management are performed on all the dynamic data in the dynamic database based on the data detection results to achieve:
[0149] Analyzing the data detection results by the pre-trained adaptive data model to determine the quality assessment results, trend analysis results, correlation analysis results and / or distribution analysis results of each of the dynamic data;
[0150] Performing quantitative evaluations on the quality assessment results, the trend analysis results, the correlation analysis results, and / or the distribution analysis results based on a preset multi-dimensional analysis system to determine quantitative scores for each dimension of the dynamic data;
[0151] Perform multi-dimensional analysis and management on each of the dynamic data based on the quantitative scores of each dimension.
[0152] In one embodiment, after multi-dimensional analysis and management of all the dynamic data in the dynamic database is performed according to the data detection results, it is used to achieve:
[0153] determining an update frequency according to the amount of the dynamic data in the dynamic database;
[0154] performing data update detection on each of the dynamic data in the dynamic database according to the update frequency;
[0155] When it is detected that there is data update in the dynamic data, an update operation is performed on the dynamic data through the pre-trained adaptive data model, wherein the update operation includes a modification operation, a supplement operation and / or a deletion operation.
[0156] In one embodiment, based on an exception handling mechanism, each dynamic data in the dynamic database is dynamically detected by a pre-trained adaptive data model, and before generating a data detection result, it is used to implement:
[0157] Acquire a historical data set, determine the data type of each historical data in the historical data set using an original adaptive data model, and generate an initial data label for each historical data according to the data type;
[0158] Extracting data features and data structures of each of the historical data, and generating target data labels based on the data features and the data structures;
[0159] Comparing each of the initial data labels with each of the target data labels using a preset deep learning algorithm of the original adaptive data model to generate a historical detection result of the historical data;
[0160] The model internal parameters of the original adaptive data model are adjusted according to the historical detection results to determine the pre-trained adaptive data model.
[0161] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and the processor executes the program instructions to implement any dynamic data management method provided in the embodiment of the present application.
[0162] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.
[0163] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A dynamic data management method, characterized in that: include: Collecting various dynamic data through distributed crawler technology, and classifying all the dynamic data according to the type of each dynamic data to establish a dynamic database; Based on the exception handling mechanism, dynamically detecting each of the dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result; Perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results.
2. The dynamic data management method according to claim 1, characterized in that: The abnormality handling mechanism is based on dynamically detecting each dynamic data in the dynamic database through a pre-trained adaptive data model to generate a data detection result, including: Extracting a current data feature vector and a current data label of each of the dynamic data through the pre-trained adaptive data model; Determining a preset data tag that matches the current data tag from the dynamic database; The pre-trained adaptive data model is used to dynamically detect each of the dynamic data according to the preset data feature vector corresponding to the preset data label and the current data feature vector to generate the data detection result.
3. The dynamic data management method according to claim 1, characterized in that: The distributed crawler technology is used to collect various dynamic data, and all the dynamic data are classified according to the type of each dynamic data to establish a dynamic database, including: Collecting initial data through the distributed crawler technology, and performing data screening and data cleaning operations on the initial data to generate the dynamic data; Identifying data format information, content feature information, and metadata information of the dynamic data; The dynamic data is classified according to the data format information, the content feature information and the metadata information to establish the dynamic database.
4. The dynamic data management method according to claim 3, characterized in that: The classifying the dynamic data according to the data format information, the content feature information and the metadata information to establish the dynamic database includes: Extracting data format keywords, content feature keywords, and metadata keywords of the dynamic data respectively, wherein the data format keywords are keywords of the data format information, the content feature keywords are keywords of the content feature information, and the metadata keywords are keywords of the metadata information; Classifying the data format keywords, the content feature keywords, and the metadata keywords to determine at least one group of keyword types; The dynamic data are classified according to the keyword types, and the dynamic database is established based on the classified dynamic data.
5. The dynamic data management method according to claim 4, characterized in that: The multi-dimensional analysis and management of all the dynamic data in the dynamic database according to the data detection results includes: Analyzing the data detection results by the pre-trained adaptive data model to determine the quality assessment results, trend analysis results, correlation analysis results and / or distribution analysis results of each of the dynamic data; Performing quantitative evaluations on the quality assessment results, the trend analysis results, the correlation analysis results, and / or the distribution analysis results based on a preset multi-dimensional analysis system to determine quantitative scores for each dimension of the dynamic data; Perform multi-dimensional analysis and management on each of the dynamic data based on the quantitative scores of each dimension.
6. The dynamic data management method according to claim 1, characterized in that: After performing multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection result, the method includes: determining an update frequency according to the amount of the dynamic data in the dynamic database; performing data update detection on each of the dynamic data in the dynamic database according to the update frequency; When it is detected that there is data update in the dynamic data, an update operation is performed on the dynamic data through the pre-trained adaptive data model, wherein the update operation includes a modification operation, a supplement operation and / or a deletion operation.
7. The dynamic data management method according to any one of claims 1 to 6, characterized in that: The method of dynamically detecting each dynamic data in the dynamic database by using a pre-trained adaptive data model based on the exception handling mechanism and generating a data detection result includes: Acquire a historical data set, determine the data type of each historical data in the historical data set using an original adaptive data model, and generate an initial data label for each historical data according to the data type; Extracting data features and data structures of each of the historical data, and generating target data labels based on the data features and the data structures; Comparing each of the initial data labels with each of the target data labels using a preset deep learning algorithm of the original adaptive data model to generate a historical detection result of the historical data; The model internal parameters of the original adaptive data model are adjusted according to the historical detection results to determine the pre-trained adaptive data model.
8. A dynamic data management device, characterized in that: include: A dynamic database establishment module is used to collect various dynamic data through distributed crawler technology, and classify all the dynamic data according to the type of each dynamic data to establish a dynamic database; A data detection result generation module is used to dynamically detect each dynamic data in the dynamic database through a pre-trained adaptive data model based on an exception handling mechanism to generate a data detection result; The multi-dimensional analysis management module is used to perform multi-dimensional analysis and management on all the dynamic data in the dynamic database according to the data detection results.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the dynamic data management method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the dynamic data management method according to any one of claims 1 to 7.