Data management method and device, equipment, storage medium and program product

Abstract

The invention discloses a data management method and device, equipment, a storage medium and a program product, and the method comprises the steps: collecting current data corresponding to current environment information, and determining a data life cycle stage corresponding to the current data as a creation stage; performing data cleaning and standardization processing on the current data to obtain processed data, performing hot storage on the processed data, and updating a data life cycle stage corresponding to the current data from a creation stage to an active stage; when it is determined that the use information of the processed data meets a preset stage updating condition, updating a data life cycle stage corresponding to the current data from an active stage to an archiving stage, and performing cold storage on the processed data; and when it is determined that the processed data reaches the data service life, updating a data life cycle stage corresponding to the current data from an archiving stage to a destroying stage. By adopting the method, the data management efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data management method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of the Internet, the Internet of Things, and enterprise business systems, the generation and accumulation of data has shown an exponential growth. In the traditional data management system, data collection, storage, processing, cleaning, and analysis rely on a single technology or method. However, due to the diversity and complexity of the data itself, the traditional single processing technology can no longer effectively cope with the challenges brought by big data. For example, the traditional single processing technology is independent of each other and it is difficult to uniformly govern the data, resulting in low data management efficiency.

[0003] Therefore, how to improve the efficiency of data management has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a data management method, apparatus, device, storage medium and program product, which can improve the efficiency of data management.

[0005] In a first aspect, an embodiment of the present application provides a data management method, the method comprising:

[0006] Collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0007] Perform data cleaning and normalization on the current data to obtain the processed data, perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0008] When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold;

[0009] When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0010] In one of the embodiments, collecting current data corresponding to current environmental information includes: inputting the current environmental information into a pre-trained data collection rule prediction model to obtain a data collection rule corresponding to the current environmental information; and performing data collection based on the data collection rule to obtain current data corresponding to the current environmental information.

[0011] In one of the embodiments, the current data includes structured data, and the method further includes: obtaining unstructured data corresponding to the current environmental information, and performing data type conversion processing on the unstructured data to obtain converted data, wherein the converted data is structured data; based on the converted data and the current data, determining a data set corresponding to the current environmental information; standardizing each data in the data set to obtain a processed data set; and uploading the processed data set to a pre-deployed blockchain network.

[0012] In one of the embodiments, the method also includes: obtaining data features corresponding to the processed data set; inputting the data features into a pre-trained load trend prediction model to obtain a prediction result, and the prediction result is used to characterize the corresponding load at that moment; based on the prediction result, adjusting the priority of multiple tasks corresponding to data management, and allocating corresponding resources to each task.

[0013] In one of the embodiments, the method further includes: performing real-time data quality detection on the data in the processed data set, and when it is determined that any data in the processed data set satisfies a preset data quality abnormality condition, performing data repair processing on the abnormal data.

[0014] In one of the embodiments, the method further includes: importing the processed data into a target graph database to dynamically index the processed data; the method further includes: dynamically adjusting the index structure in the target graph database based on historical data query requests and data access frequency to optimize the data query path; the historical data query request includes a data query mode.

[0015] In a second aspect, the present application provides a data management device, the device comprising:

[0016] The collection and determination module is used to collect the current data corresponding to the current environment information and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0017] The processing module is used to clean and normalize the current data to obtain the processed data, store the processed data in hot storage, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0018] The processing module is further used to update the data life cycle stage corresponding to the current data from the active stage to the archive stage, and cold store the processed data when it is determined that the usage information of the processed data meets the preset stage update condition; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold;

[0019] The processing module is also used to update the data life cycle stage corresponding to the current data from the archiving stage to the destruction stage when it is determined that the processed data has reached the data service life.

[0020] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0022] Perform data cleaning and normalization on the current data to obtain the processed data, perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0023] When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold;

[0024] When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0025] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0027] Perform data cleaning and normalization on the current data to obtain the processed data, perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0028] When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold;

[0029] When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0030] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0031] Collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0032] Perform data cleaning and normalization on the current data to obtain the processed data, perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0033] When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold;

[0034] When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0035] The above data management method, device, equipment, storage medium and program product, the computer equipment can collect the current data corresponding to the current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage; perform data cleaning and normalization processing on the current data to obtain the processed data, and perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage; when it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archiving stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold; when it is determined that the processed data has reached the data service life, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage. Using this method, the computer equipment can automatically manage the entire process of data from creation, activation, archiving to destruction, and dynamically adjust the storage location of the data, thereby improving the efficiency of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a schematic diagram of an application scenario of a data management method provided in an embodiment of the present application;

[0038] Figure 2 It is a flowchart of a data management method provided in an embodiment of the present application;

[0039] Figure 3 It is a flowchart of another data management method provided in an embodiment of the present application;

[0040] Figure 4 It is a flowchart of another data management method provided in an embodiment of the present application;

[0041] Figure 5 It is a structural diagram of a data management device provided in an embodiment of the present application;

[0042] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] The following introduces the application scenarios of the data management method provided in the embodiments of the present application.

[0045] See also Figure 1 , Figure 1 Schematic diagram of an application scenario of a data management method provided in an embodiment of the present application. Figure 1 As shown, the computer device 100 includes multiple sensors and an application program interface (Application Program Interface, API).

[0046] Among them, the computer device 100 can collect current data corresponding to the current environmental information through multiple sensors and APIs, and then the computer device 100 can determine the data life cycle stage corresponding to the current data as the creation stage; perform data cleaning and normalization processing on the current data to obtain processed data, and perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage; when it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archiving stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold; when it is determined that the processed data has reached the data service life, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage. In this way, the computer device can automatically manage the entire process of data from creation, activation, archiving to destruction, and dynamically adjust the storage location of the data, thereby improving the efficiency of data management.

[0047] Optionally, the computer device 100 may be a terminal device or a server. The terminal device mentioned here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, a smart TV, a smart car terminal, etc. The server mentioned here may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0048] See also Figure 2 , Figure 2is a flow chart of a data management method provided in an embodiment of the present application. The method can be executed by a computer device (for example, the above-mentioned computer device 100). Figure 2 As shown, the data management method may include but is not limited to the following steps:

[0049] S201. Collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is a creation stage.

[0050] The creation phase refers to the phase after the current data is collected and before the collected current data is processed.

[0051] In an optional implementation, the computer device may collect current data corresponding to the current environmental information through a plurality of sensors and / or an API interface.

[0052] S202: clean and normalize the current data to obtain processed data, store the processed data in hot storage, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage.

[0053] The active phase refers to the phase in which the processed data obtained after the collected current data is processed is frequently queried, updated, and used.

[0054] Among them, hot storage refers to storing data on frequently accessed storage media, such as hard disks, flash memory, etc. Hot storage is usually used to store data that needs to be accessed frequently, such as online transactions, real-time data analysis, etc. The advantage of this storage method is that it has a fast access speed and is suitable for frequently accessed data, and can read and write data quickly.

[0055] In an optional embodiment, the computer device performs data cleaning and normalization processing on the current data to obtain processed data, which may include: performing abnormal data detection and correction processing on the current data to obtain first data; performing denoising processing on the first data to obtain second data; and performing normalization processing on the second data to obtain processed data.

[0056] S203: When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored.

[0057] The archiving stage refers to the stage when current data is no longer frequently used but needs to be retained to meet legal or business requirements.

[0058] Among them, cold storage refers to storing data on storage media that are not frequently accessed, such as tapes, optical disks, etc. Cold storage is usually used for long-term storage of data backup, archiving, historical records, etc., which do not need to be accessed frequently but need to be kept for a long time. The cost of cold storage is relatively low because the storage media used is relatively cheap and the storage device does not need to be updated frequently.

[0059] The preset stage update condition includes at least one of the following: the usage time of the processed data is greater than a preset time threshold, and the number of accesses to the processed data within a preset time period is less than a preset number threshold.

[0060] Optionally, the usage information of the processed data may include but is not limited to the usage time, the number of accesses to the processed data within a preset time period, etc.

[0061] For example, assuming that the usage information of the processed data includes a usage time of 100 days and the preset time threshold is 90 days, the computer device can determine that the usage time of the processed data 100 days is greater than the preset time threshold of 90 days. In this case, the computer device can determine that the usage information of the processed data meets the preset stage update conditions. At this time, the computer device can update the data life cycle corresponding to the current data from the active stage to the archive stage, and cold store the processed data.

[0062] For another example, assuming that the duration of the preset time period is 30 days, the preset number threshold is 2 times, and assuming that the usage information of the processed data includes 1 access number to the processed data within 30 days, the computer device can determine that the 1 access number to the processed data within 30 days is less than the preset number threshold of 2 times. In this case, the computer device can determine that the usage information of the processed data meets the preset stage update condition. At this time, the computer device can update the data life cycle corresponding to the current data from the active stage to the archive stage, and cold store the processed data.

[0063] For another example, assuming that the usage information of the processed data includes a usage time of 100 days, and the number of accesses to the processed data within 30 days is 1 time, assuming that the duration of the preset time period is 30 days, and the preset number threshold is 2 times, then the computer device can determine that the usage time of the processed data of 100 days is greater than the preset time threshold of 90 days, and the number of accesses to the processed data within 30 days is 1 time and less than the preset number threshold of 2 times. In this case, the computer device can determine that the usage information of the processed data meets the preset stage update conditions. At this time, the computer device can update the data life cycle corresponding to the current data from the active stage to the archive stage, and cold store the processed data.

[0064] S204: When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0065] The destruction phase refers to the phase in which the current data is deleted because there is no need to retain the current data.

[0066] For example, assuming that the data lifespan is 1 year and the usage time of the processed data is 1 year, the computer device can determine that the processed data has reached its data lifespan. In this case, the computer device can update the data life cycle stage corresponding to the current data from the archiving stage to the destruction stage.

[0067] In the embodiment of the present application, the computer device can collect the current data corresponding to the current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage; perform data cleaning and normalization processing on the current data to obtain the processed data, and perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage; when it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archiving stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold; when it is determined that the processed data reaches the data service life, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage. Using this method, the computer device can automatically manage the entire process of data from creation, activation, archiving to destruction, and dynamically adjust the storage location of the data, thereby improving the efficiency of data management.

[0068] See also Figure 3 , Figure 3 is a flow chart of another data management method provided in an embodiment of the present application, and Figure 2 The data management approach shown differs in that Figure 3 The data management method shown in FIG. 1 specifically describes how the computer device collects the current data corresponding to the current environment information. Figure 3 As shown, the data management method may include but is not limited to the following steps:

[0069] S301, inputting current environment information into a pre-trained data collection rule prediction model to obtain data collection rules corresponding to the current environment information.

[0070] The data collection rules may include but are not limited to data collection frequency, data collection scope, and data collection mode.

[0071] In an optional implementation, the data collection rule prediction model can be obtained by training a computer device through the following steps:

[0072] Step 1: Obtain historical data sets corresponding to multiple historical environmental information collected through sensors and / or APIs, as well as actual data collection rules corresponding to each historical data set.

[0073] Step 2: Perform abnormal data detection and correction, denoising and formatting on the data in each historical data set to obtain multiple processed historical data sets.

[0074] Step 3: Based on business requirements and data characteristics, feature extraction is performed on the data in each processed historical data set to obtain data features corresponding to the processed historical data set.

[0075] Optionally, the data features corresponding to the processed historical data set may include but are not limited to timestamp, geographic location information, user activity patterns, etc.

[0076] Step 4: Convert the data features corresponding to each processed historical data set into machine-recognizable features to obtain the converted data features.

[0077] Optionally, the computer device may use the method of encoding categorical variables to convert the data features corresponding to each processed historical data set into machine-recognizable features to obtain the converted data features; or it may use the method of normalizing data features to convert the data features corresponding to each processed historical data set into machine-recognizable features to obtain the converted data features, which is not limited here.

[0078] Step 5: Input the converted data features corresponding to each processed historical data set and the historical environment information corresponding to each processed historical data set into the initialized data collection rule prediction model to obtain the prediction data collection rules corresponding to each historical data set.

[0079] Optionally, the initialized data collection rule prediction model may be selected by the computer device based on the problem type, wherein the problem type may include but is not limited to regression problems, classification problems, etc. The initialized data collection rule prediction model may include but is not limited to decision trees, random forests, support vector machines, neural networks, etc.

[0080] Step six: adjust the model parameters in the initialized data collection rule prediction model in the direction of reducing the difference between the predicted data collection rule corresponding to each historical data set and the actual data collection rule corresponding to the historical data set until the training stop conditions are met to obtain a trained data collection rule prediction model.

[0081] Optionally, after step six, the computer device may also continuously adjust the model parameters through cross-validation, network search and other technologies to obtain a data collection rule prediction model with higher prediction accuracy.

[0082] S302: Perform data collection based on data collection rules to obtain current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is a creation stage.

[0083] In an optional implementation, the computer device collects data based on the data collection rules to obtain the current data corresponding to the current environment information, which may include: dynamically adjusting the data collection rules based on business needs to obtain the adjusted data collection rules; and collecting data based on the adjusted data collection rules to obtain the current data corresponding to the current environment information. In this way, the computer device can obtain data collection rules that better meet the business needs by adjusting the data collection rules based on business needs, so that the data collected based on the adjusted data collection rules can make the current data corresponding to the current environment information better meet the current business needs.

[0084] In an optional implementation, after step S302, the computer device may also use the current data corresponding to the current environmental information as training data to train the pre-trained data collection rule prediction model again to obtain a retrained data collection rule prediction model. In this way, as external conditions change (such as seasonal fluctuations, emergencies), the computer device can automatically re-evaluate and update the data collection rule prediction model in a timely manner without manual intervention, thereby improving the adaptability and update efficiency of the data collection rule model.

[0085] S303: clean and normalize the current data to obtain processed data, store the processed data in hot storage, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage.

[0086] S304: clean and normalize the current data to obtain processed data, store the processed data in hot storage, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage.

[0087] The preset stage update condition includes at least one of the following: the usage time of the processed data is greater than a preset time threshold, and the number of accesses to the processed data within a preset time period is less than a preset number threshold.

[0088] S305: When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

[0089] In an optional implementation, the relevant descriptions of steps S303 to S305 can refer to the descriptions of the aforementioned steps S202 to S204, and will not be repeated here.

[0090] In an optional implementation, the computer device may also use an intelligent prediction model to predict the data life cycle stage of the current data based on the following two key factors:

[0091] (1) By monitoring and analyzing data access patterns, we can predict which data will become less frequently used and plan their archiving or migration in advance.

[0092] (2) Consider the cost differences of different storage media and combine them with expected usage trends to determine the optimal storage solution to minimize total cost.

[0093] In an embodiment of the present application, a computer device may input current environmental information into a pre-trained data collection rule prediction model to obtain a data collection rule corresponding to the current environmental information; perform data collection based on the data collection rule to obtain current data corresponding to the current environmental information, and determine that the data life cycle stage corresponding to the current data is the creation stage; perform data cleaning and normalization processing on the current data to obtain processed data, and perform hot storage on the processed data, and update the data life cycle stage corresponding to the current data from the creation stage to the active stage; when it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archiving stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than the preset time threshold and the number of accesses to the processed data within the preset time period being less than the preset number threshold; when it is determined that the processed data has reached the data service life, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage. In this way, on the one hand, the computer device obtains the data collection rules corresponding to the current environmental information by inputting the current environmental information into the pre-trained data collection rule prediction model, and collects data based on the data collection rules, thereby improving the accuracy of the collected current data; on the other hand, the computer device can automatically manage the entire process of data from creation, activation, archiving to destruction, and dynamically adjust the storage location of the data, thereby improving the efficiency of data management.

[0094] In an optional embodiment, Figure 2 and Figure 3In the data management method shown, the current data includes structured data, and the computer device can also obtain unstructured data corresponding to the current environmental information, and perform data type conversion processing on the unstructured data to obtain converted data, and the converted data is structured data; based on the converted data and the current data, determine the data set corresponding to the current environmental information; standardize each data in the data set to obtain a processed data set; upload the processed data set to a pre-deployed blockchain network.

[0095] The unstructured data include, for example, documents and text files obtained by scanning by computer equipment.

[0096] Optionally, the computer device performs data type conversion processing on the unstructured data to obtain converted data, which may include: using integrated optical character recognition (OCR) to extract text information from scanned documents; using natural language processing (NLP) technology to perform semantic analysis, entity recognition and sentiment analysis on the extracted text information, converting the unstructured text information into structured text information, and using the structured text information as the converted data.

[0097] Alternatively, the blockchain network can be built by computer devices based on blockchain platforms such as Hyperledger Fabric, Corda or Quorum. These platforms provide functions such as privacy protection and permission control, and are very suitable for building cross-organizational data sharing platforms.

[0098] Among them, Hyperledger Fabric is a modular distributed ledger solution support platform that provides high confidentiality, elasticity, flexibility and scalability. Its purpose is to support the pluggable implementation of different components and adapt to the complexity existing in the economic system. Hyperledger Fabric proposes a unique highly elastic and scalable architecture, which distinguishes Fabric from other blockchain solutions. In Fabric's future planning for enterprise-level blockchains, its architecture allows for comprehensive review and open source.

[0099] Corda is a distributed ledger platform designed specifically for the financial sector. It is a dedicated network designed to record, manage and synchronize contracts or other shared data between partners, as well as an open source platform that can be used to build applications for financial institutions on top of the financial foundation.

[0100] Quorum is a distributed ledger protocol platform based on Ethereum, developed to provide transaction support and contract privacy for industries such as finance, supply chain, retail, and real estate.

[0101] Optionally, users can deploy blockchain nodes within each participating organization to ensure that all participants have equal access to the network.

[0102] Optionally, the computer device can also write smart contracts to clarify the terms and conditions of data sharing, including but not limited to data ownership, usage rights, access rights, etc. Among them, the smart contract must clearly specify which data can be accessed by whom and how to use the data. In this way, using smart contracts to automatically execute preset business logic, such as triggering data transmission or updating records when specific conditions are met, can not only improve data management efficiency, but also reduce the risks caused by human intervention. In addition, by directly connecting the supply and demand sides with smart contracts, the intermediary role in the traditional model can be eliminated, reducing costs and speeding up response.

[0103] Optionally, the computer device uploads the processed data set to the pre-deployed blockchain network, which may include: encrypting the processed data set to obtain an encrypted data set; uploading the encrypted data set to the pre-deployed blockchain network. In this way, by encrypting the data uploaded to the blockchain network, data leakage during network transmission can be avoided, thereby improving data security.

[0104] Optionally, the computer device can also build a decentralized data sharing platform based on the data set corresponding to the current environmental information, and use the blockchain network as the basis for decentralization, thereby utilizing blockchain technology and zero-trust architecture to provide an additional layer of security.

[0105] By adopting this implementation mode, the data source can be enriched by combining the unstructured data corresponding to the current environmental information with the current data (structured data), which is conducive to improving the comprehensiveness of the data corresponding to the current environmental information. In addition, the computer device performs standardized processing on each data in the data set, which can help improve the consistency and interoperability of the data, making it easier for different organizations to share and understand the data. After that, the computer device uploads the processed data set to the pre-deployed blockchain network. In this way, the encryption mechanism and zero-trust architecture of the blockchain technology can be relied on to strictly control the access and operation of the data. Through fine-grained permission management, it is ensured that authorized users can access sensitive information and record logs of all access operations.

[0106] In addition, since every transaction on the blockchain is open, transparent and unchangeable, it can provide strong protection for the authenticity and integrity of the data. And every data exchange will be recorded on the blockchain, forming a complete audit trail. In this way, by providing an open and transparent data exchange environment, it can enhance the trust between organizations and promote a wider range of cooperative relationships.

[0107] In this embodiment, the computer device can also obtain data features corresponding to the processed data set; input the data features into a pre-trained load trend prediction model to obtain prediction results, and the prediction results are used to characterize the corresponding load at that moment; based on the prediction results, the priorities of multiple tasks corresponding to data management are adjusted, and corresponding resources are allocated to each task.

[0108] Optionally, the load trend prediction model may be trained by a computer device in the following manner:

[0109] Step 1: Obtain training data and the actual load corresponding to the training data.

[0110] Optionally, the computer device may obtain data from different sources through at least one sensor and / or an API interface, wherein the data from different sources may be data in a database such as InfluxDB or TimescaleDB that is specifically designed for efficient storage and query of time series data.

[0111] Among them, InfluxDB is a custom, open source, NoSQL time series database written in Go.

[0112] TimescaleDB is an open source time series database optimized for fast ingestion and complex queries with full SQL support. It is based on PostgreSQL and brings both the NoSQL and relational worlds to time series data.

[0113] Step 2: Extract training data features corresponding to the training data.

[0114] Optionally, the data feature is, for example, an average value, a peak value, a frequency component, etc.

[0115] Optionally, the computer device may extract data features corresponding to multiple training data through a long short-term memory network (LSTM), a gated recurrent unit (GRU), or an autoregressive integrated moving average model (ARIMA).

[0116] Step 3: Input the training data features into the pre-built load trend prediction model to obtain the predicted load.

[0117] Step 4: Train the pre-built load trend prediction model in a direction of reducing the difference between the actual load corresponding to the training data and the predicted load until the training stop condition is met, thereby obtaining a trained load trend prediction model.

[0118] Optionally, during the adaptive task scheduling process, the computer equipment can also combine container orchestration technology (such as Kubernetes) to achieve elastic expansion and dynamic adjustment of tasks. In this way, the stability of system performance can be guaranteed whether during system peak hours or when the load increases suddenly.

[0119] Exemplarily, a computer device can split an application into multiple independent service components and encapsulate them into Docker images for easy management and deployment. Build a Kubernetes environment to coordinate functions such as container lifecycle, resource scheduling, and service discovery. Automatically increase or decrease the number of working nodes according to real-time load conditions to ensure that the system always runs in the optimal state. In the case of insufficient resources for a single instance, it can be upgraded by adjusting resource configurations such as the central processing unit (CPU) and memory. Use tools such as CI / CD pipelines and Helm charts to simplify the deployment process, improve efficiency, and reduce the risk of human error. Among them, the CI / CD pipeline is a deployment pipeline integrated with automation tools and improved workflows. The Helm chart is a collection of YAML template files organized into a specific directory structure.

[0120] By adopting this implementation mode, the computer device adjusts the priorities of multiple tasks corresponding to data management based on the prediction results and allocates corresponding resources to each task, thereby ensuring that during the peak period of data collection and processing, task scheduling can be adaptively optimized to avoid system overload.

[0121] In an optional embodiment, Figure 2 and Figure 3 In the data management method shown, the computer device can also perform real-time data quality detection on the data in the processed data set, and when it is determined that there is any data in the processed data set that meets the preset data quality abnormality condition, perform data repair processing on the abnormal data.

[0122] Optionally, the computer device may use an autoencoder or a variational autoencoder (VAE) to perform real-time data quality detection on the data in the processed data set. Among them, the autoencoder and VAE are good at capturing the data distribution under the normal mode, thereby identifying data points that deviate from the normal range as outliers.

[0123] Optionally, for data points detected as abnormal, the computer device may choose to delete the abnormal data points, replace them with adjacent values, or fill them using interpolation. Optionally, for data items with vacancies, an appropriate filling strategy may be used, such as mean filling, nearest neighbor filling, or more complex statistical methods.

[0124] Optionally, the computer equipment can also save a record of all changes to facilitate auditing and subsequent analysis.

[0125] Optionally, the computer device can also convert the numerical data to the same scale to ensure comparability between different dimensions.

[0126] By adopting this implementation mode, the computer device performs data repair processing on the abnormal data, which can ensure the consistency (such as all fields have the correct data type (integer, floating point number, string, etc.), and the date and time stamps are in a unified format) and integrity of the data, and reduce manual intervention.

[0127] In an optional embodiment, Figure 2 and Figure 3 In the data management method shown, the computer device can also import the processed data into the target graph database to perform dynamic index management on the processed data; the method also includes: based on historical data query requests and data access frequency, dynamically adjusting the index structure in the target graph database to optimize the data query path; the historical data query request includes a data query mode.

[0128] Optionally, the target graph database is a graph database that supports efficient relational queries, such as Neo4j, ArangoDB, etc.

[0129] Among them, Neo4j is a popular graph database with high performance and ease of use. It has an active community, supports multiple languages, and provides a visual interface to help users easily manage and query data. ArangoDB is a multi-model database that supports graph, document, and collection type data models. It provides an easy-to-use query language and a flexible data model.

[0130] By adopting this implementation mode, the computer device can ensure data integrity and consistency by importing the normalized data, that is, the processed data, into the target graph database.

[0131] In an optional implementation, the various models mentioned above (such as the data collection rule prediction model, the load trend prediction model, etc.) can be trained locally by a computer device.

[0132] Optionally, computer devices can use federated learning technology to protect data privacy under the premise of multi-party collaboration, by training models locally and exchanging only model parameters instead of data, ensuring that all parties can collaboratively analyze data without sharing original data. Federated learning technology is implemented on the basis of the decentralized data sharing platform and fine-grained permission control mentioned above.

[0133] The federated learning process may include but is not limited to the following:

[0134] Each participant trains a local model using a local dataset in its own environment. This means that the original data always remains within its respective security boundaries and will not be transmitted to any other party, fundamentally eliminating the risk of data leakage.

[0135] The parties only exchange new model parameters after aggregation, rather than actual data samples. These parameters are usually not enough to reconstruct the original data, so even if they are intercepted during network transmission, attackers cannot obtain useful information.

[0136] For more stringent privacy requirements, random noise can be added when updating parameters, that is, differential privacy technology can be applied to make the impact of a single participant on the overall model negligible, further strengthening the anonymity protection of individual contributions.

[0137] See also Figure 4 , Figure 4 This is a schematic diagram of the overall process of a data management method provided by an embodiment of the present application. Figure 4 As shown, the data management method may include but is not limited to the following steps:

[0138] S401. Analyze historical data and current environmental information using a machine learning algorithm, automatically generate data collection rules, and perform data collection based on the data collection rules to obtain current data corresponding to the current environmental information.

[0139] Optionally, the computer device performs data collection based on data collection rules to obtain current data corresponding to the current environmental information, which may include: dynamically adjusting the generated data collection rules according to business needs to obtain adjusted data collection rules; and performing data collection based on the adjusted data collection rules to obtain current data corresponding to the current environmental information.

[0140] S402, obtaining unstructured data corresponding to the current environment information, and parsing the unstructured data through OCR and NLP technology to obtain parsed data, and combining the parsed data with the current data to obtain a data set corresponding to the current environment information.

[0141] S403. Using blockchain technology, a decentralized data sharing platform is constructed based on the data set corresponding to the current environmental information.

[0142] S404: Based on the data set corresponding to the current environmental information, AI technology is used to predict the changing trend of the system load, and then the priority of task execution and resource allocation are adjusted.

[0143] S405: Use a deep learning algorithm to detect and correct outliers in the data set corresponding to the current environmental information to obtain a processed data set.

[0144] S406. Based on the processed data set, deploy intelligent agents to continuously monitor the data flow.

[0145] S407. Based on the processed data set, a graph database is used to perform dynamic index management.

[0146] S408. In S403 and S406, strict permission control is implemented on data access and operations by relying on the encryption mechanism and zero-trust architecture of blockchain technology.

[0147] S409. Based on the adaptive task scheduling in S404 and combined with container orchestration technology, elastic expansion and dynamic adjustment of tasks are achieved.

[0148] S410: Combine the data collection, cleaning and task scheduling in S401, S405 and S409, and use AI prediction model to manage the data life cycle.

[0149] S411. Under the security guarantee mechanism of S403 and S408, federated learning technology is used to protect data privacy under the premise of multi-party collaboration. By training the model locally and exchanging only model parameters instead of data, it ensures that all parties can collaboratively analyze data without sharing the original data.

[0150] The following is an example of the data management method provided in the embodiment of the present application.

[0151] A fast-growing logistics company wants to improve its parcel delivery service, reduce delivery delays, optimize route planning, and provide more accurate estimated delivery times. To this end, the company can deploy an intelligent logistics distribution system that needs to process and analyze data sources from multiple channels, including but not limited to: GPS positioning information on logistics vehicles, inventory status in warehouse management systems, customer order details (such as delivery addresses, special requirements), historical delivery records, and real-time traffic conditions provided by third-party map service providers.

[0152] Specifically, the data management process may include the following steps:

[0153] Step 1: Intelligent data collection.

[0154] The system automatically obtains location information from the GPS device of the logistics vehicle, combines it with historical delivery routes, and uses machine learning algorithms to predict the optimal data collection frequency to balance accuracy and energy consumption.

[0155] Step 2: Enhanced file collection and multi-dimensional data support.

[0156] For unstructured data (such as photos or videos uploaded by customers), OCR technology and image recognition technology are used to parse, extract useful information and convert it into structured data.

[0157] Step 3: Safe and efficient data exchange.

[0158] Build an internal dedicated data exchange platform to allow different departments (such as the transportation department and customer service department) to securely share required information. All transmissions are encrypted to ensure the security of data during transmission.

[0159] Step 4: Adaptive task scheduling and resource optimization.

[0160] Analyze current and predict future distribution needs, dynamically adjust vehicle scheduling and personnel arrangements, and ensure sufficient capacity during peak hours.

[0161] Step 5: Deep data cleaning and normalization.

[0162] Clean data from various sources, eliminate duplicates, correct erroneous values, standardize address formats, etc. to provide a reliable foundation for data analysis.

[0163] Step 6: Intelligent data quality monitoring and repair.

[0164] Implement a continuous data quality monitoring mechanism and take immediate measures to correct problems and ensure service quality once abnormal situations are detected (such as an increase in delivery failure rate).

[0165] Step 7: Dynamic indexing and query optimization.

[0166] Optimize database indexes based on user query habits to speed up retrieval of specific information (such as all packages to be delivered in a certain area).

[0167] Step 8: Fine-grained security management and permission control.

[0168] Set detailed access permissions for different roles to ensure that only authorized users can view sensitive information, such as customer contact details.

[0169] Step 9: Flexible task scheduling and queue management.

[0170] During promotional activities or other high-load situations, the system can quickly expand computing resources to ensure stable operation without affecting normal business operations.

[0171] Step 10: Intelligent data lifecycle management.

[0172] Automatically evaluate the value of data and migrate no longer active historical order data to lower-cost storage media or delete it according to regulatory requirements.

[0173] Step 11: Federated learning and privacy protection.

[0174] Through the federated learning framework, warehouses in different locations can jointly train models without sharing original data, improve delivery strategies, and protect customer privacy.

[0175] Through the above solutions, logistics companies can not only improve their operational efficiency and service levels, but also fully utilize the advantages brought by big data while complying with strict data protection regulations.

[0176] In the embodiments of the present application, by integrating AI technology and automation mechanisms, data lifecycle management can be performed intelligently and efficiently, which not only meets business needs and reduces operating costs, but also ensures the security and reliability of data.

[0177] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0178] Based on the same inventive concept, the embodiment of the present application also provides a data management device for implementing the data management method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data management device embodiments provided below can refer to the limitations on the data management method above, and will not be repeated here.

[0179] See also Figure 5 , Figure 5 Schematic diagram of a data management device provided in an embodiment of the present application. Figure 5 As shown, the data management device may include but is not limited to:

[0180] The collection and determination module 501 is used to collect current data corresponding to the current environment information, and determine that the data life cycle stage corresponding to the current data is the creation stage;

[0181] The processing module 502 is used to perform data cleaning and normalization processing on the current data to obtain the processed data, and to perform hot storage on the processed data, and to update the data life cycle stage corresponding to the current data from the creation stage to the active stage;

[0182] The processing module 502 is further configured to update the data life cycle stage corresponding to the current data from the active stage to the archive stage, and cold store the processed data if it is determined that the usage information of the processed data meets the preset stage update condition; the preset stage update condition includes at least one of the usage time of the processed data being greater than a preset time threshold and the number of accesses to the processed data within a preset time period being less than a preset number threshold;

[0183] The processing module 502 is further configured to update the data life cycle stage corresponding to the current data from the archiving stage to the destruction stage when it is determined that the processed data has reached the data service life stage.

[0184] In one embodiment, when the collection and determination module 501 is used to collect current data corresponding to the current environmental information, it is specifically used to: input the current environmental information into a pre-trained data collection rule prediction model to obtain the data collection rules corresponding to the current environmental information; perform data collection based on the data collection rules to obtain the current data corresponding to the current environmental information.

[0185] In one embodiment, the device further includes a determination module. The current data includes structured data, and the processing module 502 is further used to: obtain unstructured data corresponding to the current environment information, and perform data type conversion processing on the unstructured data to obtain converted data, and the converted data is structured data; the determination module is used to determine the data set corresponding to the current environment information based on the converted data and the current data; the processing module is also used to perform standardization processing on each data in the data set to obtain a processed data set; and upload the processed data set to a pre-deployed blockchain network.

[0186] In one embodiment, the device may further include an acquisition module. The acquisition module is used to acquire data features corresponding to the processed data set; the processing module 502 is also used to: input the data features into a pre-trained load trend prediction model to obtain a prediction result, which is used to characterize the load corresponding to the time; based on the prediction result, adjust the priorities of multiple tasks corresponding to data management, and allocate corresponding resources to each task.

[0187] In one embodiment, the processing module 502 is also used to: perform real-time data quality detection on the data in the processed data set, and when it is determined that any data in the processed data set meets the preset data quality abnormality condition, perform data repair processing on the abnormal data.

[0188] In one embodiment, the processing module 502 is also used to import the processed data into the target graph database to perform dynamic index management on the processed data; the processing module is also used to: dynamically adjust the index structure in the target graph database based on historical data query requests and data access frequency to optimize the data query path; the historical data query request includes a data query mode.

[0189] Each module in the above data management device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of the processor in the terminal device in the form of hardware, or can be stored in the memory in the terminal device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0190] In an exemplary embodiment, the present application provides a computer device, which may be a terminal device, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a data management method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0191] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0192] In an exemplary embodiment, the present application provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned data management method when executing the computer program.

[0193] In an exemplary embodiment, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned data management method are implemented.

[0194] In an exemplary embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above data management method when executed by a processor.

[0195] It should be noted that the data involved in this application (including but not limited to current data, processed data, usage information of processed data, unstructured data, data sets corresponding to current environmental information, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0196] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0197] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data management method, characterized in that: The method comprises: Collect current data corresponding to the current environment information, and determine that the data life cycle stage corresponding to the current data is a creation stage; Performing data cleaning and normalization processing on the current data to obtain processed data, hot storing the processed data, and updating the data life cycle stage corresponding to the current data from the creation stage to the active stage; When it is determined that the usage information of the processed data meets the preset stage update condition, the data life cycle stage corresponding to the current data is updated from the active stage to the archive stage, and the processed data is cold stored; the preset stage update condition includes at least one of the usage time of the processed data being greater than a preset time threshold and the number of accesses to the processed data within a preset time period being less than a preset number threshold; When it is determined that the processed data has reached the end of its data life cycle, the data life cycle stage corresponding to the current data is updated from the archiving stage to the destruction stage.

2. The method according to claim 1, characterized in that: The collecting of current data corresponding to the current environment information includes: Inputting the current environment information into a pre-trained data collection rule prediction model to obtain the data collection rule corresponding to the current environment information; Data collection is performed based on the data collection rule to obtain current data corresponding to the current environmental information.

3. The method according to claim 1, characterized in that The current data includes structured data, and the method further includes: Acquire unstructured data corresponding to the current environment information, and perform data type conversion processing on the unstructured data to obtain converted data, where the converted data is structured data; Determining a data set corresponding to the current environment information based on the converted data and the current data; Performing standardization processing on each data in the data set to obtain a processed data set; The processed data set is uploaded to a pre-deployed blockchain network.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining data features corresponding to the processed data set; Inputting the data features into a pre-trained load trend prediction model to obtain a prediction result, wherein the prediction result is used to characterize the load corresponding to the time; Based on the prediction results, the priorities of multiple tasks corresponding to the data management are adjusted, and corresponding resources are allocated to each of the tasks.

5. The method according to claim 1, characterized in that The method further comprises: Real-time data quality detection is performed on the data in the processed data set, and when it is determined that any data in the processed data set meets a preset data quality abnormality condition, data repair processing is performed on the abnormal data.

6. The method according to claim 1, characterized in that The method further comprises: Importing the processed data into a target graph database to perform dynamic index management on the processed data; The method further comprises: Based on historical data query requests and data access frequency, the index structure in the target graph database is dynamically adjusted to optimize the data query path; the historical data query request includes a data query mode.

7. A data management device, characterized in that: The device comprises: A collection and determination module, used to collect current data corresponding to current environment information, and determine that the data life cycle stage corresponding to the current data is a creation stage; A processing module, used for performing data cleaning and normalization processing on the current data to obtain processed data, hot-storing the processed data, and updating the data life cycle stage corresponding to the current data from the creation stage to the active stage; The processing module is further configured to update the data lifecycle stage corresponding to the current data from the active stage to the archive stage, and cold store the processed data when it is determined that the usage information of the processed data meets a preset stage update condition; the preset stage update condition includes at least one of the usage time of the processed data being greater than a preset time threshold and the number of accesses to the processed data within a preset time period being less than a preset number threshold; The processing module is further configured to update the data life cycle stage corresponding to the current data from the archiving stage to the destruction stage when it is determined that the processed data has reached the end of its data life cycle.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images