Method and system for updating and managing data in non-extranet environment based on local deployment

Abstract

The invention discloses a method and a system for updating and managing data in a non-extranet environment based on local deployment, and relates to the technical field of local deployment data updating and management, and the method comprises the following steps: collecting to-be-updated data from a plurality of data sources, and storing the to-be-updated data in a temporary buffer area; reading the data in the buffer area for preprocessing and quality detection, and generating a data quality report; according to the data quality report, screening out to-be-updated high-quality data by using an artificial intelligence algorithm, and performing analysis to obtain an update suggestion report; formulating a data updating strategy and generating a data updating execution schedule according to the updating suggestion report; executing a data updating operation according to the data updating execution schedule, and recording updating information to the block chain at the same time; and performing deep verification on the updated data, storing the data to a local database after the verification is passed, and recording a result to the block chain to complete the whole data updating and management process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of local deployment data update and management, and in particular to a method and system for data update and management in a non-external network environment based on local deployment. BACKGROUND

[0002] With the rapid development of information technology, data update and management in a non-external network environment based on local deployment has gradually become the focus of attention in various industries. In the fields of finance, energy, manufacturing, etc., enterprises have increasingly high requirements for data security, accuracy and timeliness. Traditional data update and management techniques continue to evolve in these areas, but still have many limitations.

[0003] In the prior art, the integrity, consistency and security of data are difficult to be fully guaranteed during data update. Especially in the data screening and update strategy formulation link, there is a lack of intelligent and automated effective means, often relying on manual experience, which is prone to cause problems such as untimely update and uneven data quality, and is difficult to meet the needs of enterprises for efficient data management and accurate update. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a method for data update and management in a non-external network environment based on local deployment, which solves the problems of untimely data update, uneven data quality, and how to efficiently guarantee data security and reliability in the existing data update and management method in a non-external network environment based on local deployment.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for data update and management in a non-external network environment based on local deployment, characterized in that it comprises the following steps:

[0008] In a local non-external network environment, collect data to be updated from multiple data sources and store them in a temporary buffer;

[0009] Read the data in the buffer for preprocessing and quality detection, and generate a data quality report;

[0010] According to the data quality report, use an artificial intelligence algorithm to screen high-quality data to be updated, and analyze to obtain an update recommendation report;

[0011] According to the update recommendation report, formulate a data update strategy and generate a data update execution schedule;

[0012] Perform data update operations according to the data update execution schedule, and record the update information to a blockchain;

[0013] The updated data is deeply checked, and after the check is passed, the data is stored in the local database, and the result is recorded to the blockchain, and the whole data updating and management process is completed.

[0014] As a preferred scheme of the method for data updating and management based on local deployment in a non-external network environment according to the application, wherein: in the local non-external network environment, the local database, the file system, the sensor and the internal business system are explicitly defined as data sources of the data to be updated.

[0015] For different data sources, corresponding connection methods are adopted, such as connecting the local database using JDBC, accessing the file system through the file system API, connecting the sensor using the sensor communication protocol, and connecting the internal business system through the interface or data export function provided by the internal business system.

[0016] According to the preset data extraction rule, the data to be updated is extracted from each data source. For the local database, the newly added or modified data records are extracted according to the update timestamp of the data table; for the file system, the files that need to be updated are filtered out according to the preset file name mode and timestamp; for the sensor, the environmental parameter data collected by the sensor is read at a fixed time interval; for the internal business system, the data export interface provided by the internal business system is called or the business data is obtained through the interface.

[0017] The data extracted from different data sources is converted into a unified intermediate format for subsequent processing. For example, the date formats of different databases are uniformly converted into the format of "YYYY-MM-DDHH:MM:SS", and the text encodings in different file systems are uniformly converted into UTF-8 encoding.

[0018] The converted data is merged according to the preset data integration rule to generate an integrated data set containing all the data to be updated. The integration rule can include operations such as data deduplication, association and sorting to ensure that the integrated data is complete and consistent.

[0019] The integrated data set is stored in a temporary buffer, and the buffer is implemented by using an in-memory database or a cache system, which has the ability of fast reading and writing and data temporary storage, provides data input for subsequent preprocessing and quality detection steps, and ensures the timeliness and availability of data.

[0020] As a preferred scheme of the method for data updating and management based on local deployment in a non-external network environment according to the application, wherein: the data in the reading buffer is preprocessed and quality detected to generate a data quality report, and the specific steps are,

[0021] The integrated data set in the temporary buffer is read into the pre-processing module memory space in its entirety to provide a data basis for subsequent operations.

[0022] The data is preliminarily cleaned by using unique identifiers for deduplication, correcting data format issues, and filling in missing values.

[0023] The consistency of key fields in the data table is checked according to preset encoding rules and field consistency requirements, inconsistent data is marked and temporarily stored, and an intermediate report is generated.

[0024] According to the preset data quality evaluation index system, the comprehensive score of data quality is calculated, and the data that does not meet the quality requirements is stored in the abnormal data temporary storage area.

[0025] According to the evaluation results, a structured data quality report is generated, covering the quality status and comprehensive score of each data source, providing a basis for subsequent data screening.

[0026] As a preferred scheme of the method for data updating and management in a non-external network environment based on local deployment, the method comprises the following steps:

[0027] The data quality report is structurally analyzed, and the quality index values and problem data identifiers of each data source are extracted as key information, which are converted into a feature vector matrix as input data for the artificial intelligence algorithm.

[0028] The feature vector matrix is input into a pre-trained random forest-based screening model, which automatically identifies and screens high-quality data to be updated that meet the quality requirements by learning a large amount of historical data quality conditions and business requirements, and outputs a screening result containing high-quality data identifiers and priority.

[0029] Trend analysis is performed on the screened high-quality data, and time series analysis method is used to mine the trend and rule of data updating, such as analyzing the growth trend, fluctuation period, and correlation with business indicators, to provide data support for the updating strategy.

[0030] According to the screening result and analysis trend, a data updating suggestion report is generated. The report clearly indicates which data needs to be updated, the priority order of updating, and the expected updating effect, and proposes solutions and preventive measures for different data quality problems, providing detailed guidance schemes for subsequent data updating operations.

[0031] As a preferred scheme of the method for data updating and management in a non-external network environment based on local deployment, the method comprises the following steps:

[0032] Perform structured analysis on the update recommendation report to extract key information such as data update priority, expected effects, and quality issue resolution suggestions, and convert them into policy formulation parameters.

[0033] According to the policy formulation parameters, combined with system resources and business needs, the data update strategy is formulated using the optimization model based on genetic algorithm, the data update method, algorithm and tools are determined, and the data update time window is planned to generate a data update execution plan that includes update tasks, time and responsible persons.

[0034] As a preferred solution of the method for data update and management in a non-external network environment based on local deployment according to the present invention, wherein: the data update operation is performed according to the data update execution schedule, and the update information is recorded in the blockchain at the same time, the specific steps are:

[0035] Extract the update task list from the data update execution plan table and convert it into an executable update script set. The script type is determined by the target data platform, such as SQL scripts for relational databases and NoSQL scripts for non-relational databases.

[0036] According to the scheduled time sequence of the update tasks, the update scripts are called in sequence to perform data update operations. During the update process, the update progress and system resource usage are monitored in real time to ensure the stable operation of the update operation.

[0037] For each update operation, an update record block is generated containing information such as update time, content, operator, and operation steps. The block data is encrypted using the blockchain's hash algorithm to ensure that the record cannot be tampered with.

[0038] The encrypted update record block is linked to the existing blockchain to form an unalterable update record chain, providing reliable security and audit basis for the data update process, while also providing support for subsequent data query and tracing.

[0039] As a preferred solution of the method for data update and management in a non-external network environment based on local deployment according to the present invention, wherein: the updated data is deeply verified, stored in the local database after passing the verification, and the results are recorded in the blockchain to complete the entire data update and management process. The specific steps are:

[0040] Perform in-depth verification on the updated data, use a hash verification algorithm to calculate the hash value of the data, and compare it with the hash value of the original data before the update to ensure the accuracy and completeness of the data update.

[0041] If the data verification passes, a suitable storage strategy is selected according to the data type: for structured data, a relational database is used for storage; for unstructured data, object storage service is used for storage. At the same time, indexes are created for the stored data to facilitate subsequent efficient query and access.

[0042] The depth verification result and data storage location information are recorded to the blockchain, a record block containing verification result, data storage location, verification time and other information is generated, the block data is encrypted by using the hash algorithm of the blockchain, and the non-tamperability of the record is ensured.

[0043] The encrypted record block is linked to the existing blockchain to form a tamper-proof data update and management record chain, which provides reliable security and audit basis for the data update and management process, and supports subsequent data query, traceability and management, and completes the entire data update and management process.

[0044] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the method for data update and management in a non-external network environment based on local deployment according to the first aspect of the present application.

[0045] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the method for data update and management in a non-external network environment based on local deployment according to the first aspect of the present application.

[0046] The present application has the following advantages: by collecting the data to be updated from multiple data sources and storing them in a temporary buffer, the timeliness and availability of the data are ensured; by preprocessing and quality detection to generate a data quality report, the accuracy and integrity of the data are effectively improved; by using artificial intelligence algorithms to filter high-quality data and generate an update recommendation report, the intelligentization and automation of data update are realized; based on the update recommendation report, a data update strategy is formulated and an execution schedule is generated, improving the scientificity and rationality of data update; when performing data update operation, the update information is recorded to the blockchain, ensuring the security and traceability of the data update process; finally, the updated data is subjected to depth verification and stored in the local database, and the result is recorded to the blockchain, further ensuring the reliability and security of the data. The method effectively solves the problems of data update not being timely, data quality being uneven, and data security being unable to be guaranteed in the prior art, and realizes the efficiency, accuracy and reliability of data update and management. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0048] Figure 1 The flowchart of the method for data updating and management in a non-external network environment based on local deployment in Embodiment 1. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present application more apparent and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0050] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0051] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0052] Embodiment 1, Reference Figure 1 For the first embodiment of the present application, the embodiment provides a method for data updating and management in a non-external network environment based on local deployment, comprising the following steps:

[0053] In the local non-external network environment, collect the data to be updated from multiple data sources and store them in a temporary buffer;

[0054] Read the data in the buffer for preprocessing and quality detection, and generate a data quality report;

[0055] According to the data quality report, use an artificial intelligence algorithm to filter out high-quality data to be updated, and analyze to obtain an update recommendation report;

[0056] According to the update recommendation report, develop a data updating strategy and generate a data updating execution schedule;

[0057] Perform data updating operation according to the data updating execution schedule, and record the update information to the blockchain at the same time;

[0058] The updated data is subjected to a deep check, and after the check is passed, it is stored in the local database and the result is recorded in the blockchain, completing the entire data update and management process.

[0059] The embodiment also provides a system for data update and management in a non-outbound network environment based on local deployment, which comprises the following steps of:

[0060] In the local non-outbound network environment, the local database, the file system, the sensor and the internal business system are explicitly defined as data sources of the data to be updated.

[0061] For different data sources, corresponding connection methods are adopted, such as connecting the local database using JDBC, accessing the file system through the file system API, connecting the sensor using the sensor communication protocol, and connecting the internal business system through the interface or data export function provided by the internal business system.

[0062] According to the preset data extraction rule, the data to be updated is extracted from each data source. For the local database, the newly added or modified data records are extracted according to the update timestamp of the data table; for the file system, the files to be updated are filtered according to the preset file name mode and timestamp; for the sensor, the environmental parameter data collected by the sensor is read at a fixed time interval; for the internal business system, the data export interface provided by the internal business system is called or the business data is obtained through the interface.

[0063] The data extracted from different data sources is converted into a unified intermediate format for subsequent processing. For example, the date formats of different databases are uniformly converted into the "YYYY-MM-DDHH:MM:SS" format, and the text encodings in different file systems are uniformly converted into the UTF-8 encoding.

[0064] The converted data is merged according to the preset data integration rule to generate an integrated data set containing all the data to be updated. The integration rule can include operations such as data deduplication, association and sorting to ensure that the integrated data is complete and consistent.

[0065] The integrated data set is stored in a temporary buffer area, and the buffer area is implemented by using an in-memory database or a cache system, which has the functions of fast reading and writing and data temporary storage, provides data input for subsequent preprocessing and quality detection steps, and ensures the timeliness and availability of the data.

[0066] It should be noted that in the local non-external network environment, the local database, file system, sensor and internal business system are first determined as the data source of the data to be updated. For different data sources, corresponding connection methods are used, such as connecting the local database using JDBC, accessing the file system through the file system API, connecting the sensor using the sensor communication protocol, and connecting the internal business system through the interface or data export function provided by the internal business system. According to the preset data extraction rule, the data to be updated is extracted from each data source. For the local database, the newly added or modified data records are extracted according to the update timestamp of the data table; for the file system, the files that need to be updated are filtered according to the preset file name mode and timestamp; for the sensor, the environmental parameter data collected by the sensor is read at a fixed time interval; for the internal business system, the data export interface provided by the internal business system is called or the business data is obtained through the interface. The data extracted from different data sources is converted into a unified intermediate format for subsequent processing. For example, the date formats of different databases are uniformly converted to "YYYY-MM-DDHH:MM:SS" format, and the text encodings in different file systems are uniformly converted to UTF-8 encoding. The converted data is merged according to the preset data integration rule to generate an integrated data set containing all the data to be updated. The integration rule can include data deduplication, association and sorting operations to ensure that the integrated data is complete and consistent. Finally, the integrated data set is stored in a temporary buffer, and the buffer is implemented by using an in-memory database or a cache system, which has the ability of fast reading and writing and data temporary storage, providing data input for subsequent preprocessing and quality detection steps to ensure the timeliness and availability of data.

[0067] The step is aimed at collecting data to be updated from multiple data sources in the local non-external network environment and performing preliminary integration to provide basic data support for subsequent data processing. By clearly defining multiple data sources and using corresponding connection methods, comprehensive coverage and effective connection of different data sources are achieved, ensuring the comprehensiveness and accuracy of data collection. According to the preset data extraction rule, the data to be updated is extracted, ensuring the timeliness and relevance of the data and avoiding unnecessary data processing, improving the data processing efficiency. The data from different data sources is converted into a unified intermediate format, eliminating data format differences and providing convenience for subsequent data integration and processing, improving the consistency and accuracy of data processing. The data is merged according to the preset data integration rule to generate an integrated data set, ensuring the integrity and consistency of the data and providing a high-quality data basis for subsequent data analysis and updating. The integrated data is stored in a temporary buffer, and the in-memory database or cache system is used to achieve fast reading and writing and data temporary storage, ensuring the timeliness and availability of the data and providing efficient data input for subsequent preprocessing and quality detection steps.

[0068] The embodiment also provides a system for data updating and management in a non-external network environment based on local deployment, comprising the following steps:

[0069] The integrated data set in the temporary buffer is completely read into the memory space of the preprocessing module to provide a data basis for subsequent operations.

[0070] The unique identifier is used for deduplication, correction of data format problems, and filling of missing values to obtain the data after preliminary cleaning.

[0071] According to the preset encoding rules and field consistency requirements, the consistency of the key fields in the data table is checked, the inconsistent data is marked and temporarily stored, and an intermediate report is generated.

[0072] According to the preset data quality evaluation index system, the data quality comprehensive score is calculated, and the data that does not meet the quality requirements is stored in the abnormal data temporary storage area.

[0073] According to the evaluation results, a structured data quality report is generated, covering the quality status and comprehensive score of each data source, to provide a basis for subsequent data screening.

[0074] It should be noted that the integrated data set in the temporary buffer is completely read into the memory space of the preprocessing module to ensure the continuity and integrity of the data, and to provide a data basis for subsequent operations. Then, the unique identifier is used for deduplication of the data, correction of data format problems, such as correction of inconsistent date formats, abnormal numerical range, etc., and filling of missing values, and the missing part of the numerical data is filled with the mean or median, and the mode is filled for classification data, to obtain the data after preliminary cleaning. Then, according to the preset encoding rules and field consistency requirements, the consistency of the key fields in the data table is checked, such as verifying whether the product code follows a unified encoding format, and whether the data of the same field in different data tables is consistent. The inconsistent data is marked and temporarily stored in the abnormal data temporary storage area, and an intermediate report containing detailed information about the data consistency problem is generated. Then, according to the preset data quality evaluation index system, the cleaned and verified data is quantitatively evaluated, the evaluation indexes include the integrity, accuracy, consistency, timeliness of the data, etc., the weight of each index is set, the data quality comprehensive score is calculated by weighted average, and the data quality evaluation result is formed. Finally, the data quality evaluation result is arranged into a detailed data quality report, the report content covers the quality status of each data source, the main quality problems and distribution, the data quality comprehensive score, etc. key information, presented in a structured format, to provide an objective basis for subsequent data updating decisions.

[0075] This step aims to preprocess and quality detect the collected data, generate a data quality report, and provide a basis for subsequent data screening and updating.

[0076] By reading the data in the buffer into the pre-processing module memory space, the continuity and integrity of the data are ensured, providing a stable data foundation for subsequent pre-processing operations. The unique identifier is used to remove duplicates and correct data format problems, and missing values are filled in to achieve preliminary cleaning of the data, improving the accuracy and integrity of the data. A high-quality data input is provided for subsequent data quality detection. According to the preset encoding rules and field consistency requirements, the consistency of key fields in the data table is checked, and inconsistent data is marked and temporarily stored, and an intermediate report is generated, which helps to timely discover and handle data consistency problems, ensuring the reliability and consistency of the data. According to the preset data quality evaluation index system, the comprehensive score of data quality is calculated, providing a quantitative evaluation method for data quality, making data quality evaluation more objective and accurate, and providing a scientific basis for subsequent data screening and update strategy formulation. According to the evaluation results, a structured data quality report is generated to present the quality status and comprehensive score of each data source in an intuitive way, providing clear guidance for subsequent data update decisions, which helps to improve the pertinence and effectiveness of data update.

[0077] The embodiment also provides a system for data update and management in a non-external network environment based on local deployment, comprising:

[0078] The data quality report is structured and analyzed to extract quality indicator values, problem data identifiers, and other key information from each data source, which are converted into a feature vector matrix as input data for the artificial intelligence algorithm.

[0079] The feature vector matrix is input into a pre-trained random forest-based screening model. The model automatically identifies and screens high-quality data that meets the quality requirements by learning from a large amount of historical data quality and business demand, and outputs a screening result containing high-quality data identifiers and priority.

[0080] Trend analysis is performed on the screened high-quality data to mine the trends and rules of data update, such as analyzing the growth trend, fluctuation period, and correlation with business indicators, etc., to provide data support for the update strategy.

[0081] According to the screening results and analysis trends, a data update recommendation report is generated. The report clearly indicates which data needs to be updated, the priority order of the update, and the expected update effect, and proposes solutions and preventive measures for different data quality problems, providing detailed guidance for subsequent data update operations.

[0082] It should be noted that the data quality report is structured and analyzed to extract quality indicator values, problem data identification and other key information of each data source, which is converted into a feature vector matrix as input data for the artificial intelligence algorithm. Then, the feature vector matrix is input into a pre-trained random forest-based screening model. The model automatically identifies and screens high-quality data that meets the quality requirements for updating by learning from a large amount of historical data quality and business demand, and outputs a screening result containing high-quality data identification and priority. Then, the screened high-quality data is analyzed for trends, and time series analysis is used to mine the trends and rules of data updating, such as analyzing the growth trend, fluctuation period, and correlation with business indicators, to provide data support for updating strategies. Finally, according to the screening results and analysis trends, a data update recommendation report is generated. The report clearly indicates which data needs to be updated, the priority order of updating, and the expected update effect, and proposes solutions and preventive measures for different data quality problems, providing detailed guidance for subsequent data updating operations.

[0083] This step aims to use artificial intelligence algorithms to analyze data quality reports in depth, screen high-quality data for updating, and generate update recommendation reports to provide scientific basis and detailed guidance for subsequent data updating.

[0084] By structurally analyzing and converting the data quality report into a feature vector matrix, the data is quantified and standardized, providing a high-quality data basis for the input of artificial intelligence algorithms, ensuring the accuracy and reliability of the screening results. The random forest-based screening model automatically identifies and screens high-quality data that meets the quality requirements for updating, improving the efficiency and accuracy of data screening, avoiding the subjectivity and errors of manual screening, and ensuring the quality of updated data. Trend analysis is performed on the screened high-quality data, and time series analysis is used to mine the trends and rules of data updating, providing data-driven decision support for updating strategies, making the updating strategy more scientific and reasonable, better meeting business needs, and generating a data update recommendation report that clearly identifies the data range, priority order, and expected effect that needs to be updated, and proposes solutions and preventive measures for different data quality problems, providing detailed guidance for subsequent data updating operations, improving the pertinence and effectiveness of data updating, and ensuring the smooth progress and expected effect of data updating.

[0085] The embodiment also provides a system for data updating and management in a non-external network environment based on local deployment, comprising:

[0086] The structured analysis of the update suggestion report extracts key information such as data update priority, expected effect, and quality problem solving suggestions, and converts them into strategy formulation parameters.

[0087] According to the strategy formulation parameters, combined with system resources and business needs, an optimization model based on genetic algorithm is used to formulate a data update strategy, determine the data update method, algorithm and tool, and plan the data update time window, and generate a data update execution schedule containing update tasks, time, and responsible persons.

[0088] It should be noted that the structured analysis of the update suggestion report extracts key information such as data update priority, expected effect, and quality problem solving suggestions, and converts them into strategy formulation parameters. Then, according to the strategy formulation parameters, combined with system resources and business needs, an optimization model based on genetic algorithm is used to formulate a data update strategy. In this process, the data update method, algorithm and tool are determined, and the data update time window is planned. Finally, a data update execution schedule containing update tasks, time, and responsible persons is generated. The schedule lists the specific content of each update task, execution time, responsible person, and expected completion effect, ensuring the orderly progress of data update operations.

[0089] This step aims to convert the key information in the update suggestion report into specific execution plans, providing clear guidance and time arrangements for data update operations, ensuring efficient execution of data updates.

[0090] By structurally analyzing the update suggestion report and converting it into strategy formulation parameters, the quantification and standardization of update requirements are achieved, providing a solid data foundation for formulating a scientific and reasonable data update strategy. The use of an optimization model based on genetic algorithm to formulate a data update strategy allows for comprehensive consideration of system resources and business needs, optimization of data update methods, algorithms and tool selection, and reasonable planning of update time windows. This approach not only improves the efficiency of data updates, but also ensures the stability and reliability of the update process. The generated data update execution schedule lists the specific content of each update task, execution time, responsible person, and expected completion effect, providing clear guidance and time arrangements for data update operations. This helps to improve the organization and planning of data updates, ensures that update tasks are completed on time, reduces potential impact on business, and enhances the traceability and responsibility implementation of the data update process, which helps to improve the quality and effect of data updates and ensures that data updates meet business needs.

[0091] The embodiment also provides a system for data update and management in a non-external network environment based on local deployment, which includes performing data update operations according to the data update execution schedule, and recording update information to a blockchain, with specific steps being,

[0092] The update task list is extracted from the data update execution schedule and converted into an executable update script set, with the script type determined according to the target data platform, such as SQL script for relational databases and NoSQL script for non-relational databases.

[0093] According to the predetermined time sequence of the update tasks, the update scripts are called in sequence to perform data update operations, and the update progress and system resource usage are monitored in real time during the update process to ensure stable operation of the update operations.

[0094] For each update operation, an update record block containing information such as update time, content, operator, and operation steps is generated, and the block data is encrypted using the hash algorithm of the blockchain to ensure the non-tamperability of the record.

[0095] The encrypted update record block is linked to the existing blockchain to form a non-tamperable update record chain, providing reliable security and audit evidence for the data update process, as well as support for subsequent data query and traceability.

[0096] It should be noted that the update task list is extracted from the data update execution schedule and converted into an executable update script set, with the script type determined according to the target data platform, such as SQL script for relational databases and NoSQL script for non-relational databases. Then, according to the predetermined time sequence of the update tasks, the update scripts are called in sequence to perform data update operations. During the update process, the update progress and system resource usage are monitored in real time to ensure stable operation of the update operations. For each update operation, an update record block containing information such as update time, content, operator, and operation steps is generated. The block data is encrypted using the hash algorithm of the blockchain to ensure the non-tamperability of the record. Finally, the encrypted update record block is linked to the existing blockchain to form a non-tamperable update record chain, providing reliable security and audit evidence for the data update process, as well as support for subsequent data query and traceability.

[0097] This step aims to efficiently perform data update operations according to the predetermined execution schedule and ensure the security, reliability, and traceability of the data update process using blockchain technology.

[0098] The automation and standardization of data update operations are realized by extracting an update task list from the execution schedule and converting it into a corresponding set of update scripts, the update efficiency and accuracy are improved, the manual intervention and potential errors are reduced, the update scripts are called in sequence according to the predetermined time sequence to perform the data update operations, and the update progress and system resource usage are monitored in real time to ensure the orderly progress of the update operations and the stable operation of the system, abnormal conditions in the update process are discovered and handled in a timely manner to ensure the reliability of the data update, update record blocks containing detailed update information are generated and encrypted using the hash algorithm of the blockchain to ensure the non-tamperability and security of the update records, reliable security and audit evidence are provided for the data update process, the credibility and transparency of the data update are enhanced, the encrypted update record blocks are linked to the existing blockchain to form a non-tamperable update record chain, the permanent preservation and traceability of the data update history are realized, and the subsequent data query, audit and problem tracing are facilitated, thereby providing strong support for data management and compliance checking.

[0099] The embodiment also provides a system for data update and management in a non-external network environment based on local deployment, which comprises: the data after completion of the update is subjected to deep checking, and after the checking is passed, the data is stored in a local database and the result is recorded in a blockchain, and the entire data update and management process is completed, and the specific steps are,

[0100] The data after the update is subjected to deep checking, and a hash value of the data is calculated using a hash checking algorithm and compared with a hash value of the original data before the update to ensure the accuracy and integrity of the data update.

[0101] If the data checking is passed, a suitable storage strategy is selected according to the data type: for structured data, a relational database is used for storage; and for unstructured data, an object storage service is used for storage. Meanwhile, an index is created for the stored data to facilitate subsequent efficient query and access.

[0102] The deep checking result and the data storage location information are recorded in the blockchain to generate a record block containing the checking result, the data storage location, the checking time and the like, and the block data is encrypted using the hash algorithm of the blockchain to ensure the non-tamperability of the record.

[0103] The encrypted record block is linked to the existing blockchain to form a non-tamperable data update and management record chain, reliable security and audit evidence are provided for the data update and management process, and support is provided for subsequent data query, tracing and management, and the entire data update and management process is completed.

[0104] It should be noted that the updated data is deeply checked, the hash value of the data is calculated by using the hash check algorithm, and the hash value of the original data before the update is compared to ensure the accuracy and integrity of the data update. Then, if the data check passes, a suitable storage strategy is selected according to the data type: for structured data, a relational database is used for storage; for unstructured data, object storage service is used for storage. At the same time, an index is created for the stored data to facilitate subsequent efficient query and access. Then, the deep check result and the data storage location information are recorded to the blockchain to generate a record block containing the check result, data storage location, check time and other information. The block data is encrypted by using the hash algorithm of the blockchain to ensure the non-tamperability of the record. Finally, the encrypted record block is linked to the existing blockchain to form an unalterable data update and management record chain, providing reliable security protection and audit support for the data update and management process, while providing support for subsequent data query, traceability and management, completing the entire data update and management process.

[0105] This step aims to perform final quality verification and secure storage on the updated data, ensure the accuracy and integrity of the data, and record the data update and management process by using blockchain technology to provide reliable security protection and audit support.

[0106] By using the hash check algorithm to deeply check the updated data, strict verification of the accuracy and integrity of the data update is realized, ensuring the quality and reliability of the data, providing a solid foundation for subsequent data use, selecting a suitable storage strategy according to the data type, improving the efficiency and query performance of data storage. An index is created for the stored data to further enhance the accessibility of the data, meeting the business demand for fast retrieval and access of data. The deep check result and the data storage location information are recorded to the blockchain, and the hash algorithm of the blockchain is used for encryption processing to ensure the non-tamperability and security of the record, providing reliable security protection and audit support for the data update and management process, enhancing the transparency and credibility of data management. The encrypted record block is linked to the existing blockchain to form an unalterable data update and management record chain, realizing the permanent preservation and traceability of the data update history, facilitating subsequent data query, audit and problem traceability, providing strong support for data management and compliance check, ensuring the integrity and security of the data update and management process.

[0107] The embodiment also provides a computer device suitable for the method of data update and management in a non-external network environment based on local deployment, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the method of data update and management in a non-external network environment based on local deployment as proposed in the above embodiment.

[0108] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, a trackball or a touchpad arranged on the shell of the computer device, or an external keyboard, a touchpad or a mouse, etc.

[0109] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for updating and managing data based on a locally deployed non-external network environment according to the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0110] In summary, the present application ensures the timeliness and availability of data by collecting the data to be updated from multiple data sources and storing in a temporary buffer, effectively improves the accuracy and integrity of the data by generating a data quality report through preprocessing and quality detection, realizes the intelligentization and automation of data updating by using artificial intelligence algorithm to screen high-quality data and generate an update recommendation report, improves the scientificity and rationality of data updating based on the update recommendation report to formulate a data update strategy and generate an execution schedule, ensures the security and traceability of the data updating process by recording the update information to the blockchain when executing the data updating operation, and finally performs deep checking on the updated data and stores it to the local database while recording the result to the blockchain, further guaranteeing the reliability and security of the data. This method effectively solves the problems of data updating not in time, uneven data quality, and data security cannot be guaranteed in the prior art, and realizes the efficiency, accuracy and reliability of data updating and management.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A method for updating and managing data in a non-external network environment based on local deployment, characterized in that: The following steps are involved: In a local, non-external network environment, data to be updated is collected from multiple data sources and stored in a temporary buffer; Read the data in the buffer for preprocessing and quality testing, and generate a data quality report; Based on the data quality report, use artificial intelligence algorithms to screen out high-quality data to be updated, and analyze it to produce an update recommendation report; Develop data update strategies and generate data update execution plans based on update recommendation reports; Execute data update operations according to the data update execution plan, and record the update information to the blockchain; The updated data is deeply verified, stored in the local database after verification, and the results are recorded in the blockchain to complete the entire data update and management process.

2. The method for updating and managing data in a local, non-external network environment according to claim 1, characterized in that: In the local non-external network environment, the data to be updated is collected from multiple data sources and stored in a temporary buffer. The specific steps are: In a local, non-external network environment, clearly identify local databases, file systems, sensors, and internal business systems as data sources for the data to be updated. For different data sources, corresponding connection methods are adopted, such as using JDBC to connect to the local database, accessing the file system through the file system API, connecting to sensors using the sensor communication protocol, and connecting to the internal business system through the interface or data export function provided by the internal business system. Extract the data to be updated from various data sources based on the preset data extraction rules. For local databases, extract new or modified data records based on the update timestamp of the data table; For the file system, filter out the files that need to be updated according to the preset file name pattern and timestamp; for the sensor, read the environmental parameter data collected by it at fixed time intervals; for the internal business system, call the data export interface provided by it or obtain business data through the interface. Convert data extracted from different data sources into a unified intermediate format for subsequent processing. For example, convert the date format of different databases to the "YYYY-MM-DDHH:MM:SS" format, or convert the text encoding in different file systems to UTF-8 encoding. The converted data is merged according to pre-set data integration rules to generate a consolidated dataset containing all the data to be updated. Integration rules can include operations such as data deduplication, association, and sorting to ensure the integrity and consistency of the consolidated data. The integrated data set is stored in a temporary buffer, which is implemented using an in-memory database or a high-speed cache system. The buffer has fast reading and writing and data temporary storage capabilities, providing data input for subsequent preprocessing and quality inspection steps to ensure the timeliness and availability of data.

3. The method for updating and managing data in a non-external network environment based on local deployment according to claim 2, characterized in that: The data in the read buffer is preprocessed and quality tested to generate a data quality report. The specific steps are: The integrated data set in the temporary buffer is completely read into the memory space of the preprocessing module to provide a data basis for subsequent operations. Use unique identifiers to remove duplicates, correct data format problems, and fill in missing values ​​to obtain preliminary cleaned data. Based on preset coding rules and field consistency requirements, check the consistency of key fields in the data table, mark inconsistent data and temporarily store it, and generate an intermediate report. According to the preset data quality assessment indicator system, the comprehensive data quality score is calculated, and the data that does not meet the quality requirements is stored in the abnormal data temporary storage area. Generate a structured data quality report based on the assessment results, covering the quality status and comprehensive score of each data source, providing a basis for subsequent data screening.

4. The method for updating and managing data in a local, non-external network environment according to claim 3, wherein: Based on the data quality report, the artificial intelligence algorithm is used to screen out high-quality data to be updated, and the update recommendation report is obtained through analysis. The specific steps are as follows: Perform structured analysis on the data quality report to extract key information such as the quality indicator values ​​of each data source, problem data identifiers, etc., and convert them into a feature vector matrix as input data for the artificial intelligence algorithm. The feature vector matrix is ​​input into a pre-trained random forest-based screening model. By learning from the quality of a large amount of historical data and business needs, the model automatically identifies and screens out high-quality data to be updated that meets the quality requirements, and outputs the screening results including the identification and priority of high-quality data. Conduct trend analysis on the selected high-quality data and use time series analysis methods to explore the trends and patterns of data updates. For example, analyze the data's growth trend, fluctuation cycle, and correlation with business indicators to provide data support for update strategies. Based on the screening results and trend analysis, a data update recommendation report is generated. The report clearly indicates which data needs to be updated, the priority of the update, and the expected update results. It also provides solutions and preventative measures for different data quality issues, providing detailed guidance for subsequent data update operations.

5. The method for updating and managing data in a non-external network environment based on local deployment according to claim 4, characterized in that: According to the update suggestion report, a data update strategy is formulated and a data update execution plan is generated. The specific steps are: Perform structured analysis on the update recommendation report to extract key information such as data update priority, expected effects, and quality issue resolution suggestions, and convert them into policy formulation parameters. According to the policy formulation parameters, combined with system resources and business needs, the data update strategy is formulated using the optimization model based on genetic algorithm, the data update method, algorithm and tools are determined, and the data update time window is planned to generate a data update execution plan that includes update tasks, time and responsible persons.

6. The method for updating and managing data in a local non-external network environment according to claim 5, characterized in that: The data update operation is performed according to the data update execution schedule, and the update information is recorded in the blockchain. The specific steps are: Extract the update task list from the data update execution plan table and convert it into an executable update script set. The script type is determined by the target data platform, such as SQL scripts for relational databases and NoSQL scripts for non-relational databases. According to the scheduled time sequence of the update tasks, the update scripts are called in sequence to perform data update operations. During the update process, the update progress and system resource usage are monitored in real time to ensure the stable operation of the update operation. For each update operation, an update record block is generated containing information such as update time, content, operator, and operation steps. The block data is encrypted using the blockchain's hash algorithm to ensure that the record cannot be tampered with. The encrypted update record block is linked to the existing blockchain to form an unalterable update record chain, providing reliable security and audit basis for the data update process, while also providing support for subsequent data query and tracing.

7. The method for updating and managing data in a local non-external network environment according to claim 6, characterized in that: The updated data is deeply verified, and after passing the verification, it is stored in the local database, and the results are recorded in the blockchain to complete the entire data update and management process. The specific steps are: Perform in-depth verification on the updated data, use a hash verification algorithm to calculate the hash value of the data, and compare it with the hash value of the original data before the update to ensure the accuracy and completeness of the data update. If the data passes verification, the appropriate storage strategy is selected based on the data type: for structured data, a relational database is used; for unstructured data, an object storage service is used. At the same time, indexes are created for the stored data to facilitate efficient subsequent query and access. The deep verification results and data storage location information are recorded in the blockchain, and a record block containing verification results, data storage location, verification time and other information is generated. The block data is encrypted using the blockchain's hash algorithm to ensure that the record cannot be tampered with. The encrypted record block is linked to the existing blockchain to form an unalterable data update and management record chain, providing reliable security and audit basis for the data update and management process, while providing support for subsequent data query, traceability and management, completing the entire data update and management process.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for updating and managing data in a local deployment non-external network environment according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for updating and managing data in a local deployment non-external network environment according to any one of claims 1 to 7 are implemented.

Images