A data lifecycle management method and system based on data freeze identification

Abstract

The present application relates to a kind of data lifecycle management method and system based on data freeze identification, first query service and second query service of independent configuration are routed to online database and nearline database respectively, and freeze label is added to online data to produce frozen data as the buffer between online data and nearline data, so that business can access historical data in real time without waiting for big data T+1 recovery data under the condition that current database structure is unchanged;Solve the problem of growing online data space, reduce the waste of hardware resources, and ensure stable operation of database;Realize the automatic filing and cleaning of online data, avoid manual operation, reduce the operation risk, and protect the stability of business.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing and database operation and maintenance, and particularly relates to a data life cycle management method and system based on data freezing identification. BACKGROUND

[0002] Database data life cycle management is an information management model, which focuses on the life cycle of data from creation, protection, access, migration, archiving, and recycling (destruction). According to the specific characteristics of data generation and use, typical data life cycle management can classify various data into different categories of online data, nearline data, and offline data. Among them, online data refers to data stored in various business systems of data production area, data integration area, and data application area, which is used for access in the daily operation process of business systems. The retention value, data storage, user access frequency, and security protection of online data are relatively high; the saving time should be able to maintain the normal operation of business systems and maintain the normal activities of business. Nearline data refers to data in various business systems of data production area, data integration area, and data application area that exceeds the retention period, which is migrated to the historical data query platform and the source data is deleted, and the nearline data supports online access. Offline data is data backup, which is used for data recovery after business system failure, and the source data is not deleted. Generally, offline data cannot be accessed online.

[0003] When performing data life cycle management, multiple influencing factors need to be considered. Typically, different types and formats of data may come from various systems and platforms, which need to be uniformly collected, converted, and stored, so appropriate ETL processes and powerful and flexible tool support need to be established; a large amount of unstructured or semi-structured data (such as text files, pictures, videos, etc.) is involved in modern enterprises, and its processing method is different from that of traditional relational databases; since a large amount of sensitive information exists in the enterprise, data protection needs to be carried out in accordance with security and compliance requirements, and appropriate data access control strategies also need to be formulated to ensure data privacy; data asset management needs to break down data silos, and data can only play a greater value when integrated, and only when data integration is achieved can the data warehouse be called a data warehouse, otherwise even if the data is centralized, it cannot achieve the effect of aggregated utilization.

[0004] Current technologies primarily employ a solution of importing historical data into a data warehouse, deleting existing online data, and controlling the scale of online data. However, historical data query requirements typically require a T+1 cycle to be fulfilled. Therefore, in practical applications, existing technologies cannot support real-time historical data query needs. For business systems to query historical data already in the warehouse, they must first restore data from the data warehouse for a specified period to the query platform, which takes at least one day, making real-time querying impossible. Furthermore, the continuous storage of historical data after it's entered into the warehouse leads to a continuous increase in data volume, resulting in wasted resources. Deleting existing online data requires manual operation, which is slow and prone to errors.

[0005] This reveals that existing technologies for data lifecycle management have serious problems such as significantly impacting the scale of online data and wasting server physical resources, posing serious risks to resource management and the stable operation of business. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a data lifecycle management method and system based on data freeze identification. Businesses can access historical data in real time without waiting for T+1 data recovery for large datasets; it solves the problem of continuously increasing online data storage space in databases, reduces waste of hardware resources, and ensures stable database operation; it also enables automatic archiving and cleanup of online data, avoiding manual operations, reducing operational risks, and ensuring business stability.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0008] A data lifecycle management method based on data freeze identification, characterized in that it includes:

[0009] S1. Configure a first query service and a second query service that are independent of each other. The first query service is routed to the online database, and the second query service is routed to the near-line database.

[0010] S2. Determine whether the online data in the online database has not been updated for more than the first preset threshold based on the first preset inspection cycle;

[0011] S3. When it is determined that the online data in the online database has not been updated for a longer period than the first preset threshold, a freeze tag is added to the online data to generate frozen data.

[0012] S4. Determine whether the time during which the frozen data in the online database has not been updated exceeds the second preset threshold based on the second preset inspection cycle;

[0013] S5. When it is determined that the frozen data in the online database has not been updated for more than the second preset threshold, the frozen data is cut and transferred to the nearline database and converted into data to be stored.

[0014] S6. Determine whether there is existing nearline data in the nearline database that matches the data to be stored. If it is determined that there is no existing nearline data that matches the data to be stored, record the data to be stored as nearline data and save it. The nearline data includes a corresponding freeze tag.

[0015] S7. When it is determined that there is existing nearline data that matches the data to be stored, delete the frozen tags of the existing nearline data and cut and transfer it to the offline database, and record the data to be stored as nearline data and save it.

[0016] Furthermore, the method also includes:

[0017] S8. Obtain the data request and distribute the data request to the first query service and the second query service respectively;

[0018] S9. Feedback the data request based on the output information of the first query service and the second query service.

[0019] Further, step S9 includes:

[0020] The first query service determines whether the target data matching the data request exists in the online database. When it is determined that the target data matching the data request exists in the online database, the target data in the online database is used to respond to the data request.

[0021] When it is determined that the target data matching the data request does not exist in the online database, the second query service is used to determine whether the target data matching the data request exists in the near-line database. When it is determined that the target data matching the data request exists in the near-line database, the target data in the near-line database is used to respond to the data request.

[0022] When it is determined that the target data matching the data request does not exist in the nearline database, a prompt message is output to provide feedback on the data request.

[0023] Furthermore, step S9 also includes:

[0024] Determine whether the target data includes frozen tags;

[0025] When it is determined that the target data includes frozen tags, check whether the target data has been updated and use the updated target data to respond to the data request.

[0026] Furthermore, the interval length of the first preset inspection cycle is shorter than the second preset inspection cycle; the duration of the first preset threshold is longer than the second preset threshold.

[0027] Furthermore, step S7 also includes:

[0028] Determine whether the nearline data storage time exceeds the third preset threshold;

[0029] When it is determined that the nearline data storage time exceeds the third preset threshold, the frozen tag of the nearline data is deleted and the data is cut and transferred to the offline database.

[0030] This invention also relates to a data lifecycle management system based on data freeze identification, characterized in that it includes:

[0031] The query service management module is used to configure the independent first query service and second query service.

[0032] The freeze judgment module is used to determine whether the time during which online data in the online database has not been updated exceeds the first preset threshold based on the first preset inspection cycle, and to add a freeze tag to the online data to generate frozen data;

[0033] The near-line judgment module is used to determine whether the frozen data in the online database has not been updated for a period of time exceeding a second preset threshold, based on a second preset inspection cycle, and to cut and transfer the frozen data to the near-line database and convert it into data to be stored.

[0034] The nearline storage module is used to determine whether there is existing nearline data in the nearline database that matches the data to be stored, and to record the data to be stored as nearline data for saving, as well as to delete the frozen tags of existing nearline data and cut and transfer it to the offline database.

[0035] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0036] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0037] The memory is used to store the frozen tags and the data to be stored;

[0038] The processor is used to execute the above method by invoking the frozen tag and the data to be stored.

[0039] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0040] The beneficial effects of this invention are as follows:

[0041] By adopting the data lifecycle management method and system based on data freeze identification as described in this invention, businesses can access historical data in real time without waiting for big data T+1 data recovery; it solves the problem of the continuous growth of online data space occupied by the database, reduces the waste of hardware resources, and ensures the stable operation of the database; data that no longer changes but still has query needs is transferred to online data archiving, controlling the scale of online data, realizing automatic archiving and cleaning of online data, avoiding manual operation, reducing operational risks, and ensuring business stability. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the data lifecycle management method based on data freeze identification according to the present invention.

[0043] Figure 2 This is a schematic diagram of the data lifecycle management system based on data freeze identification according to the present invention. Detailed Implementation

[0044] To better understand the content of this invention, a detailed description will be provided in conjunction with the accompanying drawings and embodiments.

[0045] The first aspect of this invention relates to a process flow as follows: Figure 1 The data lifecycle management method based on data freeze identification shown includes:

[0046] S1. Configure a first query service and a second query service that are independent of each other. The first query service is routed to the online database, and the second query service is routed to the near-line database.

[0047] In practice, first and second query services can be provided by building a historical data query platform. Preferably, a proxy component, HistoricalDataProxy, receives application SQL requests and automatically distributes them to the online or historical data platform, enabling real-time access to near-line data without requiring application modifications. When a business sends an SQL request to HistoricalDataProxy, the proxy parses the received SQL and, based on the SQL's operation type, filter fields, and other conditions, distributes the SQL to the corresponding platform according to the proxy's distribution strategy.

[0048] S2. Determine whether the time during which online data in the online database has not been updated exceeds the first preset threshold based on the first preset inspection cycle.

[0049] S3. When it is determined that the online data in the online database has not been updated for more than the first preset threshold, a freeze tag is added to the online data to generate frozen data.

[0050] For the initial processing of a newly built system, you can choose the existing tool - data pipeline, to import historical data once for initialization and import incremental data periodically.

[0051] For example, assuming data that hasn't changed for more than a year, it can be synchronized to the historical data query platform with a synchronization cycle of one month and a data verification time of one month. The current time is T. The current month verifies and repairs monthly data up to T-13, synchronizes monthly data up to T-12, and can delete monthly data up to T-14. To ensure the accuracy of query results, online data must be stored for at least 16 months; at least 15 months of data should be configured for querying online data; for data older than 15 months, recent online data should be retrieved through the historical data query platform.

[0052] S4. Determine whether the time during which the frozen data in the online database has not been updated exceeds the second preset threshold, based on the second preset inspection cycle.

[0053] Preferably, the interval length of the first preset inspection cycle is less than the second preset inspection cycle; and the duration of the first preset threshold is greater than the second preset threshold.

[0054] The platform automatically creates new synchronization tasks at the beginning of each month, and synchronizes online data to the historical data platform monthly according to the configured time rules.

[0055] S5. When it is determined that the frozen data in the online database has not been updated for more than the second preset threshold, the frozen data is cut and transferred to the nearline database and converted into data to be stored.

[0056] S6. Determine whether there is existing nearline data in the nearline database that matches the data to be stored. If it is determined that there is no existing nearline data that matches the data to be stored, record the data to be stored as nearline data and save it. The nearline data includes a corresponding freeze tag.

[0057] S7. When it is determined that there is existing nearline data that matches the data to be stored, delete the frozen tags of the existing nearline data and cut and transfer it to the offline database, and record the data to be stored as nearline data and save it.

[0058] Preferably, the system also includes determining whether the near-line data storage time exceeds a third preset threshold; when the near-line data storage time exceeds the third preset threshold, the frozen tag of the near-line data is deleted and the data is cut and transferred to the offline database. The system periodically scans the near-line data of the historical query platform, and after discovering historical data that has exceeded the storage time, it calls the work order system to automatically generate a pending task to delete the historical data.

[0059] Preferably, the synchronized data can be validated monthly according to preset time rules. For example, the platform compares the amount of online data and near-online data, the average, maximum, minimum, and sum of the validation fields within the validation time range based on the validation fields of the table to determine whether the data is consistent.

[0060] S8. Obtain the data request and distribute the data request to the first query service and the second query service respectively.

[0061] S9. Feedback the data request based on the output information of the first and second query services. During execution, the execution results from the online data platform and the historical data platform are received, the results are summarized, and then returned to the application.

[0062] Specifically, the first query service determines whether the target data matching the data request exists in the online database. If the target data matching the data request exists in the online database, the target data in the online database is used to respond to the data request. If the target data matching the data request does not exist in the online database, the second query service determines whether the target data matching the data request exists in the near-line database. If the target data matching the data request exists in the near-line database, the target data in the near-line database is used to respond to the data request. If the target data matching the data request does not exist in the near-line database, a prompt message is output to respond to the data request.

[0063] Preferably, during step S9, it can also be determined whether the target data includes a frozen label; when it is determined that the target data includes a frozen label, it is checked whether the target data has been updated and the updated target data is used to send a data request.

[0064] Another aspect of the present invention relates to a data lifecycle management system based on data freeze identification, the structure of which is as follows: Figure 2 As shown, it includes:

[0065] The query service management module is used to configure the independent first query service and second query service.

[0066] The freeze judgment module is used to determine whether the time during which online data in the online database has not been updated exceeds the first preset threshold based on the first preset inspection cycle, and to add a freeze tag to the online data to generate frozen data;

[0067] The near-line judgment module is used to determine whether the frozen data in the online database has not been updated for a period of time exceeding a second preset threshold, based on a second preset inspection cycle, and to cut and transfer the frozen data to the near-line database and convert it into data to be stored.

[0068] The nearline storage module is used to determine whether there is existing nearline data in the nearline database that matches the data to be stored, and to record the data to be stored as nearline data for saving, as well as to delete the frozen tags of existing nearline data and cut and transfer it to the offline database.

[0069] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0070] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0071] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device includes at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as frozen tags and data to be stored. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0072] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0073] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0078] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data lifecycle management method based on data freeze identification, characterized in that, include: S1. Configure a first query service and a second query service that are independent of each other. The first query service is routed to the online database, and the second query service is routed to the near-line database. S2. Determine whether the online data in the online database has not been updated for more than the first preset threshold based on the first preset inspection cycle; S3. When it is determined that the online data in the online database has not been updated for a longer period than the first preset threshold, a freeze tag is added to the online data to generate frozen data. S4. Determine whether the time during which the frozen data in the online database has not been updated exceeds the second preset threshold based on the second preset inspection cycle; S5. When it is determined that the frozen data in the online database has not been updated for more than the second preset threshold, the frozen data is cut and transferred to the nearline database and converted into data to be stored. S6. Determine whether there is existing nearline data in the nearline database that matches the data to be stored. If it is determined that there is no existing nearline data that matches the data to be stored, record the data to be stored as nearline data and save it. The nearline data includes a corresponding freeze tag. S7. When it is determined that there is existing nearline data that matches the data to be stored, delete the frozen tags of the existing nearline data and cut and transfer it to the offline database, and record the data to be stored as nearline data and save it.

2. The method as described in claim 1, characterized in that, The method further includes: S8. Obtain the data request and distribute the data request to the first query service and the second query service respectively; S9. Feedback the data request based on the output information of the first query service and the second query service.

3. The method as described in claim 2, characterized in that, S9 includes: The first query service determines whether the target data matching the data request exists in the online database. When it is determined that the target data matching the data request exists in the online database, the target data in the online database is used to respond to the data request. When it is determined that the target data matching the data request does not exist in the online database, the second query service is used to determine whether the target data matching the data request exists in the near-line database. When it is determined that the target data matching the data request exists in the near-line database, the target data in the near-line database is used to respond to the data request. When it is determined that the target data matching the data request does not exist in the nearline database, a prompt message is output to provide feedback on the data request.

4. The method as described in claim 3, characterized in that, S9 further includes: Determine whether the target data includes frozen tags; When it is determined that the target data includes frozen tags, check whether the target data has been updated and use the updated target data to respond to the data request.

5. The method as described in claim 1, characterized in that, The interval length of the first preset check cycle is less than the second preset check cycle; the duration of the first preset threshold is greater than the second preset threshold.

6. The method as described in claim 1, characterized in that, The S7 also includes: Determine whether the nearline data storage time exceeds the third preset threshold; When it is determined that the nearline data storage time exceeds the third preset threshold, the frozen tag of the nearline data is deleted and the data is cut and transferred to the offline database.

7. A data lifecycle management system based on data freeze identification, characterized in that, include: The query service management module is used to configure the independent first query service and second query service. The freeze judgment module is used to determine whether the time during which online data in the online database has not been updated exceeds the first preset threshold based on the first preset inspection cycle, and to add a freeze tag to the online data to generate frozen data; The near-line judgment module is used to determine whether the frozen data in the online database has not been updated for a period of time exceeding a second preset threshold, based on a second preset inspection cycle, and to cut and transfer the frozen data to the near-line database and convert it into data to be stored. The nearline storage module is used to determine whether there is existing nearline data in the nearline database that matches the data to be stored, and to record the data to be stored as nearline data for saving, as well as to delete the frozen tags of existing nearline data and cut and transfer it to the offline database.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store the frozen tags and the data to be stored; The processor is configured to execute the method of any one of claims 1 to 6 by invoking the frozen tag and the data to be stored.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images