Data archiving method, computer device, computer-readable storage medium and computer program product

Abstract

This application relates to a data archiving method, computer equipment, computer-readable storage medium, and computer program product, belonging to the field of big data technology. The method includes: acquiring target data to be archived; during the synchronization of the target data to a data warehouse, providing feedback on the data synchronization status of the target data to the client; once the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archiving storage area, and providing feedback on the data archiving status of the target data in the archiving storage area to the client; instructing the client to display the data archiving status of the target data once it has been synchronized to the data warehouse; and ensuring that data modification permissions for the target data already archived to the archiving storage area are disabled. This method ensures the security of data archiving.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data archiving method, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of big data technology, data archiving has become one of the important means of effectively managing and maintaining data.

[0003] Current technology typically stores data to be archived in a database to achieve data archiving. However, databases are susceptible to modification, and the security of archived data cannot be guaranteed. Summary of the Invention

[0004] Therefore, it is necessary to provide a data archiving method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can ensure the security of data archiving, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a data archiving method applied to a big data platform, which includes a data warehouse and an archive storage area. The method includes: acquiring target data to be archived; during the process of synchronizing the target data to the data warehouse, providing feedback on the data synchronization status of the target data to the client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to the archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data already archived to the archive storage area is closed.

[0006] Secondly, this application also provides a data archiving method applied to a client, comprising: in response to a data archiving event, determining the target data to be archived as indicated by the data archiving event; sending the target data to a big data platform and displaying the data synchronization status of the target data in the data warehouse of the big data platform; when the data synchronization status indicates that the target data has been synchronized to the data warehouse, displaying the data archiving status of the target data in the archive storage area of ​​the big data platform; and the data modification permission of the target data that has been archived to the archive storage area is in a closed state.

[0007] Thirdly, this application also provides a data archiving device applied to a big data platform, which includes a data warehouse and an archive storage area. The device includes: an archive data acquisition module for acquiring target data to be archived; a data synchronization module for providing feedback on the data synchronization status of the target data to the client during the process of synchronizing the target data to the data warehouse; a data archiving module for archiving the target data in the data warehouse to the archive storage area after the target data has been synchronized to the data warehouse, and providing feedback on the data archiving status of the target data in the archive storage area to the client; a data synchronization status for instructing the client to display the data archiving status of the target data after the target data has been synchronized to the data warehouse; and the data modification permission for the target data already archived to the archive storage area is in a closed state.

[0008] Fourthly, this application also provides a data archiving device applied to a client, comprising: a data determination module for determining the target data to be archived as indicated by the data archiving event in response to the data archiving event; a data synchronization status display module for sending the target data to a big data platform and displaying the data synchronization status of the target data in the data warehouse of the big data platform; and a data archiving status display module for displaying the data archiving status of the target data in the archiving storage area of ​​the big data platform when the data synchronization status indicates that the target data has been synchronized to the data warehouse; and the data modification permission of the target data that has been archived to the archiving storage area is in a closed state.

[0009] Fifthly, this application also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to the client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data already archived to the archive storage area is closed.

[0010] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it performs the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to the client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to the archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data already archived to the archive storage area is closed.

[0011] Seventhly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to the client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data already archived to the archive storage area is disabled.

[0012] The aforementioned data archiving methods, apparatus, computer equipment, computer-readable storage media, and computer program products are applied to big data platforms. It is understood that a big data platform is a platform used for storing, managing, and analyzing massive amounts of data. Compared to existing technologies that use databases to store data, big data platforms can overcome the limitations of databases, such as limitations on the amount of data that can be stored, limitations on data structure, and limitations on storage space, thereby effectively improving the efficiency of data storage. When data archiving is required, the big data platform first obtains the target data to be archived and synchronizes it to the data warehouse. In this way, a reliable synchronized copy of the target data exists within the big data platform. Considering that the data warehouse can still be modified through specific instructions, the big data platform will perform further data archiving processing on the target data in the data warehouse after the target data has been successfully synchronized to the data warehouse. This means archiving the target data in the data warehouse to a pre-created archiving storage area. After successful archiving, the data modification permissions for the target data in the archiving storage area are closed, thereby ensuring that the target data in the archiving storage area cannot be modified again, thus ensuring the security of the target data. Throughout the process, the big data platform will provide feedback to the client on the data synchronization status and data archiving status of the target data, allowing relevant personnel to clearly understand the data archiving situation. Therefore, this application can overcome the technical problem of low data archiving security in the prior art, and can improve the efficiency of data archiving while ensuring data archiving security. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a diagram illustrating the application environment for data archiving in one embodiment;

[0015] Figure 2 This is a flowchart illustrating a data archiving method applied to a big data platform in one embodiment;

[0016] Figure 3 This is a flowchart illustrating the data consistency comparison process in one embodiment;

[0017] Figure 4 This is a schematic diagram of the data archiving retry process in one embodiment;

[0018] Figure 5This is a flowchart illustrating the process of deleting target data from a source database in one embodiment;

[0019] Figure 6 This is a schematic diagram of the data archiving process in one embodiment;

[0020] Figure 7 This is a schematic diagram illustrating the process of creating a data warehouse and archive storage area in one embodiment;

[0021] Figure 8 This is a flowchart illustrating a data archiving method applied to a client in one embodiment;

[0022] Figure 9 This is a schematic diagram of data archiving in a specific embodiment;

[0023] Figure 10 This is a structural block diagram of a data archiving device applied to a big data platform in one embodiment;

[0024] Figure 11 This is a structural block diagram of a data archiving device applied to a client in one embodiment;

[0025] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] The data archiving method provided in this application embodiment can be applied to, for example... Figure 1 The big data platform shown includes a client 102 and a server 104. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0028] Specifically, in response to a data archiving event, client 102 identifies the target data to be archived as indicated by the event and sends it to server 104. After receiving the target data, server 104 first performs data synchronization, that is, synchronizes the target data to the data warehouse in the big data platform. During this process, server 104 provides feedback on the data synchronization status of the target data to client 102, allowing client 102 to view the synchronization status. Once the target data has been successfully synchronized to the data warehouse, server 104 performs data archiving, that is, archives the target data from the data warehouse to the archive storage area, and provides feedback on the data archiving status of the target data in the archive storage area to client 102, allowing client 102 to view the data archiving status. After the target data in the data warehouse is successfully archived to the archive storage area, server 104 disables data modification permissions for the target data in the archive storage area to ensure that the target data in the archive storage area cannot be modified again.

[0029] In one exemplary embodiment, such as Figure 2 As shown, a data archiving method is provided, which can be applied to... Figure 1 Taking server 104 of the big data platform as an example, the explanation includes:

[0030] Step S202: Obtain the target data to be archived.

[0031] Here, the target data to be archived refers to the data that needs to be archived. In practical applications, different types of data can be archived, and this embodiment does not impose any restrictions.

[0032] It's important to note that the target data to be archived cannot be updated in any way within the business logic; otherwise, inconsistencies may arise between the archived data and the original data. Therefore, the target data to be archived needs to be data that cannot be updated, such as data from several months ago or even longer ago.

[0033] For example, a user can initiate a data archiving event for target data to be archived through the client of a big data platform. The client can then obtain the target data to be archived and send it to the server of the big data platform, so that the server can perform data synchronization and data archiving processing on the target data.

[0034] Optionally, users can pre-configure data archiving tasks on the server. These tasks can include a source database (the database to be synchronized), a target database (the database to be synchronized, such as the Hive database of a big data platform), and a synchronization cycle such as daily or monthly. In this way, the scheduling system in the big data platform can automatically execute the data archiving tasks, that is, extract data from the source database daily or monthly and synchronize it to the target database.

[0035] Step S204: During the process of synchronizing the target data to the data warehouse, the data synchronization status of the target data is fed back to the client.

[0036] In this context, a data warehouse can be understood as a database within a big data platform used to store data to be archived; specifically, it can be a Hive library within the big data platform. Within the data warehouse, or Hive library, the target data to be archived is mapped to database tables, and a SQL-like query language (Hive SQL) is provided, allowing user objects to perform data operations on the database tables using SQL syntax. Data synchronization status is used to characterize the data synchronization progress of the target data, and can include states such as data synchronization started, data synchronization in progress, data synchronization completed, data synchronization completed, and data synchronization failed. The data synchronization status can also be used to indicate to the client the data archiving status of the target data once it has been synchronized to the data warehouse.

[0037] For example, the server of a big data platform will synchronize the target data to be archived to a Hive database, or in other words, synchronize the target data to a pre-created Hive table in the Hive database that matches the source database of the target data. During this process, the server can send the data synchronization status of the target data to the client in real time, so that the client can display the data synchronization status on the front-end interface in real time, thereby making it easy for users to clearly understand the synchronization status of the target data.

[0038] Step S206: If the target data has been synchronized to the data warehouse, archive the target data in the data warehouse to the archive storage area and report the data archiving status of the target data in the archive storage area to the client.

[0039] The archive storage area is a region or space within a big data platform used to store archived data. It can be a separate file directory, a separate database, or an archive system. The data archiving status indicates the archiving progress of the target data and can include states such as data archiving has begun, data archiving is in progress, data archiving has ended, data archiving is complete, and data archiving has failed. Data modification permissions for target data already archived to the archive storage area are disabled.

[0040] For example, considering that data in a data warehouse can still be modified using SQL syntax, making it impossible to guarantee the security of the target data in the data warehouse, this embodiment adds data archiving processing. An archive storage area corresponding to the data warehouse is pre-created in the big data platform. After successfully synchronizing the target data to be archived to the data warehouse, the big data platform's server archives the target data from the data warehouse into this archive storage area. After successful archiving, data modification permissions for the target data in this archive storage area are disabled, meaning that external parties can no longer modify the target data in this archive storage area, thus ensuring the security of the archived data. During this process, the server can also send the data archiving status of the target data to the client in real time, allowing the client to display the data archiving status on the front-end interface, thus enabling users to clearly understand the archiving status of the target data.

[0041] Optionally, when human error causes the target data in the data warehouse to be mistakenly deleted or modified, or when the data warehouse is abnormal, the data can be restored using the target data in the archive storage area.

[0042] This embodiment applies to a big data platform, which is used to store, manage, and analyze massive amounts of data. Compared to existing technologies that use databases to store data, big data platforms overcome the limitations of databases, such as limitations on the amount of data that can be stored, limitations on data structure, and limitations on storage space, thereby effectively improving the efficiency of data storage. When data archiving is required, the big data platform first obtains the target data to be archived and synchronizes it to the data warehouse. Thus, a reliable synchronized copy of the target data exists within the big data platform. Considering that the data warehouse can still be modified through specific instructions, the big data platform performs further data archiving processing on the target data in the data warehouse after successful synchronization. This involves archiving the target data in the data warehouse to a pre-created archiving storage area. After successful archiving, the data modification permissions for the target data in the archiving storage area are disabled, ensuring that the target data in the archiving storage area cannot be modified again, thereby ensuring the security of the target data. Throughout the process, the big data platform provides feedback to the client on the data synchronization status and data archiving status of the target data, allowing relevant personnel to clearly understand the data archiving situation. Therefore, this embodiment can overcome the technical problem of low data archiving security in the prior art, and can improve data archiving efficiency while ensuring data archiving security.

[0043] In one exemplary embodiment, such as Figure 3 As shown, the data archiving process also includes:

[0044] Step S302: When the data archiving of the target data in the data warehouse is completed, the target data in the data warehouse is compared with the target data in the archive storage area to obtain the consistency comparison result.

[0045] It's important to note that while the completion of data archiving signifies the end of the archiving process, it doesn't guarantee that the target data in the data warehouse has been accurately archived to the archive storage area. Therefore, a consistency comparison is necessary after archiving is complete. This involves comparing the target data in the data warehouse with the target data in the archive storage area to determine if they are consistent. The consistency comparison result can be either consistent or inconsistent.

[0046] For example, when the archiving of target data in the data warehouse is complete, the server of the big data platform can perform a consistency comparison between the target data in the data warehouse and the target data in the archive storage area. Specifically, this can involve comparing whether the size of the target data in the data warehouse and the target data in the archive storage area are consistent; comparing whether the total amount of target data in the data warehouse and the total amount of target data in the archive storage area are consistent; comparing whether key fields of the target data in the data warehouse, such as amount and status, are consistent with key fields of the target data in the archive storage area; comparing whether the data table structure corresponding to the target data in the data warehouse and the data table structure corresponding to the target data in the archive storage area are consistent; and so on. Data sampling comparison can also be performed, that is, based on an appropriate sampling ratio, such as one in ten thousand, corresponding data is extracted from the target data in the data warehouse and the target data in the archive storage area respectively, and then compared whether this portion of data is consistent. Of course, in practical applications, other comparison rules can be set according to the actual situation, and this embodiment does not impose any limitations on this.

[0047] Optionally, a data consistency comparison should also be performed during data synchronization. That is, at the end of the synchronization of the target data to be archived, the server will compare the target data in the data warehouse with the initial target data. If the target data in the data warehouse matches the initial target data, it means that the target data to be archived has been accurately synchronized to the data warehouse, and the server can then continue with the data archiving process. If the target data in the data warehouse does not match the initial target data, it means that the target data to be archived has not been accurately synchronized to the data warehouse, and data synchronization should be re-performed. The server will then report the data synchronization failure to the client so that users can troubleshoot the problem promptly.

[0048] Step S304: If the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, determine that the target data in the data warehouse has been archived to the archive storage area, and update the data modification permission of the target data in the archive storage area to the closed state.

[0049] Among them, data modification permission refers to the permission to modify the data stored in the archive storage area, which may include permissions to update data, add data, and delete data.

[0050] For example, if the consistency comparison result indicates that the target data in the data warehouse matches the target data in the archive storage area, it means that the target data in the data warehouse has been accurately archived to the archive storage area. Since the server has been continuously writing target data to the archive storage area, the data modification permission in the archive storage area is enabled. After successful data archiving, the server will update the enabled state of the data modification permission to disabled, ensuring that external parties cannot modify the data in the archive storage area again, thereby guaranteeing the security of the archived data. If the consistency comparison result indicates that the target data in the data warehouse does not match the target data in the archive storage area, it means that the target data in the data warehouse has not been accurately archived to the archive storage area. In this case, data archiving can be re-processed, and the client will be notified of the data archiving failure so that users can troubleshoot the problem promptly.

[0051] In this embodiment, after data archiving is completed, a consistency comparison is performed between the target data in the data warehouse and the target data in the archive storage area to ensure the accuracy of data archiving. If the consistency comparison passes, data modification permissions in the archive storage area are disabled, thereby ensuring that the target data in the archive storage area cannot be modified, and thus ensuring the security of data archiving.

[0052] In one exemplary embodiment, such as Figure 4 As shown, the following are also included when data archiving fails:

[0053] Step S402: If the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, the retry mechanism is initiated.

[0054] The retry mechanism refers to the mechanism for re-archiving data.

[0055] For example, if the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, it means that the target data was not accurately archived to the archive storage area, and data archiving failed. In this case, the big data platform's server can initiate a retry mechanism, that is, re-archive the target data in the data warehouse. Similarly, when data synchronization fails, the server can also initiate a retry mechanism, that is, re-synchronize the target data to be archived.

[0056] Step S404: Perform data archiving processing on the target data in the data warehouse again according to the number of retries and the retry interval in the retry mechanism.

[0057] The number of retries refers to the number of times the data is re-archived. In this embodiment, to ensure the accuracy of the data re-archiving, the number of retries can be configured to at least two. Of course, in practical applications, if the data volume is relatively small, one retrieval may suffice. The retry interval refers to the interval between each re-archiving of the data.

[0058] For example, after the big data platform's server initiates the retry mechanism, it can re-archive the target data in the data warehouse according to the number of retries and retry intervals set in the retry mechanism. Each retrieval can be spaced a certain amount of time apart. Similarly, when data synchronization fails, the big data platform's server can also re-synchronize the target data to be archived according to the number of retries and retry intervals in the retry mechanism.

[0059] Optionally, if the data synchronization or archiving attempt fails multiple times, the server can activate an alert mechanism, sending an error message to relevant personnel via SMS, telephone, email, or other instant messaging tools.

[0060] In this embodiment, a retry mechanism is initiated in the event of data archiving failure. The target data in the data warehouse is then re-archived according to the number of retries and retry intervals specified in the retry mechanism. This reduces the likelihood of data archiving failures due to network fluctuations or anomalies, thereby improving the stability and reliability of data archiving.

[0061] In one exemplary embodiment, such as Figure 5 As shown, after the data is successfully archived, the following also applies:

[0062] Step S502: If the target data in the data warehouse has been archived to the archive storage area, determine the source database of the target data to be archived.

[0063] Step S504: During off-peak periods, delete the target data to be archived from the source database; off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

[0064] In this context, the source database refers to the original database containing the target data to be archived. The off-peak period is defined as the time when the access volume to the source database is below a threshold. Access volume refers to the number of times the source database is accessed, and the threshold is a pre-set standard value for the number of accesses to the source database. If the access volume exceeds this threshold, the source database is considered to have excessive access volume and overload, making it unsuitable for deleting the target data. Conversely, if the access volume does not exceed the threshold, the source database is considered to have low access volume and low load, making it suitable for deleting the target data. In a preferred example, the off-peak period can be set from midnight to 6:00 AM to avoid excessive access volume putting pressure on the source database and thus reducing its impact.

[0065] For example, if the target data in the data warehouse has been archived to the archive storage area, meaning both data synchronization and archiving have been successfully completed, then the target data in the source database needs to be deleted to free up resources. Furthermore, to avoid putting pressure on the source database due to excessive access volume, this embodiment sets a suitable off-peak period and deletes the target data to be archived from the source database during this period.

[0066] In one exemplary embodiment, such as Figure 6 As shown, archiving target data in the data warehouse to the archive storage area includes:

[0067] Step S602: Determine the archive storage area that matches the data warehouse.

[0068] For example, when a big data platform archives data, it first needs to determine the archiving storage area, which corresponds to the data warehouse. The archiving storage area can be queried using the data warehouse's name. For instance, if an archiving storage area's name matches the data warehouse's name, and the name includes an archiving identifier such as arcHive, it indicates that the archiving storage area stores data from the data warehouse; that is, it is considered a region that matches the data warehouse. Of course, other archiving identifiers can be defined in practical applications.

[0069] Optionally, the archive storage area can be created at the beginning, which can improve the efficiency of data archiving. Alternatively, it can be created in real-time during the data archiving process, allowing the archive storage area to better reflect the actual situation.

[0070] Step S604: Copy the target data from the data warehouse to the archive storage area.

[0071] For example, the data archiving process is essentially a data copying or duplication process, which involves copying the target data from the data warehouse to the archive storage area without changing the data format or content. A big data platform can copy all the target data to the archive storage area at once, or it can copy it in batches.

[0072] In this embodiment, when archiving data, the archiving storage area that matches the data warehouse is first determined, and then the target data in the data warehouse is copied to the archiving storage area. This ensures the accuracy of the archiving storage area, thereby ensuring the accuracy of data archiving.

[0073] In this embodiment, when the target data in the data warehouse has been archived to the archive storage area, by deleting the target data to be archived from the source database during off-peak periods, the resources of the source database can be released while reducing the impact on the source database.

[0074] In one exemplary embodiment, such as Figure 7 As shown, the process of creating a data warehouse and archive storage area also includes:

[0075] Step S702: For the source database of the target data, obtain the data partitioning identifier of the source database.

[0076] Step S704: Create a data warehouse that matches the source database based on the data partition identifier, and create an archive storage area that matches the data warehouse based on the data partition identifier.

[0077] The data partition identifier can also be understood as the data partition identifier, which is the identifier for dividing or partitioning data in the source database. For example, the source database may divide data by month or by day.

[0078] For example, when creating a data warehouse and archive storage area, the server of a big data platform first obtains the data partition identifier of the source database of the target data and creates a corresponding data warehouse within the big data platform. Thus, the data warehouse and the source database share the same data partition identifier. Similarly, the server first creates an independent data area within the big data platform, such as an independent file directory. This file directory has the same data partition identifier as the data warehouse, but it will also have an additional arcHive identifier to indicate that it is used to store data from the data warehouse.

[0079] Optionally, the server can also obtain information such as the source database's database name, table structure, and table names, and combine this with data partitioning identifiers to create a data warehouse corresponding to this information. Then, it creates an archive storage area corresponding to this data warehouse. This further ensures the consistency of configuration information among the source database, data warehouse, and archive storage area.

[0080] In this embodiment, a corresponding data warehouse is created based on the data partitioning identifier of the source database, and then an archive storage area corresponding to the data warehouse is created, thereby ensuring consistency between the source database, the data warehouse, and the archive storage area. This ensures the accuracy of data synchronization between the source database and the data warehouse, as well as the accuracy of data archiving between the data warehouse and the archive storage area.

[0081] In one exemplary embodiment, such as Figure 8 As shown, a data archiving method is provided, which can be applied to... Figure 1 Taking the client 102 of the big data platform as an example, the explanation includes:

[0082] Step S802: In response to the data archiving event, determine the target data to be archived as indicated by the data archiving event.

[0083] Among them, data archiving events can also be understood as data archiving instructions, which are used to instruct the target data to be archived to be archived.

[0084] For example, a user can trigger a data archiving event for the target data to be archived through controls or buttons on the client interface of the big data platform. After receiving the data archiving event, the client can extract the target data to be archived from the data archiving event.

[0085] Step S804: Send the target data to the big data platform and display the data synchronization status of the target data in the data warehouse of the big data platform.

[0086] The data synchronization status is used to characterize the data synchronization status of the target data, and can include data synchronization in progress, data synchronization completed, data synchronization completed, and data synchronization failed.

[0087] For example, after obtaining the target data, the client can send it to the server of the big data platform for data synchronization, that is, to synchronize the target data to the data warehouse of the big data platform. During this process, the client will receive and display the data synchronization status feedback from the server in real time, so that users can clearly understand the synchronization status of the target data.

[0088] Step S806: If the data synchronization status indicates that the target data has been synchronized to the data warehouse, display the data archiving status of the target data in the archive storage area of ​​the big data platform.

[0089] The data archiving status describes the archiving progress of the target data and can include states such as data archiving has begun, data archiving is in progress, data archiving has ended, data archiving has been completed, and data archiving has failed. Data modification permissions for target data already archived to the archive storage area are disabled.

[0090] For example, if the data synchronization status indicates that the target data has been synchronized to the data warehouse, it means that the data synchronization was successful. At this point, the server will then archive the target data in the data warehouse to the archive storage area and provide real-time feedback to the client on the data archiving status in the archive storage area of ​​the big data platform. After receiving the data archiving status, the client can display it in real time, allowing users to clearly understand the archiving status of the target data.

[0091] Optionally, in addition to data synchronization status and data archiving status, the client can also display process data such as the start time of data synchronization / archiving, the end time of data synchronization / archiving, the duration of data synchronization / archiving, and data synchronization / archiving logs for the target data.

[0092] This embodiment is applied to the client of a big data platform. During the entire data synchronization and data archiving process, the client can display the data synchronization status and data archiving status of the target data in real time, so that relevant personnel can clearly understand the data archiving status.

[0093] Figure 9This diagram illustrates a data archiving process in a specific embodiment. It shows that the big data platform can pre-create a Hive database (data warehouse) corresponding to the source database and an archiving storage area. The source database and Hive database have the same database name, and the table names and structures in the source database are identical to those in the Hive database. When the big data platform receives a data archiving event for target data to be archived in the source database, it first performs data synchronization, that is, synchronizes the target data to the Hive database. After synchronization, a data comparison is performed to verify whether the target data is consistent with the data in the Hive database. After the data comparison passes, the big data platform then performs data archiving, that is, copies the data from the Hive database to the archiving storage area. Similarly, after archiving, a data comparison is also required to verify whether the data in the archiving storage area is consistent with the data in the Hive database. After the data comparison passes, data modification permissions in the archiving storage area are disabled. The target data in the source database is then deleted during off-peak hours. The big data platform client displays the data synchronization and archiving status of the target data in real time, allowing relevant personnel to understand the data synchronization and archiving status promptly.

[0094] This embodiment applies to a big data platform, which is used to store, manage, and analyze massive amounts of data. Compared to existing technologies that use databases to store data, big data platforms overcome the limitations of databases, such as limitations on the amount of data that can be stored, limitations on data structure, and limitations on storage space, thereby effectively improving the efficiency of data storage. When data archiving is required, the big data platform first obtains the target data to be archived and synchronizes it to the data warehouse. Thus, a reliable synchronized copy of the target data exists within the big data platform. Considering that the data warehouse can still be modified through specific instructions, the big data platform performs further data archiving processing on the target data in the data warehouse after successful synchronization. This involves archiving the target data in the data warehouse to a pre-created archiving storage area. After successful archiving, the data modification permissions for the archiving storage area are disabled, ensuring that the archiving storage area cannot be modified again, thereby ensuring the security of the target data. Throughout the process, the big data platform provides feedback to the client on the data synchronization and archiving status of the target data, allowing relevant personnel to clearly understand the data archiving situation. Therefore, this embodiment can overcome the technical problem of low data archiving security in the prior art, and can improve the efficiency of data archiving while ensuring data archiving security.

[0095] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0096] Based on the same inventive concept, this application also provides a data archiving apparatus for implementing the data archiving method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data archiving apparatus embodiments provided below can be found in the limitations of the data archiving method described above, and will not be repeated here.

[0097] In one exemplary embodiment, such as Figure 10 As shown, a data archiving device is provided for use in a big data platform, including: an archive data acquisition module 1002 for acquiring target data to be archived; a data synchronization module 1004 for providing feedback on the data synchronization status of the target data to the client during the process of synchronizing the target data to the data warehouse; a data archiving module 1006 for archiving the target data in the data warehouse to the archive storage area after the target data has been synchronized to the data warehouse, and providing feedback on the data archiving status of the target data in the archive storage area to the client; a data synchronization status for instructing the client to display the data archiving status of the target data after it has been synchronized to the data warehouse; and the data modification permission for the target data already archived to the archive storage area is in a closed state.

[0098] In an exemplary embodiment, the apparatus is further configured to: when the data archiving of target data in the data warehouse is completed, perform a consistency comparison between the target data in the data warehouse and the target data in the archive storage area to obtain a consistency comparison result; if the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, determine that the target data in the data warehouse has been archived to the archive storage area, and update the data modification permission of the archive storage area to the closed state.

[0099] In an exemplary embodiment, the data archiving module 1006 is further configured to: determine an archiving storage area that matches the data warehouse; and copy the target data from the data warehouse to the archiving storage area.

[0100] In an exemplary embodiment, the apparatus is further configured to: determine the source database of the target data to be archived when the target data in the data warehouse has been archived to the archive storage area; and delete the target data to be archived from the source database during off-peak periods; the off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

[0101] In an exemplary embodiment, the apparatus is further configured to: obtain a data partition identifier for a source database of target data; create a data warehouse matching the source database based on the data partition identifier; and create an archive storage area matching the data warehouse based on the data partition identifier.

[0102] In an exemplary embodiment, the apparatus is further configured to: if the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, initiate a retry mechanism; and perform data archiving processing on the target data in the data warehouse again in the archive storage area according to the number of retries and the retry interval in the retry mechanism.

[0103] In one exemplary embodiment, such as Figure 11 As shown, a data archiving device is provided for a client application, including: a data determination module 1102, used to determine the target data to be archived as indicated by a data archiving event in response to the data archiving event; a data synchronization status display module 1104, used to send the target data to a big data platform and display the data synchronization status of the target data in the data warehouse of the big data platform; and a data archiving status display module 1106, used to display the data archiving status of the target data in the archiving storage area of ​​the big data platform when the data synchronization status indicates that the target data has been synchronized to the data warehouse; the data modification permission of the target data that has been archived to the archiving storage area is in a closed state.

[0104] Each module in the aforementioned data archiving device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0105] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores archived data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data archiving method.

[0106] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0107] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to a client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data archived to the archive storage area is disabled.

[0108] In one embodiment, when the processor executes the computer program, it further performs the following steps: when the data archiving of the target data in the data warehouse is completed, the target data in the data warehouse is compared with the target data in the archive storage area to obtain a consistency comparison result; if the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, it is determined that the target data in the data warehouse has been archived to the archive storage area, and the data modification permission of the archive storage area is updated to the closed state.

[0109] In one embodiment, the processor, when executing a computer program, also performs the following steps: determining an archive storage area that matches the data warehouse; and copying the target data from the data warehouse to the archive storage area.

[0110] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the target data in the data warehouse has been archived to the archive storage area, determine the source database of the target data to be archived; delete the target data to be archived from the source database during off-peak periods; the off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

[0111] In one embodiment, when the processor executes the computer program, it further performs the following steps: for the source database of the target data, obtain the data partition identifier of the source database; create a data warehouse that matches the source database based on the data partition identifier; and create an archive storage area that matches the data warehouse based on the data partition identifier.

[0112] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, a retry mechanism is initiated; according to the number of retries and the retry interval in the retry mechanism, the target data in the data warehouse is archived again in the archive storage area.

[0113] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: in response to a data archiving event, determining the target data to be archived as indicated by the data archiving event; sending the target data to a big data platform and displaying the data synchronization status of the target data in the data warehouse of the big data platform; if the data synchronization status indicates that the target data has been synchronized to the data warehouse, displaying the data archiving status of the target data in the archive storage area of ​​the big data platform; and the data modification permission of the target data archived to the archive storage area is in a closed state.

[0114] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When executed by a processor, the computer program performs the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to a client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data archived to the archive storage area is disabled.

[0115] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the data archiving of the target data in the data warehouse is completed, the target data in the data warehouse is compared with the target data in the archive storage area to obtain a consistency comparison result; if the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, it is determined that the target data in the data warehouse has been archived to the archive storage area, and the data modification permission of the archive storage area is updated to the closed state.

[0116] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining an archive storage area that matches the data warehouse; and copying the target data from the data warehouse to the archive storage area.

[0117] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the target data in the data warehouse has been archived to the archive storage area, determine the source database of the target data to be archived; during off-peak periods, delete the target data to be archived from the source database; the off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

[0118] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the source database of the target data, obtain the data partition identifier of the source database; create a data warehouse that matches the source database based on the data partition identifier; and create an archive storage area that matches the data warehouse based on the data partition identifier.

[0119] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, a retry mechanism is initiated; according to the number of retries and the retry interval in the retry mechanism, the target data in the data warehouse is archived again in the archive storage area.

[0120] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: in response to a data archiving event, determining the target data to be archived as indicated by the data archiving event; sending the target data to a big data platform and displaying the data synchronization status of the target data in the data warehouse of the big data platform; if the data synchronization status indicates that the target data has been synchronized to the data warehouse, displaying the data archiving status of the target data in the archive storage area of ​​the big data platform; and the data modification permission for the target data already archived to the archive storage area is closed.

[0121] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring target data to be archived; during the process of synchronizing the target data to a data warehouse, providing feedback on the data synchronization status of the target data to a client; when the target data has been synchronized to the data warehouse, archiving the target data in the data warehouse to an archive storage area, and providing feedback on the data archiving status of the target data in the archive storage area to the client; instructing the client to display the data archiving status of the target data when the target data has been synchronized to the data warehouse; and ensuring that the data modification permission for the target data already archived to the archive storage area is disabled.

[0122] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the data archiving of the target data in the data warehouse is completed, the target data in the data warehouse is compared with the target data in the archive storage area to obtain a consistency comparison result; if the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, it is determined that the target data in the data warehouse has been archived to the archive storage area, and the data modification permission of the archive storage area is updated to the closed state.

[0123] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining an archive storage area that matches the data warehouse; and copying the target data from the data warehouse to the archive storage area.

[0124] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the target data in the data warehouse has been archived to the archive storage area, determine the source database of the target data to be archived; during off-peak periods, delete the target data to be archived from the source database; the off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

[0125] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the source database of the target data, obtain the data partition identifier of the source database; create a data warehouse that matches the source database based on the data partition identifier; and create an archive storage area that matches the data warehouse based on the data partition identifier.

[0126] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, a retry mechanism is initiated; according to the number of retries and the retry interval in the retry mechanism, the target data in the data warehouse is archived again in the archive storage area.

[0127] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: in response to a data archiving event, determining the target data to be archived as indicated by the data archiving event; sending the target data to a big data platform and displaying the data synchronization status of the target data in the data warehouse of the big data platform; if the data synchronization status indicates that the target data has been synchronized to the data warehouse, displaying the data archiving status of the target data in the archive storage area of ​​the big data platform; and ensuring that the data modification permission for the target data already archived to the archive storage area is closed.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0130] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this application. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data archiving method, characterized in that, Applied to a big data platform, which includes a data warehouse and an archive storage area, the method includes: Retrieve the target data to be archived; During the process of synchronizing the target data to the data warehouse, the data synchronization status of the target data is fed back to the client; Once the target data has been synchronized to the data warehouse, the target data in the data warehouse is archived to the archive storage area, and the data archiving status of the target data in the archive storage area is fed back to the client. The data synchronization status is used to indicate to the client that, when the target data has been synchronized to the data warehouse, the data archiving status of the target data is displayed; the data modification permission of the target data that has been archived to the archive storage area is in a closed state.

2. The method according to claim 1, characterized in that, The method further includes: When the data archiving process of the target data in the data warehouse is completed, the target data in the data warehouse is compared with the target data in the archive storage area to obtain the consistency comparison result. If the consistency comparison result indicates that the target data in the data warehouse is consistent with the target data in the archive storage area, it is determined that the target data in the data warehouse has been archived to the archive storage area, and the data modification permission of the target data in the archive storage area is updated to the closed state.

3. The method according to claim 2, characterized in that, The method further includes: If the consistency comparison result indicates that the target data in the data warehouse is inconsistent with the target data in the archive storage area, a retry mechanism is initiated. According to the number of retries and the retry interval in the retry mechanism, the target data in the data warehouse is archived again.

4. The method according to claim 2, characterized in that, The method further includes: If the target data in the data warehouse has been archived to the archive storage area, determine the source database of the target data to be archived; During off-peak periods, the target data to be archived is deleted from the source database; the off-peak periods are the periods when the access volume of the source database is lower than the access volume threshold.

5. The method according to claim 1, characterized in that, The step of archiving the target data in the data warehouse to the archive storage area includes: Determine the archive storage area that matches the data warehouse; Copy the target data from the data warehouse to the archive storage area.

6. The method according to claim 1, characterized in that, The method further includes: For the source database of the target data, obtain the data partitioning identifier of the source database; A data warehouse matching the source database is created based on the data partition identifier, and an archive storage area matching the data warehouse is created based on the data partition identifier.

7. A data archiving method, characterized in that, Applied to a client, the method includes: In response to a data archiving event, determine the target data to be archived as indicated by the data archiving event; Send the target data to the big data platform and display the data synchronization status of the target data in the data warehouse of the big data platform; When the data synchronization status indicates that the target data has been synchronized to the data warehouse, the data archiving status of the target data in the archive storage area of ​​the big data platform is displayed; the data modification permission of the target data that has been archived to the archive storage area is closed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images