Data archiving method, apparatus, computer device, storage medium and program product

Abstract

The application relates to a data archiving method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining first target business data to be archived of a plurality of users; sorting the first target business data to be archived according to a business type in the first target business data to be archived and business data corresponding to the plurality of users under the business type, obtaining second target business data to be archived and an archiving amount corresponding to the second target business data to be archived; calling a main process to group the second target business data to be archived according to the archiving amount and the number of work processes, obtaining at least one grouped business data; obtaining an archiving result of the at least one grouped business data from at least one first work process based on the main process; and obtaining an archiving result of the first target business data to be archived according to the archiving result of the at least one grouped business data. The method can improve archiving efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data archiving method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] As the business continues to expand, the storage of existing business data has shifted from paper data to electronic data. By archiving business data, we can ensure the privacy of the business data, and by analyzing the archived business data, we can better analyze users' business needs.

[0003] Typically, when archiving business data, staff need to break it down into smaller parts based on the specific content of the data, which makes archiving inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a data archiving method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve archiving efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data archiving method. The method includes:

[0006] Retrieve the primary target business data to be archived from multiple users;

[0007] Based on the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, the first target business data to be archived is sorted to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived;

[0008] Based on the amount of data to be archived and the number of working processes, the main process is invoked to group the second target business data to be archived, and at least one group of business data is obtained.

[0009] The main process obtains the archiving results of the at least one group service data from at least one first worker process; the number of the at least one first worker process is the number of worker processes.

[0010] Based on the archiving results of the at least one group of business data, the archiving results of the first target business data to be archived are obtained.

[0011] In one embodiment, the step of sorting the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived, includes:

[0012] Based on the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, sort the multiple target business data in the first target business data to be archived to obtain the sorted multiple target business data;

[0013] Determine whether the sorted multiple target service data entries contain multiple service data entries for the same user;

[0014] If they exist, based on the multiple business data of the same user among the sorted multiple target business data, obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived.

[0015] In one embodiment, obtaining the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived based on multiple business data of the same user from the sorted multiple target business data includes:

[0016] In the case of merging multiple business data of the same user into one business data, the second target business data to be archived is obtained;

[0017] Based on the row number set in the second target business data to be archived, the archiving quantity corresponding to the second target business data to be archived is obtained.

[0018] In one embodiment, the at least one first working process includes multiple processes; obtaining the archiving result of the at least one group service data from the at least one first working process based on the master process includes:

[0019] When at least one first working process allocates the corresponding group service data to each of the processes for parallel processing based on the number of the plurality of processes to obtain the corresponding archive package, a response message is sent to the main process based on the at least one first working process; the response message is used to indicate that the at least one first working process has completed processing the corresponding group service data, and the response message includes the name and storage path of the corresponding archive package.

[0020] When the main process receives the corresponding response message from the at least one first worker process, it extracts the archive package whose name matches the corresponding archive package from the storage path of the archive package corresponding to the at least one first worker process, and obtains the archiving result of the at least one group service data.

[0021] In one embodiment, obtaining the archiving result of the first target business data to be archived based on the archiving result of the at least one group of business data includes:

[0022] Get the pre-set number of merges;

[0023] Based on the number of merges, the archive package corresponding to the at least one first working process is compressed based on the main process to obtain the archiving result of the first target business data to be archived.

[0024] In one embodiment, the method further includes:

[0025] If the main process detects a failure in the first working process, it reallocates the packet service data corresponding to the failed first working process to the second working process; the second working process is a process other than the at least one first working process or a process in the at least one first working process that has already processed the corresponding packet service data.

[0026] Secondly, this application also provides a data archiving apparatus, the apparatus comprising:

[0027] The data acquisition module is used to acquire the primary target business data to be archived from multiple users;

[0028] The data processing module is used to sort the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, so as to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived;

[0029] The grouping module is used to call the main process to group the second target business data to be archived according to the amount of data to be archived and the number of working processes, so as to obtain at least one group of business data.

[0030] The group archiving result acquisition module is used to obtain the archiving result of the at least one group business data from at least one first working process based on the main process; the number of the at least one first working process is the number of working processes.

[0031] The archiving result integration module is used to obtain the archiving result of the first target business data to be archived based on the archiving result of the at least one group of business data.

[0032] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] Retrieve the primary target business data to be archived from multiple users;

[0034] Based on the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, the first target business data to be archived is sorted to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived;

[0035] Based on the amount of data to be archived and the number of working processes, the main process is invoked to group the second target business data to be archived, and at least one group of business data is obtained.

[0036] The main process obtains the archiving results of the at least one group service data from at least one first worker process; the number of the at least one first worker process is the number of worker processes.

[0037] Based on the archiving results of the at least one group of business data, the archiving results of the first target business data to be archived are obtained.

[0038] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0039] Retrieve the primary target business data to be archived from multiple users;

[0040] Based on the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, the first target business data to be archived is sorted to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived;

[0041] Based on the amount of data to be archived and the number of working processes, the main process is invoked to group the second target business data to be archived, and at least one group of business data is obtained.

[0042] The main process obtains the archiving results of the at least one group service data from at least one first worker process; the number of the at least one first worker process is the number of worker processes.

[0043] Based on the archiving results of the at least one group of business data, the archiving results of the first target business data to be archived are obtained.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Retrieve the primary target business data to be archived from multiple users;

[0046] Based on the business type in the first target business data to be archived and the business data corresponding to the multiple users under the business type, the first target business data to be archived is sorted to obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived;

[0047] Based on the amount of data to be archived and the number of working processes, the main process is invoked to group the second target business data to be archived, and at least one group of business data is obtained.

[0048] The main process obtains the archiving results of the at least one group service data from at least one first worker process; the number of the at least one first worker process is the number of worker processes.

[0049] Based on the archiving results of the at least one group of business data, the archiving results of the first target business data to be archived are obtained.

[0050] The aforementioned data archiving method, apparatus, computer equipment, storage medium, and computer program product acquire first target business data to be archived from multiple users. Based on the business type within the first target business data and the corresponding business data for multiple users under each business type, the first target business data to be archived is sorted to obtain second target business data to be archived and the corresponding archiving quantity. Then, based on the archiving quantity and the number of working processes corresponding to the second target business data to be archived, the main process is invoked to group the second target business data to be archived. At least one group of business data archiving results is obtained from at least one first working process, and the archiving result of the first target business data to be archived is obtained based on the archiving results of at least one group of business data. Thus, when acquiring the business data to be archived, the main process is invoked, enabling parallel data processing between working processes to obtain archiving results. The data archiving process requires no human intervention, reducing workload and improving archiving efficiency. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the application environment of a data archiving method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a data archiving method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the process of sorting the first target business data to be archived and the business data corresponding to multiple users under the business type in the first target business data to be archived, to obtain the second target business data to be archived and the archived amount corresponding to the second target business data to be archived in one embodiment.

[0054] Figure 4 This is a flowchart illustrating the process of obtaining the second target business data to be archived and the corresponding archived quantity of the second target business data based on multiple target business data of the same user from multiple sorted target business data in one embodiment.

[0055] Figure 5 This is a flowchart illustrating the process of obtaining the archiving results of at least one group of business data from at least one first worker process based on the main process in one embodiment.

[0056] Figure 6 This is a flowchart illustrating the process of obtaining the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data in one embodiment.

[0057] Figure 7 This is a structural block diagram of a data archiving device in one embodiment;

[0058] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The data archiving method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the data archiving system 100 includes a server 102, a main process 104, and at least one first worker process 106. Specifically, the server 102 obtains first target business data to be archived from multiple users, sorts the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to multiple users under the business type, and obtains second target business data to be archived and the archive quantity corresponding to the second target business data to be archived; and calls the main process 104 to group the second target business data to be archived according to the archive quantity and the number of worker processes, and obtains at least one group of business data; and obtains the archiving result of at least one group of business data from at least one first worker process 106 based on the archiving result of the at least one group of business data obtained by the main process 104, so as to obtain the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data.

[0061] In one embodiment, such as Figure 2 As shown, a data archiving method is provided, which can be applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps:

[0062] S202, Obtain the first target business data to be archived from multiple users.

[0063] In some embodiments, obtaining the first target business data to be archived from multiple users includes: receiving business source tables from multiple users of at least one business system; the business source tables refer to the basic table data collected by at least one business system when multiple users perform corresponding business operations; using Spark SQL and commonly used Spark operators, processing the business source tables of multiple users into corresponding business archive data; processing the corresponding business archive data into corresponding archive metadata according to the archive type; associating the corresponding business archive and the corresponding archive metadata to obtain the business data of each user; and obtaining the first target business data to be archived from multiple users based on the business data of multiple users.

[0064] Here, "archive type" refers to the type of business archive data. For example, archive types include contracts, agreements, or other types. The specific type can be set according to the actual application scenario, and this embodiment does not limit it.

[0065] In some embodiments, business profile data for different users can be stored in different databases, or, to facilitate the extraction and processing of business profile data, business profile data for different users can be stored in the same database.

[0066] Furthermore, after obtaining the first target business data to be archived, the data can be stored in multiple partition tables in Hive, each with a corresponding partition name. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop.

[0067] In some embodiments, an archiving period can be set, and the first target business data to be archived can be obtained by obtaining the business data of multiple users within the archiving period; wherein, the archiving period can be one day, two days, one week, one month or other values; for example, if the archiving period is one day, the first target business data to be archived of multiple users obtained from at least one business system each day can be archived.

[0068] S204. Based on the business type and the business data corresponding to multiple users under the business type in the first target business data to be archived, sort the first target business data to be archived to obtain the second target business data to be archived and the archived quantity corresponding to the second target business data to be archived.

[0069] In this context, the archive quantity corresponding to the second target business data to be archived is used to represent the data volume of the second target business data to be archived. In some embodiments, when the server obtains the archive quantity corresponding to the second target business data to be archived, it can store the archive quantity information and the second target business data to be archived in the same database and send an instruction message to the main process. The instruction message is used to instruct the main process to group the second target business data to be archived. The instruction message includes the location and name of the database storing the second target business data to be archived and the archive quantity information. The information to be archived includes the archive quantity.

[0070] S206, based on the amount of data to be archived and the number of working processes, call the main process to group the second target business data to be archived, and obtain at least one group of business data.

[0071] In some embodiments, in conjunction with S204, when the main process receives an instruction message from the server, the instruction message includes the location and name of the database storing the second target business data to be archived and the archive quantity information. Therefore, the main process can obtain the second target business data to be archived and the archive quantity corresponding to the second target business data from the database based on the location and name of the database. Then, the main process groups the second target business data to be archived according to the archive quantity and the number of working processes to obtain at least one group of business data.

[0072] One example is that if the amount of data (i.e. the amount of data to be archived) of the second target business data is 100 and the number of worker processes is 4, then the main process can divide the second target business data to be archived into 4 groups of business data, and the amount of data to be archived for each group of business data is 25.

[0073] S208, based on the main process obtaining the archiving result of at least one group of business data from at least one first working process.

[0074] Wherein, the number of at least one first working process is the number of working processes described in S206; in conjunction with the above example, the main process can allocate the four grouped business data to the four first working processes respectively, and each first working process processes the corresponding grouped business data; thus, the main process can obtain the archiving results of the four grouped business data from the four first working processes.

[0075] It should be noted that when the main process groups the second target business data to be archived according to the amount of archived data and the number of working processes, and allocates at least one group of business data to at least one first working process, the slow execution of the data processing process due to resource competition between the first working processes can be avoided. This is because, in this application, since the second target business data to be archived is grouped in advance, the business data processed by each first working process is determined, thereby avoiding the competition problem between the first working processes.

[0076] S210, based on the archiving results of at least one group of business data, obtain the archiving results of the first target business data to be archived.

[0077] Specifically, by compressing the archiving results of at least one group of business data, the archiving results of the first target business data to be archived can be obtained, and the archiving results of the first target business data to be archived can be stored in the archiving database for easy viewing and querying.

[0078] In summary, based on Figure 2 The method described involves acquiring first-target business data to be archived from multiple users, sorting the first-target business data according to its business type and the corresponding business data for multiple users within that business type, and obtaining second-target business data to be archived and the corresponding archiving quantity for that second-target business data. Based on the archiving quantity and the number of working processes, the second-target business data can be grouped by calling the main process, and at least one group of business data can be archived from at least one first working process. The archiving result of the first-target business data can then be obtained based on the archiving result of at least one group of business data. Thus, by calling the main process when acquiring the business data to be archived, archiving results can be obtained through parallel data processing between working processes. The data archiving process requires no human intervention, reducing workload and improving archiving efficiency. Furthermore, it enables the processing of massive amounts of data, providing a flexible and stable archiving capability for electronic documents.

[0079] In one embodiment, such as Figure 3 The diagram illustrates a process for sorting the first target business data to be archived based on the business type and the business data corresponding to multiple users under that business type, to obtain the second target business data to be archived and the corresponding archive volume. This method is then applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps:

[0080] S302, based on the business type and the business data corresponding to multiple users under the business type in the first target business data to be archived, sort the multiple target business data in the first target business data to be archived to obtain the sorted multiple target business data.

[0081] In this process, the business data of the same user is grouped together in the sorted target business data, and the business data of different users of the same business type are also grouped together. After exporting the sorted target business data, it is easy to query the business data of the corresponding user from the sorted target business data, thus improving query efficiency. On the other hand, based on the sorted target business data, it is easy to determine the amount of second target business data to be archived. Based on the amount of archived data and the number of work processes, the amount of data processed by each first work process can be determined, avoiding the situation where the data processing of the same first work process fails due to the large amount of data being processed, thus reducing the data processing failure rate.

[0082] S304, determine whether multiple target business data entries after sorting exist for the same user.

[0083] The sorted target business data may include business data from multiple users, and each user's business data includes multiple types of business data; for example, a user may conduct deposit transactions, repayment transactions, and bill payment transactions in a day, then the user's business types include deposit, repayment, and bill payment.

[0084] Specifically, each target business data corresponds to a user identifier. Therefore, the user identifier can be used to determine whether multiple target business data exist for the same user.

[0085] S306, if it exists, based on the multiple business data of the same user in the sorted multiple target business data, obtain the second target business data to be archived and the archive quantity corresponding to the second target business data to be archived.

[0086] In some embodiments, by merging multiple pieces of business data of the same user in multiple pieces of target business data, second target business data to be archived can be obtained, and then the amount of archived data corresponding to the second target business data to be archived can be obtained based on the second target business data to be archived.

[0087] In summary, based on Figure 3The method described involves sorting multiple target business data items within the first target business data to be archived based on the business type and the business data corresponding to multiple users under that business type, thus obtaining sorted target business data items. It then determines whether the sorted target business data items contain multiple business data items from the same user. If so, it obtains the second target business data to be archived and the corresponding archiving quantity based on these multiple business data items from the same user within the sorted target business data items. Furthermore, based on the archiving quantity and the number of working processes, the second target business data to be archived is grouped, and at least one group of business data is archived from at least one first working process. The archiving result of the first target business data to be archived is then obtained based on the archiving result of at least one group of business data. Thus, when obtaining the business data to be archived, the main process is invoked, enabling parallel data processing between working processes to obtain the archiving result. The data archiving process does not require human intervention, reducing workload and improving archiving efficiency.

[0088] In one embodiment, such as Figure 4 The diagram illustrates a process for obtaining second target business data to be archived and the corresponding archive quantity from multiple business data of the same user within sorted target business data. This method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps:

[0089] S402, when merging multiple business data entries of the same user into one business data entry, obtain the second target business data to be archived.

[0090] S404: Obtain the archiving quantity corresponding to the second target business data to be archived based on the row number set in the second target business data to be archived.

[0091] In this embodiment, when setting row numbers for the second target business data to be archived, each row number corresponds to one piece of business data. The maximum value among the row numbers set for the second target business data to be archived can be determined as the archiving quantity corresponding to the second target business data to be archived. For example, if the row numbers set for the second target business data to be archived are sorted from 1 to 10, and row number 10 is the maximum row number, then 10 is the archiving quantity corresponding to the second target business data to be archived. Each row number corresponds to business data belonging to a different user, and the number of business data items is at least one. When the number of business data items corresponding to a row number is multiple, it can be determined that multiple pieces of business data from different users have been merged.

[0092] Furthermore, when determining the maximum value of the row number set in the second target business data to be archived as the archiving quantity corresponding to the second target business data to be archived, the second target business data to be archived can be grouped according to the row number (i.e., archiving quantity) and the number of working processes to obtain at least one group of business data, and the row number range corresponding to at least one group of business data is determined, so that the row number range corresponding to the business data processed by each first working process is determined.

[0093] One example is as follows: If the row numbers of the second target business data to be archived are sorted from 1 to 100, then the archive quantity corresponding to the second target business data to be archived is 100. With 4 worker processes, the main process can divide the second target business data to be archived into 4 groups, each group corresponding to an archive quantity of 25. Specifically, the first group corresponds to row numbers 1-25, the second group to row numbers 26-50, the third group to row numbers 51-75, and the fourth group to row numbers 76-100. Then, when the main process assigns the 4 groups to the 4 first worker processes, the first worker process processes the business data corresponding to row numbers 1-25, the second worker process processes the business data corresponding to row numbers 26-50, the third worker process processes the business data corresponding to row numbers 51-75, and the fourth worker process processes the business data corresponding to row numbers 76-100.

[0094] It is understandable that by pre-allocating the business data to be processed to each first working process, and the different line numbers of the business data allocated to different first working processes, the situation of resource contention between working processes is avoided, which effectively improves the success rate of data processing.

[0095] In one embodiment, at least one first working process includes multiple processes; specifically, such as Figure 5 The diagram illustrates a process where the main process obtains archiving results of at least one group of business data from at least one first worker process. This method is applied to… Figure 1 Taking server 102 as an example, the explanation includes the following steps:

[0096] S502, when at least one first working process allocates the corresponding grouped business data to each process for parallel processing according to the number of multiple processes and obtains the corresponding archive package, a response message is sent to the main process based on at least one first working process; the response message is used to indicate that at least one first working process has completed processing the corresponding grouped business data, and the response message includes the name and storage path of the corresponding archive package.

[0097] Specifically, in conjunction with S202, the grouped business data includes business archive data and archive metadata. When the first working process allocates the corresponding grouped business data to each process for parallel processing based on the number of multiple processes, each process can generate the corresponding archiving result based on the archive business fields in the business archive data and the archive metadata fields in the archive metadata in the allocated business data. At least one first working process can obtain the corresponding archive package based on the archive results obtained by multiple processes. The archive package can be an Electronic Records Encapsulation Package (EEP), which includes metadata files, explanatory files, and related PDF attachments.

[0098] Specifically, at least one first working process can allocate corresponding grouped business data to each process for parallel processing based on the number of multiple processes, thereby obtaining the corresponding archive package; specifically, at least one first working process can determine the amount of business data processed by each process based on the number of multiple processes; and then allocate the corresponding grouped business data to multiple processes based on the amount of business data processed by each process and the row number range corresponding to the allocated grouped business data, thereby obtaining the corresponding archive package.

[0099] For example, referring to the example shown in S404, if the grouped business data allocated to the first working process corresponds to the business data with row numbers 1-25, and the first working process includes 5 processes, then it can be determined that the amount of business data processed by each process is 5. The first process processes the business data with row numbers 1-5, the second process processes the business data with row numbers 6-10, the third process processes the business data with row numbers 11-15, the fourth process processes the business data with row numbers 16-20, and the fifth process processes the business data with row numbers 21-25. Moreover, these 5 processes process the data in parallel. In this way, the concurrent execution of processes can be guaranteed. At the same time, by allocating the business data to be processed to each process, the competition problem between processes can be avoided.

[0100] S504, when the main process receives a corresponding response message from at least one first working process, the main process extracts an archive package whose name matches the corresponding archive package from the storage path of the archive package corresponding to at least one first working process, and obtains the archiving result of at least one group of business data.

[0101] When at least one first working process finishes processing the corresponding group business data, it carries the name and storage path of the corresponding archive package in the response message, so that the main process can accurately obtain the archive package processed by at least one first working process, thereby improving the accuracy of data archiving.

[0102] In summary, based on Figure 5 The method shown allows the main process to know that the data allocated by at least one first worker process has been processed when at least one first worker process has completed processing the corresponding group business data. In this way, when the main process knows that the data of all first worker processes has been processed, it can promptly obtain the archiving result of the first target business data to be archived based on the archiving result of at least one group business data, thus improving data archiving efficiency. By carrying the name and storage path of the corresponding archive package in the response message, the main process can accurately obtain the archive package processed by at least one first worker process, thus improving the accuracy of data archiving.

[0103] Understandable. Figure 5 The method shown employs a distributed, highly available multi-process task communication group. Through communication between the master process and worker processes, it ensures signal transmission between task nodes, achieves parallel processing of business data, and improves data processing efficiency. Simultaneously, based on... Figure 5 The method shown improves the efficiency of metadata generation in the context of massive electronic document archiving, incorporates an automatic process scheduling balancing strategy (i.e., the main process pre-allocates business data to the first working process) and multiple processes (i.e., at least one first working process), and enhances the collaborative utilization rate among multiple processes.

[0104] In one embodiment, if the master process detects a failure in the first working process, it reallocates the packet service data corresponding to the failed first working process to the second working process. The second working process is either a process other than the first working process or a process within the first working process that has already processed the corresponding packet service data.

[0105] The data archiving system can be configured with a worker process monitoring process. When the main process determines that the first worker process has failed based on the worker process monitoring process, it can reduce the data processing failure rate by reallocating the grouped business data to the second worker process.

[0106] In some embodiments, a daemon process can also be set up in the data archiving system to monitor whether the main process is abnormal. If an abnormality is detected in the main process, the daemon process can restart the main process to prevent the cluster from crashing when the main process is unavailable, thus ensuring the safety and availability of the main process.

[0107] It is understandable that by setting up worker process listeners and daemon processes in a data archiving system, situations where the main process and worker processes fail to execute due to abnormalities can be avoided, thus ensuring that the main process and worker processes can achieve high-performance data processing while improving their throughput.

[0108] In one embodiment, such as Figure 6The diagram illustrates a process for obtaining the archiving result of a first target business data to be archived based on the archiving result of at least one group of business data. This method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps:

[0109] S602, retrieve the preset number of merges.

[0110] S604, based on the number of merges, compress the archive package corresponding to at least one first working process based on the main process to obtain the archiving result of the first target business data to be archived.

[0111] In some embodiments, the main process can compress at least one archive package corresponding to a first working process into an archive package with a preset file extension, based on the number of merges, and can store the compressed archive packages in batches in a preset database of the data archiving system.

[0112] For example, the pre-set merge count is 5, the number of archive packages obtained by the main process from at least one first worker process is 20, the archive packages are EEP packages, the main process can compress every 5 EEP packages into archive packages with the file extension "tar" (hereinafter referred to as tar packages), resulting in 4 tar packages, which are the archiving results of the first target business data to be archived.

[0113] In summary, based on Figure 6 The method shown reduces data storage volume and saves data storage resources by compressing the archive package corresponding to at least one first working process based on the master process according to the pre-set merging number.

[0114] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0115] Based on the same inventive concept, this application also provides a data archiving apparatus for implementing the data archiving method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data archiving apparatus embodiments provided below can be found in the limitations of the data archiving method described above, and will not be repeated here.

[0116] In one embodiment, such as Figure 7 As shown, a data archiving device is provided, including: a data acquisition module 702, a data processing module 704, a grouping module 706, a grouping archiving result acquisition module 708, and an archiving result integration module 710, wherein:

[0117] The data acquisition module 702 is used to acquire the first target business data to be archived from multiple users.

[0118] The data processing module 704 is used to sort the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to multiple users under the business type, and to obtain the second target business data to be archived and the archived quantity corresponding to the second target business data to be archived.

[0119] The grouping module 706 is used to call the main process to group the second target business data to be archived according to the amount of archived data and the number of working processes, so as to obtain at least one group of business data.

[0120] The group archiving result acquisition module 708 is used to obtain the archiving result of at least one group business data from at least one first working process based on the main process; the number of at least one first working process is the number of working processes.

[0121] The archiving result integration module 710 is used to obtain the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data.

[0122] In one embodiment, the data processing module 704 is further configured to: sort multiple target business data in the first target business data to be archived according to the business type and the business data corresponding to multiple users under the business type, and obtain sorted multiple target business data; determine whether there are multiple business data of the same user in the sorted multiple target business data; if so, obtain the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived according to the multiple business data of the same user in the sorted multiple target business data.

[0123] In one embodiment, the data processing module 704 is further configured to: obtain the second target business data to be archived when merging multiple business data of the same user into one business data; and obtain the archiving quantity corresponding to the second target business data to be archived according to the row number set in the second target business data to be archived.

[0124] In one embodiment, the group archiving result acquisition module 708 is further configured to: when at least one first working process allocates corresponding group business data to each process for parallel processing according to the number of multiple processes to obtain the corresponding archive package, send a response message to the main process based on at least one first working process; the response message is used to indicate that at least one first working process has completed processing the corresponding group business data, and the response message includes the name and storage path of the corresponding archive package; when the main process receives the corresponding response message from at least one first working process, extract the archive package whose name matches the corresponding archive package based on the storage path of the archive package corresponding to at least one first working process, and obtain the archiving result of at least one group business data.

[0125] In one embodiment, the archiving result integration module 710 is further configured to: obtain a pre-set number of merges; and, based on the number of merges, compress the archive package corresponding to at least one first working process based on the main process to obtain the archiving result of the first target business data to be archived.

[0126] In one embodiment, the group archiving result acquisition module 708 is further configured to: if the main process detects that the first working process has failed, reallocate the group service data corresponding to the failed first working process to the second working process based on the main process; the second working process is a process other than at least the first working process or a process in at least the first working process that has completed processing the corresponding group service data.

[0127] Each module in the aforementioned data archiving device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0128] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores primary target business data to be archived by multiple users. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a data archiving method.

[0129] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring first target business data to be archived from multiple users; sorting the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to multiple users under the business type, to obtain second target business data to be archived and the archive quantity corresponding to the second target business data to be archived; calling a main process to group the second target business data to be archived according to the archive quantity and the number of working processes, to obtain at least one group of business data; obtaining the archiving result of at least one group of business data from at least one first working process based on the main process; the number of at least one first working process is the number of working processes; and obtaining the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data.

[0131] In one embodiment, when the processor executes the computer program, it further performs the following steps: sorting multiple target business data in the first target business data to be archived according to the business type and the business data corresponding to multiple users under the business type, to obtain sorted multiple target business data; determining whether there are multiple business data of the same user in the sorted multiple target business data; if so, obtaining the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived according to the multiple business data of the same user in the sorted multiple target business data.

[0132] In one embodiment, when the processor executes the computer program, it further implements the following steps: in the case of merging multiple business data of the same user into one business data, obtaining the second target business data to be archived; and obtaining the archiving quantity corresponding to the second target business data to be archived according to the row number set in the second target business data to be archived.

[0133] In one embodiment, when the processor executes the computer program, it further implements the following steps: when at least one first working process allocates corresponding grouped business data to each process for parallel processing according to the number of multiple processes to obtain corresponding archive packages, a response message is sent to the main process based on at least one first working process; the response message is used to indicate that at least one first working process has completed processing the corresponding grouped business data, and the response message includes the name and storage path of the corresponding archive package; when the main process receives the corresponding response message from at least one first working process, the main process extracts the archive package whose name matches the corresponding archive package from the storage path of the archive package corresponding to at least one first working process, and obtains the archiving result of at least one grouped business data.

[0134] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a pre-set number of merges; and, based on the number of merges, compressing the archive package corresponding to at least one first working process based on the main process to obtain the archiving result of the first target business data to be archived.

[0135] In one embodiment, when the processor executes the computer program, it further implements the following steps: if the main process detects that the first working process has failed, the main process reallocates the packet service data corresponding to the failed first working process to the second working process; the second working process is a process other than at least the first working process or a process in at least the first working process that has completed processing the corresponding packet service data.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: acquiring first target business data to be archived from multiple users; sorting the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to multiple users under the business type, to obtain second target business data to be archived and the archive quantity corresponding to the second target business data to be archived; calling a main process to group the second target business data to be archived according to the archive quantity and the number of working processes, to obtain at least one group of business data; obtaining the archiving result of at least one group of business data from at least one first working process based on the main process; the number of at least one first working process is the number of working processes; and obtaining the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data.

[0137] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sorting multiple target business data in the first target business data to be archived according to the business type and the business data corresponding to multiple users under the business type, to obtain sorted multiple target business data; determining whether there are multiple business data of the same user in the sorted multiple target business data; if so, obtaining the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived according to the multiple business data of the same user in the sorted multiple target business data.

[0138] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in the case of merging multiple business data of the same user into one business data, obtaining the second target business data to be archived; and obtaining the archiving quantity corresponding to the second target business data to be archived according to the row number set in the second target business data to be archived.

[0139] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when at least one first working process allocates corresponding grouped business data to each process for parallel processing according to the number of multiple processes to obtain corresponding archive packages, a response message is sent to the main process based on at least one first working process; the response message is used to indicate that at least one first working process has completed processing the corresponding grouped business data, and the response message includes the name and storage path of the corresponding archive package; when the main process receives the corresponding response message from at least one first working process, the main process extracts the archive package whose name matches the corresponding archive package from the storage path of the archive package corresponding to at least one first working process, and obtains the archiving result of at least one grouped business data.

[0140] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a pre-set number of merges; and, based on the number of merges, compressing the archive package corresponding to at least one first working process based on the main process to obtain the archiving result of the first target business data to be archived.

[0141] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the main process detects that the first working process has failed, the main process reallocates the packet service data corresponding to the failed first working process to the second working process; the second working process is at least one process other than the first working process or at least one process in the first working process that has completed processing the corresponding packet service data.

[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring first target business data to be archived from multiple users; sorting the first target business data to be archived according to the business type in the first target business data to be archived and the business data corresponding to multiple users under the business type, to obtain second target business data to be archived and the archive quantity corresponding to the second target business data to be archived; calling a main process to group the second target business data to be archived according to the archive quantity and the number of worker processes, to obtain at least one group of business data; obtaining the archiving result of at least one group of business data from at least one first worker process based on the main process; the number of at least one first worker process is the number of worker processes; and obtaining the archiving result of the first target business data to be archived based on the archiving result of at least one group of business data.

[0143] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sorting multiple target business data in the first target business data to be archived according to the business type and the business data corresponding to multiple users under the business type, to obtain sorted multiple target business data; determining whether there are multiple business data of the same user in the sorted multiple target business data; if so, obtaining the second target business data to be archived and the archiving quantity corresponding to the second target business data to be archived according to the multiple business data of the same user in the sorted multiple target business data.

[0144] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in the case of merging multiple business data of the same user into one business data, obtaining the second target business data to be archived; and obtaining the archiving quantity corresponding to the second target business data to be archived according to the row number set in the second target business data to be archived.

[0145] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when at least one first working process allocates corresponding grouped business data to each process for parallel processing according to the number of multiple processes to obtain corresponding archive packages, a response message is sent to the main process based on at least one first working process; the response message is used to indicate that at least one first working process has completed processing the corresponding grouped business data, and the response message includes the name and storage path of the corresponding archive package; when the main process receives the corresponding response message from at least one first working process, the main process extracts the archive package whose name matches the corresponding archive package based on the storage path of the archive package corresponding to at least one first working process, and obtains the archiving result of at least one grouped business data.

[0146] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a pre-set number of merges; and, based on the number of merges, compressing the archive package corresponding to at least one first working process based on the main process to obtain the archiving result of the first target business data to be archived.

[0147] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the main process detects that the first working process has failed, the main process reallocates the packet service data corresponding to the failed first working process to the second working process; the second working process is at least one process other than the first working process or at least one process in the first working process that has completed processing the corresponding packet service data.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data archiving method, characterized by, The method comprises: obtaining a plurality of users' first target business data to be archived; sorting the first target business data to be archived according to the business types in the first target business data to be archived and the business data corresponding to the plurality of users under the business types, obtaining second target business data to be archived and an archiving amount corresponding to the second target business data to be archived; grouping the second target business data to be archived based on a master process according to the archiving amount and a number of work processes, obtaining at least one grouped business data; obtaining an archiving result of the at least one grouped business data from at least one first work process based on the master process; the number of the at least one first work process is the number of the work processes; obtaining an archiving result of the first target business data to be archived according to the archiving result of the at least one grouped business data.

2. The method of claim 1, wherein, The sorting the first target business data to be archived according to the business types in the first target business data to be archived and the business data corresponding to the plurality of users under the business types, obtaining second target business data to be archived and an archiving amount corresponding to the second target business data to be archived, comprises: sorting a plurality of target business data in the first target business data to be archived according to the business types in the first target business data to be archived and the business data corresponding to the plurality of users under the business types, obtaining the plurality of target business data after sorting; determining whether the plurality of target business data after sorting includes a plurality of business data of a same user; if yes, obtaining the second target business data to be archived and the archiving amount corresponding to the second target business data to be archived according to the plurality of business data of the same user in the plurality of target business data after sorting.

3. The method of claim 2, wherein, The obtaining the second target business data to be archived and the archiving amount corresponding to the second target business data to be archived according to the plurality of business data of the same user in the plurality of target business data after sorting, comprises: in a case where the plurality of business data of the same user are combined into one piece of business data, obtaining the second target business data to be archived; obtaining the archiving amount corresponding to the second target business data to be archived according to a line number set by the second target business data to be archived.

4. The method of claim 1, wherein, The at least one first work process comprises a plurality of processes; the obtaining the archiving result of the at least one grouped business data from the at least one first work process based on the master process, comprises: when the at least one first work process assigns the corresponding grouped business data to each process for parallel processing according to the number of the plurality of processes, obtaining a corresponding archiving package, the at least one first work process sends a response message to the master process; the response message is used to indicate that the at least one first work process has completed processing of the corresponding grouped business data, and the response message comprises a name and a storage path of the corresponding archiving package; The main process receives the response message from the at least one first worker process, extracts the archive package matching the name of the corresponding archive package based on the storage path of the corresponding archive package of the at least one first worker process, and obtains the archiving result of the at least one packet service data.

5. The method of claim 1, wherein, The method further comprises: obtaining the archiving result of the first target service data to be archived based on the archiving result of the at least one packet service data. The method further comprises:

6. The method of claim 1, wherein, If the main process detects that the first worker process fails, the main process reassigns the packet service data corresponding to the failed first worker process to a second worker process; the second worker process is a process other than the at least one first worker process or a process in the at least one first worker process that has completed processing of the corresponding packet service data. The apparatus comprises:

7. A data archiving apparatus characterized by comprising: The data acquisition module is configured to acquire first target service data to be archived of a plurality of users. The data processing module is configured to sort the first target service data to be archived according to a service type in the first target service data to be archived and service data corresponding to the plurality of users under the service type, to obtain second target service data to be archived and an archiving amount corresponding to the second target service data to be archived. The grouping module is configured to group the second target service data to be archived by calling a main process according to the archiving amount and a number of worker processes, to obtain at least one packet service data. The packet archiving result acquisition module is configured to obtain an archiving result of the at least one packet service data based on the main process and at least one first worker process; the number of the at least one first worker process is the number of worker processes. The archiving result integration module is configured to obtain an archiving result of the first target service data to be archived based on the archiving result of the at least one packet service data. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, ​

Images