Data recovery processing method and platform, storage medium and computer equipment

Abstract

The invention discloses a data recovery processing method and platform, a storage medium and computer equipment, the method is applied to the data recovery processing platform, the data recovery processing platform comprises at least one uploading system, a cloud recovery system and an object storage system, and the method comprises the following steps: the uploading system obtains uploading configuration information in the cloud recovery system; the uploading system determines a data uploading priority according to the uploading configuration information, uploads the original data in the solid state disk to the object storage system according to the data uploading priority, and sends notification information of data uploading completion to the cloud recovery system after the original data is uploaded; and the cloud recovery system downloads the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpacks the original data according to the unpacking configuration information to obtain recovery data, and uploads the recovery data to the object storage system. According to the method, the resource utilization rate can be improved, and the data uploading and recycling speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a data recovery processing method, platform, storage medium and computer equipment. Background Art

[0002] In the field of autonomous driving, algorithm training requires a large amount of data, which is collected by the vehicle and written to the SSD. The size of the SSD for collecting data is often around 1TB, and the data in the SSD needs to be recovered to the cloud in order to train the algorithm model. The collected data is very rich and large, including long-term content recorded by multiple sensors. Therefore, it is necessary to classify and filter these data, combine them into new recovered data, and hand them over to different business parties for use.

[0003] In the prior art, the original data in the SSD is uploaded to the cloud object storage system mainly through the upload system, and then the original data is unpacked into the recovered data and uploaded to the object storage system. However, the limited bandwidth of the upload system is used for uploading and unpacking data at the same time, the bandwidth resource utilization rate is low, and the processing speed is slow. Especially when the upload system needs to upload and recover data from multiple SSDs at the same time, there will be problems such as slow task scheduling, many resource conflicts, and easy jamming. Summary of the invention

[0004] In view of this, the present application provides a data recovery processing method, platform, storage medium and computer equipment, the main purpose of which is to solve the technical problems of slow data upload, recovery speed and low resource utilization.

[0005] According to a first aspect of the present invention, a data recovery processing method is provided, which is applied to a data recovery processing platform, wherein the data recovery processing platform includes at least one upload system, a cloud recovery system and an object storage system, wherein:

[0006] The uploading system obtains the uploading configuration information in the cloud recycling system;

[0007] The upload system determines the data upload priority according to the upload configuration information, and uploads the original data in the solid state drive to the object storage system according to the data upload priority, and sends a notification message of data upload completion to the cloud recycling system after the original data is uploaded;

[0008] The cloud recycling system downloads the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpacks the original data according to the unpacking configuration information to obtain recycled data, and uploads the recycled data to the object storage system.

[0009] According to a second aspect of the present invention, a data recovery processing platform is provided, the platform comprising at least one upload system, a cloud recovery system and an object storage system, wherein:

[0010] The upload system is used to obtain upload configuration information in the cloud recycling system, determine the data upload priority according to the upload configuration information, upload the original data in the solid state drive to the object storage system according to the data upload priority, and send a notification message of data upload completion to the cloud recycling system after the original data is uploaded;

[0011] The cloud recycling system is used to download the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpack the original data according to the unpacking configuration information to obtain the recycled data, and upload the recycled data to the object storage system;

[0012] The object storage system is used to store the original data and the recycled data.

[0013] According to a third aspect of the present invention, there is provided a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned data recovery processing method is implemented.

[0014] According to a fourth aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data recovery processing method when executing the program.

[0015] The data recovery processing method, platform, storage medium and computer equipment provided by the present invention can facilitate unified management of configuration files of different upload systems by using a cloud recovery system to store upload configuration information and first obtain the upload configuration information in the cloud recovery system before uploading data, thereby facilitating the maintenance and update of configuration files. In addition, by using the upload system to upload the original data in the solid-state hard disk to the object storage system, and sending a notification to the cloud recovery system, and then using the cloud recovery system to download the original data, and unpacking the original data to obtain the recovered data, the tasks of data upload and recovery can be split, thereby decoupling and pipeline processing the tasks of data upload and recovery, effectively improving resource utilization and accelerating the speed of data upload and recovery.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A schematic diagram of a data recovery process method provided by an embodiment of the present invention is shown;

[0019] Figure 2 A schematic diagram of a process of a data recovery processing platform provided by an embodiment of the present invention is shown;

[0020] Figure 3 A structural schematic diagram of another data recovery processing method provided by an embodiment of the present invention is shown;

[0021] Figure 4 A structural schematic diagram of another data recovery processing platform provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0023] In one embodiment, Figure 1 As shown, a data recovery processing method is provided, which can be applied to Figure 2 In the data recovery processing platform shown, the data recovery processing platform includes at least one upload system, a cloud recovery system and an object storage system. Among them, the upload system can obtain the upload configuration information in the cloud recovery system before uploading the data, and then determine the data upload priority according to the upload configuration information, and upload the original data in the solid-state hard disk to the object storage system according to the data upload priority, and then send the notification information of the completion of data upload to the cloud recovery system after the original data upload is completed. Furthermore, after receiving the notification information of the completion of data upload, the cloud recovery system can download the original data and the unpacking configuration information corresponding to the original data in the object storage system, and unpack the original data according to the unpacking configuration information to obtain the recovered data, and finally upload the recovered data to the object storage system.

[0024] Specifically, when the upload system is started, it can pull the latest upload configuration information from the cloud recycling system, where the upload configuration information can include the maximum number of working threads, storage bucket information, environmental information and other information required for uploading various types of data. After the upload system is started, the user can connect at least one solid-state drive (SSD) that has completed the collection to the upload system in the form of insertion, and upload the data, where the data in the SSD can be packaged into at least one data packet. The upload system can upload the data in each SSD in units of data packets to the object storage system in accordance with the data upload priority of each connected SSD and the data upload priority of each data packet in each SSD, and send the notification information of the completion of the data upload to the cloud recycling system after the data upload is completed. Furthermore, after receiving the notification information of the completion of data upload, the cloud recycling system can call the unpacking program to parse and process the uploaded data packet. Specifically, the unpacking program can download the original data and the unpacking configuration information corresponding to the original data in the object storage system, wherein the unpacking configuration information includes the unpacking configuration information pre-configured in the data packet and / or the unpacking configuration information pre-configured in the cloud recycling system, etc. Then, the original data can be unpacked according to the unpacking configuration information to obtain the recovered data, and finally the recovered data can be uploaded to the object storage system.

[0025] It should be noted that, by obtaining the upload configuration information in the cloud recovery system before uploading the data, this embodiment can ensure that the upload system can obtain the latest configuration information. In addition, by uniformly recording the upload configuration information of different upload systems in the cloud recovery system, it can facilitate the unified management and update maintenance of the configuration information, thereby reducing the problem of complex configuration files and high management difficulty caused by different configuration files of each upload system and configuration file updates. Furthermore, by splitting the task of data upload and recovery into uploading the original data in the solid-state hard disk through the upload system and unpacking and uploading the original data through the cloud recovery system, the tasks of data upload and data recovery can be decoupled and scheduled in parallel, thereby improving the bandwidth resource utilization of the upload system, and also improving the overall speed of data upload and data recovery.

[0026] In this embodiment, the functions and hardware equipment (such as the type and number of servers) of the cloud recycling system can be freely expanded to improve the efficiency of data unpacking. Moreover, the cloud recycling system can also classify and combine the original data in a targeted manner according to various unpacking configuration information, and can uniformly support various data unpacking methods. In addition, by managing the upload system through the cloud recycling system, the maintenance cost of the upload system can be reduced, and the throughput of the upload system can be doubled.

[0027] The data recovery processing method provided in this embodiment can facilitate unified management of configuration files of different upload systems by using the cloud recovery system to store the upload configuration information and first obtain the upload configuration information in the cloud recovery system before uploading the data, thereby facilitating the maintenance and update of the configuration files. In addition, by using the upload system to upload the original data in the solid-state hard disk to the object storage system and sending a notification to the cloud recovery system, and then using the cloud recovery system to download the original data and unpack the original data to obtain the recovered data, the tasks of data upload and recovery can be split, thereby decoupling and pipeline processing the tasks of data upload and recovery, effectively improving resource utilization and accelerating the speed of data upload and recovery.

[0028] In one embodiment, the upload system obtains the upload configuration information in the cloud recycling system, which can be obtained when the upload system is started (i.e., when the user opens the software of the upload system), or can be obtained in a timed manner at a preset time interval. In this way, it is guaranteed that the latest configuration information in the cloud recycling system is obtained. In this embodiment, the cloud recycling system records the upload configuration information of multiple upload systems, and the upload configuration information of each upload system can include at least one of the maximum number of working threads, storage bucket information, and environmental information.

[0029] In the above embodiment, the maximum number of working threads is the maximum number of threads that the upload system can initiate at the same time, which represents the maximum number of data upload tasks that the upload system can handle simultaneously. For example, if the maximum number of working threads is 5, the maximum number of data upload tasks that the upload system can handle simultaneously is also 5, that is, the upload system can upload data from 5 solid-state drives at the same time. The storage bucket information mainly includes the account password information of the object storage system, the number of parallel uploads, the size of the data packet and other parameters. The environmental information includes the domain name of the cloud service, the regional environmental information and other information. Through these parameters, the upload system can upload the data in the solid-state drive to the object storage system in units of data packets. This embodiment can facilitate the unified management and update maintenance of the configuration information by uniformly recording the upload configuration information of different upload systems in the cloud recovery system, thereby reducing the problem of complex configuration files and high management difficulty caused by different configuration files of each upload system and the update of configuration files.

[0030] In one embodiment, referring to Figure 3, before the upload system determines the data upload priority according to the upload configuration information, the above data recovery processing method may also include the following steps: after the user connects the solid-state hard disk to the upload system, the upload system may respond to the connection operation of the solid-state hard disk and send the notification information of the solid-state hard disk connection to the cloud recovery system, and then the upload system may scan the files in the solid-state hard disk and divide the files in the solid-state hard disk into at least one data packet according to the data source and / or data collection time. For example, for the solid-state hard disk connected to the vehicle end, its data may be stored in the solid-state hard disk in the form of files. When packaging the files in such solid-state hard disk, each file may be packaged according to the collection time corresponding to the file and the license plate number corresponding to the file, such as each file collected at the same vehicle end from 7 am to 9 am on a certain day may be packaged into a data packet. It can be understood that the files in the solid-state hard disk can be packaged into one or more data packets. In addition, the size and classification method of the data packet can also be set according to the parameters in the upload configuration information. This embodiment does not specifically limit the packaging method of the data packet.

[0031] In the above embodiment, after the data in the solid state drive is packaged into at least one data packet, each data packet in the solid state drive can be uploaded to the object storage system in units of data packets, and after each data packet is uploaded, the uploaded data packet is downloaded and unpacked by the cloud recovery system, so as to obtain the recovered data of each data packet in the solid state drive. This embodiment can reduce the granularity of data upload and recovery by dividing the files in the solid state drive into at least one data packet according to the data source and / or data acquisition time, thereby improving the speed of data processing. In addition, in the scenario with a large amount of data, there will be a problem that hundreds of data in solid state drives need to be uploaded and recovered every day. In this scenario, this embodiment can realize the pipeline processing of data upload and recovery by uploading each data packet in each solid state drive one by one according to the data upload priority, and unpacking and recovering the data in each data packet one by one by using the cloud recovery system, further accelerating the overall speed of data upload and data recovery. At the same time, this pipeline data processing method also alleviates the problems of slow data processing task scheduling, many resource conflicts, and easy jamming.

[0032] In one embodiment, referring to Figure 3The specific process of the upload system determining the data upload priority according to the upload configuration information and uploading the original data in the solid-state drive to the object storage system according to the data upload priority can be achieved through the following steps: First, the upload system can determine the first data upload priority of each currently connected solid-state drive and the second data upload priority of each data packet in the solid-state drive according to the upload configuration information; then, according to the first data upload priority and the second data upload priority, determine the data upload order of each solid-state drive and each data packet in the solid-state drive; finally, according to the data upload order, upload each data packet in each solid-state drive to the object storage system in turn.

[0033] In the above embodiment, the data upload priority includes the first data upload priority of the solid state drive and the second data upload priority of the data packet. Specifically, before uploading data, the upload system can determine the first data upload priority of each currently connected solid state drive according to the upload configuration information, and then determine the data upload order of each solid state drive according to the first data upload priority of each solid state drive, so as to determine the first batch of solid state drives to upload data according to the data upload order of the solid state drive, that is, determine the solid state drives to upload data. Further, before uploading the data in the solid state drive, the second data upload priority of each data packet in the solid state drive can be determined according to the upload configuration information, and each data packet in the solid state drive is uploaded one by one according to the second data upload priority of each data packet until all the data packets in the solid state drive are uploaded. Further, after the data in a solid state drive in the first batch is uploaded, the solid state drive to which the data is to be uploaded next can be determined according to the data upload order of the solid state drive, and the above-mentioned upload process of each data packet in the solid state drive is repeated until the data in all solid state drives connected in the upload system are uploaded. This embodiment uploads the data in the solid-state hard disk in units of data packets, which can reduce the granularity of data uploading and recovery, thereby improving the speed of data uploading. In addition, after all data packets in the solid-state hard disk are uploaded, the solid-state hard disk can be removed from the server where the uploading system is installed without waiting for the data packets to be unpacked. In this way, the speed of removing the solid-state hard disk and the utilization rate can be effectively improved.

[0034] In one embodiment, referring to Figure 3Before the upload system uploads the raw data in the solid-state hard disk to the object storage system, the data recovery processing method further includes the following steps: the upload system sends a notification message of the start of data upload to the cloud recovery system, and the cloud recovery system records the raw data to be uploaded in the database. Furthermore, after the upload system sends a notification message of the completion of data upload to the cloud recovery system, the data recovery processing method further includes the following steps: the cloud recovery system records the uploaded raw data in the database and sets the uploaded raw data to a decompressible state.

[0035] In the above embodiment, the upload system can synchronously report each step of the data processing process executed in the upload system to the cloud recycling system, wherein the content of the report may include notification information of the user connecting to the solid-state hard disk in the upload system, notification information of the solid-state hard disk that determines the data to be uploaded, notification information of the data packet to be uploaded, notification information of the completion of the data packet upload, and notification information of the completion of the upload of all data packets in the solid-state hard disk, etc. Further, when the cloud recycling system receives each notification information, it can synchronously record the received notification information in the database for easy query at any time. This embodiment can enable the cloud recycling system to timely understand the progress of data upload by synchronously reporting the notification information of each step of data upload to the cloud recycling system. In this way, on the one hand, it can facilitate the cloud recycling system to download and unpack the uploaded data in a timely manner, and on the other hand, it can also facilitate the user to query the progress of data upload at any time, thereby improving the user experience.

[0036] In one embodiment, the cloud-based recycling system downloads the original data and the unpacking configuration information corresponding to the original data in the object storage system, and unpacks the original data according to the unpacking configuration information to obtain the specific process of recycled data, which can be achieved through the following steps: the cloud-based recycling system polls the records in the database, and when the record of the original data in the unpackable state is found, the cloud-based recycling system downloads the original data and at least one unpacking configuration information corresponding to the original data in the object storage system, and classifies and combines the original data according to the unpacking configuration information to obtain at least one piece of recycled data.

[0037] In the above embodiment, the cloud recycling system can facilitate real-time monitoring of the data upload progress in the upload system by recording each data processing process in the upload system in the database. In this embodiment, the cloud recycling system can search for uploaded data marked as unpackable in the database by polling. After finding such data, the cloud recycling system can download the original data corresponding to this record and at least one unpacking configuration information corresponding to the original data in the object storage system, wherein the unpacking configuration information may include the unpacking configuration information pre-configured in the data packet and / or the unpacking configuration information pre-configured in the cloud recycling system, etc. Finally, the cloud recycling system can reclassify and combine the downloaded original data according to the unpacking configuration information to obtain at least one piece of recycled data. In this embodiment, by pre-setting the unpacking configuration information in the data packet and the cloud recycling system, and unpacking the data packet in the solid-state hard disk through the unpacking configuration information, the recycled data for different data processing systems can be obtained in a targeted manner, thereby improving the efficiency of data recycling processing.

[0038] In one embodiment, referring to Figure 2 and Figure 3 , the above-mentioned data recovery processing method may also include the following steps: the cloud-based recovery system may write the recovered data and the metadata of the recovered data into the data storage system (Data Storage System, DSS), and / or send the recovered data to the data computing system (Data Computing Service, DCS). In this embodiment, after unpacking the data, the cloud-based recovery system may send the unpacked data to the corresponding data computing system according to the data computing system corresponding to the recovered data recorded in the unpacking configuration information. In addition, the cloud-based recovery system may also write the recovered data and the metadata of the recovered data into the data storage system, so as to facilitate subsequent query operations on the data through the front-end service system. This embodiment can effectively improve the utilization rate of the data by sending the recovered data to the downstream data storage system and data computing system.

[0039] In one embodiment, Figure 4As shown, the data recovery processing platform also includes a front-end service system. On this basis, the above data recovery processing method can also include the following steps: the front-end service system responds to the task query request and sends the task query request to the cloud recovery system. The cloud recovery system queries the data list corresponding to the task query request in the database and sends the data list to the front-end service system. The front-end service system displays the data list and responds to the data viewing request and sends the data viewing request to the cloud recovery system. The cloud recovery system obtains the query data corresponding to the data viewing request in the object storage system and sends the query data to the front-end service system. In this embodiment, the front-end service system can interact with the cloud recovery system through API interaction to query the original data and recovered data stored in the object storage system. In addition, the upload system can also interact with the cloud recovery system through API interaction. By setting up the front-end service system, this embodiment can facilitate timely query of the data stored in the object storage system, thereby effectively improving the efficiency of data query and the visibility of data, thereby improving the user experience.

[0040] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the implementation process of this embodiment, a data recovery processing method is provided, such as Figure 3 As shown, the method comprises the following steps:

[0041] 1) When the uploading system is started, it obtains the uploading configuration information from the cloud recycling system, wherein the cloud recycling system records the uploading configuration information of multiple uploading systems.

[0042] 2) The solid state drive is connected to the upload system in an inserted form. The upload system scans the data in the solid state drive and pre-processes the data to be uploaded.

[0043] 3) After the upload system completes preprocessing, it notifies the cloud recycling system, which then adds a record of the data to be uploaded to the database.

[0044] 4) The uploading system starts uploading data and continuously reports the upload progress to the cloud recycling system during the uploading process. The cloud recycling system writes the data upload progress into the database.

[0045] 5) After the upload system completes uploading a data packet, the cloud recovery system updates the data table in the database and sets the uploaded data packet to a decompressible state.

[0046] 6) The cloud recycling system triggers the unpacking program, downloads the original data and unpacking configuration information in the object storage system, and unpacks the original data according to the unpacking configuration information to obtain the recycled data.

[0047] 7) After unpacking is completed, the cloud system notifies the downstream data storage system and data computing system to proceed to the next step of processing.

[0048] The specific processing process of each step in this embodiment can refer to the data processing process described in the above-mentioned data recovery processing method embodiments, which will not be repeated here. The technical effect of the data processing method proposed in this embodiment is demonstrated through a set of test data.

[0049] 1. Maximum capacity of data recovery per day: increased from 100TB to 200TB;

[0050] 2. Data upload time (from inserting the disk to removing it):

[0051] a. Average time: from 6 hours to 2 hours;

[0052] b. P90 time (i.e. 90% of SSDs): from 17 hours to 6 hours;

[0053] 3. Overall data recovery time:

[0054] a. Average time: from 6 hours to 3 hours;

[0055] b. P90 time (i.e. 90% of SSDs): increased from 17 hours to 8 hours;

[0056] 4. From disk insertion to unpacking completion time, the 99% weekly average recovery completion time was accelerated from 28.27h to 17.73h, an increase of 37%;

[0057] 5. The 99% weekly average time from inserting the disk to removing it was accelerated from 28.27 hours to 8.78 hours, an improvement of 68%.

[0058] It can be seen from the above test data that the data processing method proposed in this embodiment can effectively improve the processing time of data uploading and recovery, thereby improving the speed of data processing. At the same time, the above method can also effectively reduce the bandwidth resources of the upload system, thereby avoiding the occurrence of problems such as slow task scheduling, multiple resource conflicts and jamming, and effectively improving the stability of data uploading and recovery.

[0059] Further, as Figure 1 , Figure 3 The specific implementation of the method shown in the embodiment also provides a data recovery processing platform, such as Figure 2 As shown, the data recovery processing platform includes an upload system, a cloud recovery system and an object storage system, wherein:

[0060] The upload system is used to obtain the upload configuration information in the cloud recycling system, determine the data upload priority according to the upload configuration information, upload the original data in the solid-state drive to the object storage system according to the data upload priority, and send the notification information of the completion of the data upload to the cloud recycling system after the original data is uploaded;

[0061] A cloud recycling system is used to download the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpack the original data according to the unpacking configuration information, obtain the recycled data, and upload the recycled data to the object storage system;

[0062] Object storage system for storing raw and recycled data.

[0063] In a specific application scenario, the upload system is specifically used to obtain upload configuration information in the cloud recycling system at startup, and / or, to obtain upload configuration information in the cloud recycling system at preset time intervals, wherein the cloud recycling system records upload configuration information of multiple upload systems, and the upload configuration information of each upload system includes at least one of the maximum number of working threads, storage bucket information and environmental information.

[0064] In a specific application scenario, the upload system is also used to respond to the connection operation of the solid-state drive and send the notification information of the solid-state drive connection to the cloud recovery system; scan the files in the solid-state drive and divide the files in the solid-state drive into at least one data packet according to the data source and / or data collection time.

[0065] In a specific application scenario, the upload system is specifically used to determine the first data upload priority of each currently connected solid-state drive and the second data upload priority of each data packet in the solid-state drive according to the upload configuration information; determine the data upload order of each solid-state drive and each data packet in the solid-state drive according to the first data upload priority and the second data upload priority; and upload each data packet in each solid-state drive to the object storage system in turn according to the data upload order.

[0066] In a specific application scenario, the upload system is also used to send notification information about the start of data upload to the cloud recovery system. The cloud recovery system is also used to record the original data to be uploaded in the database, as well as the uploaded original data in the database, and set the uploaded original data to an unpackable state.

[0067] In a specific application scenario, the cloud recycling system is specifically used to poll the records in the database, and when the records of the original data in the unpackable state are found, the original data and at least one unpacking configuration information corresponding to the original data are downloaded from the object storage system; according to the unpacking configuration information, the original data is classified and combined to obtain at least one piece of recycled data.

[0068] In a specific application scenario, the cloud recycling system is also used to write the recycled data and the metadata of the recycled data into a data storage system, and / or send the recycled data to a data computing system.

[0069] In specific application scenarios, such as Figure 4 As shown, the data recovery processing platform also includes a front-end service system, wherein the front-end service system is used to respond to the task query request and send the task query request to the cloud recovery system; the cloud recovery system is also used to query the data list corresponding to the task query request in the database, and send the data list to the front-end service system; the front-end service system is also used to display the data list, and respond to the data viewing request and send the data viewing request to the cloud recovery system; the cloud recovery system is also used to obtain the query data corresponding to the data viewing request in the object storage system, and send the query data to the front-end service system.

[0070] It should be noted that for other corresponding descriptions of the functional units involved in the data recovery processing platform provided in this embodiment, please refer to Figure 1 , Figure 3 The corresponding description in will not be repeated here.

[0071] Based on the above Figure 1 and Figure 3 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 , Figure 3 The data recovery processing method shown.

[0072] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product. The software product to be identified can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0073] Based on the above Figure 1 and Figure 3 The method shown, and Figure 2 and Figure 4In order to achieve the above-mentioned purpose, the present embodiment also provides a data recovery processing computer device, which can be a personal computer, a server, a smart phone, a tablet computer, a smart watch, or other network devices, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program and an operating system; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 , Figure 3 The method shown.

[0074] Optionally, the computer device may also include an internal memory, a communication interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, a display, an input platform such as a keyboard, etc. Optionally, the communication interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0075] Those skilled in the art will appreciate that the computer device structure for identifying an operation action provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0076] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the above-mentioned computer device hardware and the software resources to be identified, and supports the operation of the information processing program and other software and / or programs to be identified. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing computer device.

[0077] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. The present application first uses the upload system to obtain the upload configuration information in the cloud recovery system, and determines the data upload priority according to the upload configuration information, and uploads the original data in the solid state drive to the object storage system according to the data upload priority, and after the original data is uploaded, the notification information of the completion of the data upload is sent to the cloud recovery system, and then the cloud recovery system is used to download the original data and the unpacking configuration information corresponding to the original data in the object storage system, and unpack the original data according to the unpacking configuration information to obtain the recovered data, and finally upload the recovered data to the object storage system. Compared with the prior art, the above data recovery method or data recovery platform can improve resource utilization and speed up data upload and recovery.

[0078] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the platform in the implementation scenario can be distributed in the platform of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more platforms different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0079] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A data recovery processing method, characterized in that: The data recovery processing method is applied to a data recovery processing platform, which includes at least one upload system, a cloud recovery system and an object storage system, wherein: The uploading system obtains the uploading configuration information in the cloud recycling system; The upload system determines the data upload priority according to the upload configuration information, and uploads the original data in the solid state drive to the object storage system according to the data upload priority, and sends a notification message of data upload completion to the cloud recycling system after the original data is uploaded; The cloud recycling system downloads the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpacks the original data according to the unpacking configuration information to obtain recycled data, and uploads the recycled data to the object storage system.

2. The method according to claim 1, characterized in that The uploading system obtains the uploading configuration information in the cloud recycling system, including: The upload system obtains the upload configuration information in the cloud recycling system when it is started, and / or the upload system obtains the upload configuration information in the cloud recycling system at a preset time interval, wherein the cloud recycling system records the upload configuration information of multiple upload systems, and the upload configuration information of each upload system includes at least one of the maximum number of working threads, storage bucket information and environmental information.

3. The method according to claim 1, characterized in that: Before the uploading system determines the data uploading priority according to the uploading configuration information, the method further includes: The upload system sends notification information of the solid state drive connection to the cloud recycling system in response to the solid state drive connection operation; The upload system scans the files in the solid state drive and divides the files in the solid state drive into at least one data packet according to data source and / or data collection time.

4. The method according to claim 3, characterized in that The uploading system determines the data uploading priority according to the uploading configuration information, and uploads the original data in the solid state drive to the object storage system according to the data uploading priority, including: The uploading system determines, according to the upload configuration information, a first data upload priority for each currently connected solid state drive and a second data upload priority for each data packet in the solid state drive; The uploading system determines the data uploading order of each of the solid state drives and each data packet in the solid state drives according to the first data uploading priority and the second data uploading priority; The uploading system uploads each data packet in each of the solid state drives to the object storage system in sequence according to the data uploading order.

5. The method according to claim 1, characterized in that Before the uploading system uploads the original data in the solid state drive to the object storage system, the method further includes: The uploading system sends notification information of the start of data uploading to the cloud recycling system, and the cloud recycling system records the raw data to be uploaded in the database; After the uploading system sends notification information of data upload completion to the cloud recycling system, the method further includes: The cloud recycling system records the uploaded original data in a database and sets the uploaded original data to an unpackable state.

6. The method according to claim 5, characterized in that The cloud recycling system downloads the original data and the unpacking configuration information corresponding to the original data in the object storage system, and unpacks the original data according to the unpacking configuration information to obtain the recycled data, including: The cloud recycling system polls the records in the database, and when finding the record of the original data in the unpackable state, downloads the original data and at least one unpacking configuration information corresponding to the original data in the object storage system; The original data is classified and combined according to the unpacking configuration information to obtain at least one piece of the recovered data.

7. The method according to claim 1, characterized in that The method further comprises: The cloud recycling system writes the recycled data and the meta-information of the recycled data into a data storage system; and / or The cloud recycling system sends the recycling data to a data computing system.

8. The method according to claim 1, characterized in that The data recovery processing platform also includes a front-end service system, and the method also includes: The front-end service system responds to the task query request by sending the task query request to the cloud recycling system; The cloud recycling system searches the database for a data list corresponding to the task query request, and sends the data list to the front-end service system; The front-end service system displays the data list and, in response to a data viewing request, sends the data viewing request to the cloud recycling system; The cloud-based recycling system obtains query data corresponding to the data viewing request in the object storage system, and sends the query data to the front-end service system.

9. A data recovery processing platform, characterized in that: The data recovery processing platform includes at least one upload system, a cloud recovery system and an object storage system, wherein: The upload system is used to obtain upload configuration information in the cloud recycling system, determine the data upload priority according to the upload configuration information, upload the original data in the solid state drive to the object storage system according to the data upload priority, and send a notification message of data upload completion to the cloud recycling system after the original data is uploaded; The cloud recycling system is used to download the original data and the unpacking configuration information corresponding to the original data in the object storage system, unpack the original data according to the unpacking configuration information to obtain the recycled data, and upload the recycled data to the object storage system; The object storage system is used to store the original data and the recycled data.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images