A batch data unfreezing method based on a blue light recording scene

Abstract

The application discloses a kind of batch data thawing methods based on blue light recording scene, including step one, user end calls the aws protocol interface of s3 service and initiates to generate list request, generates list file, upload list file to s3;Step two, user end calls the aws protocol interface of s3control service and initiates to create batch task request, creates task queue;Step three, from the object set in list file in task queue, the object name of the object that needs to be cached is obtained from object set, thread pool is created, respectively from single thread in thread pool Send a cache request to s3, obtain the cache content generated according to cache request, generate cache result list according to cache content, upload cache result list to s3;Step four, return to execute step three, until task queue has no task to be executed. Batch cache and system resource when batch sending task are managed, improve cache efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of blue-ray storage, in particular to a batch data unfreezing method based on a blue-ray recording scene. BACKGROUND

[0002] At present, there are three ways of data storage in China: magnetic tape (data stream), hard disk and blue-ray disc. The use of magnetic tape to save data has the following disadvantages: easy to be affected by moisture, demagnetization and tape breakage; short storage time, generally about 10 years, and can only be discarded at the end of the life, and can not be used again; format and cross-generation are not compatible, and need to be copied and copied every three to five years, which is equivalent to repeated construction, slow response speed, and a large amount of manpower and material resources are needed for repeated rescue, and the storage condition needs to be in a constant temperature and humidity environment, with high energy consumption; due to the above defects, magnetic tape is being gradually eliminated. Due to its physical properties, the service life and safety of hard disk make it unsuitable as a long-term storage medium; from the current and long-term point of view, blue-ray is the best choice for long-term data security storage.

[0003] Blue-ray products have the following advantages: (1) longer life, up to 50-100 years; (2) lower cost: the cost of blue-ray disc has been reduced to a very low level, and now 50G blue-ray disc currently only converts to less than one yuan per GB, with the continuous expansion of the market, the continuous improvement of disc capacity, and the continuous popularization of blue-ray technology, the price of blue-ray disc will continue to decline. (3) lower energy consumption: blue-ray disc storage method does not need to load high-power power supply in real time, and the storage environment has no special requirements.

[0004] Based on the storage platform of blue-ray equipment, at present, through hierarchical storage, data is first written into the cache with faster writing speed, and then the data is read from the cache to record the blue-ray disc. At present, this scheme is the mainstream scheme in the field of blue-ray storage. At the same time, when reading data from the disc or cache, a problem is encountered: using the aws protocol interface RestoreObject (RestoreObject, restore the archived copy of the object back to Amazon S3) can only cache one object at a time, and cannot cache batch data. SUMMARY

[0005] The present application provides a batch data unfreezing method based on a blue-ray recording scene to overcome the above technical problems.

[0006] A batch data unfreezing method based on a blue-ray recording scene, comprising,

[0007] Step one, the user end calls the aws protocol interface of s3 service and initiates the generation of the list request, transmits the prefix character, the object prefix character that needs to be cached, the bucket name of the list generation, the bucket name of the object that needs to be cached, the list generation id to the aws protocol interface, filters the object name with the prefix character from the storage bucket, writes the filtered object into the list file, and uploads the list file to s3;

[0008] Step two, the user end calls the aws protocol interface of s3 control service and initiates the creation of batch task request, obtains the object name according to the list file, takes the object name as the target object name of the batch task, the attributes of the task include the task priority, the path of the object in the list file, the storage bucket name of the task execution result, the task state, and adds the created task to the task queue;

[0009] Step three, obtains the to-be-executed task from the task queue, obtains the object name of the object that needs to be cached from the object set in the list file, creates a thread pool, selects a single thread from the thread pool to send a cache request to s3, updates the cache result of the object to complete when the cache request for a single object is sent successfully, obtains the cache content generated according to the cache request, generates a cache result list according to the cache content, uploads the cache result list to s3, and updates the state of the selected executed task to complete;

[0010] Step four, returns to step three until there is no to-be-executed task in the task queue.

[0011] Preferably, the step of obtaining the to-be-executed task from the task queue includes selecting the task with the highest priority from the task list for execution.

[0012] Preferably, the step of creating a thread pool includes calculating the number of threads in the thread pool according to the number of concurrent processors, the number of processor cores, the memory, and the network sockets of the user end.

[0013] Preferably, the step of selecting a single thread from the thread pool to send a cache request to s3 includes submitting the cache request to the thread pool, determining whether there is an available worker in the thread pool,

[0014] if there is an available worker, assigning the cache request task to the worker, and releasing the worker to the cache pool after the worker processes the cache request,

[0015] if there is no available worker, determining whether the number of workers in the thread pool exceeds the capacity of the thread pool,

[0016] if the capacity of the thread pool is exceeded, continuing to determine whether the thread pool is non-blocking, if yes, returning empty, and continuing to determine whether it is non-blocking, if no, waiting for an available worker,

[0017] If the capacity of the thread pool is not exceeded, a new worker is created to handle the cache request, and the worker is released to the cache pool after the cache request is handled.

[0018] Preferably, the uploading of the first object list to s3 includes uploading to s3 through an aws multipart upload protocol interface, the aws multipart upload protocol interface including a create multipart upload interface, an upload part interface, and a complete multipart upload interface.

[0019] The present application provides a batch data thawing method based on a Blu-ray recording scene, which can be compatible with the aws protocol to batch cache specific objects, so that the objects selected by the user can be batch cached, and the system resources during batch sending tasks are managed, so that the server resource usage is always maintained within a reasonable range, and the service will not stop working due to too many tasks in a short time, avoiding errors caused by too many tasks in a short time. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a method flowchart of the present application;

[0022] Figure 2 is a method execution sequence diagram of the present application;

[0023] Figure 3 is a thread pool task execution flowchart of the present application;

[0024] Figure 4 is a thread pool management diagram of the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] Figure 1 is a method flowchart of the present application, as shown in Figure 1 The method of the present embodiment can include:

[0027] Step 1: The user client calls the AWS protocol interface of the S3 service and initiates a request to generate a manifest, such as... Figure 2 As shown in the execution process, the prefix character, the prefix character of the object to be cached, the bucket name of the generated manifest, the bucket name of the object to be cached, and the generated manifest ID are transmitted to the AWS protocol interface. Objects whose names start with the prefix character are filtered from the storage bucket. The filtered objects are written to the manifest file and uploaded to S3. The AWS chunked upload protocol interface includes the chunked upload interface, the chunked upload interface, and the chunked upload completion interface.

[0028] Step 2: The client calls the AWS protocol interface of the s3control service and initiates a request to create batch tasks. The object name is obtained from the manifest file and used as the target object name of the batch task. The attributes of the task include the task priority, the path of the object in the manifest file, the name of the bucket where the task execution results are stored, and the task status. The created task is added to the task queue.

[0029] Step 3: Retrieve tasks to be executed from the task queue. For each task, retrieve the object collection from the manifest file. Then, retrieve the object names of the objects to be cached from this collection. Create a thread pool. Thread pool creation involves calculating the pool's capacity based on the number of concurrent processors, processor cores, memory, and network sockets on the client side. Finally, select individual threads from the thread pool to send cache requests to S3. Figure 3 As shown, the step of selecting a single thread from the thread pool to send the cache request to s3 includes submitting the cache request to the thread pool and determining whether there are available workers in the thread pool.

[0030] If a worker is available, assign the cache request task to the worker. After the worker finishes sending the cache request, release the worker back into the cache pool.

[0031] If no workers are available, then check if the number of workers in the thread pool exceeds the thread pool's capacity.

[0032] If the thread pool capacity is exceeded, continue checking if there are non-blocking threads in the pool. If so, return null and continue checking if there are non-blocking threads. If not, wait for available workers.

[0033] If the thread pool capacity is not exceeded, a new worker is created to handle the cache request. After the worker finishes handling the cache request, the worker is released into the cache pool.

[0034] Since the number of objects to be cached in the batch task is large, and the number of threads working is large, too many threads will occupy system resources and bring scheduling overhead, thereby affecting cache locality and overall performance, the embodiment uses a thread pool ants to maintain multiple threads, waiting for the supervisor to allocate tasks that can be executed concurrently. This avoids the cost of creating and destroying threads when processing short-time tasks. The thread pool not only ensures the full use of the kernel, but also prevents excessive scheduling.

[0035] The thread pool mainly consists of two concepts, one is a task queue, and the other is a worker thread. The task queue is a blocking queue that saves tasks to be executed. The worker thread is a loop that accepts tasks from the queue and executes them. When there are available workers (the number of threads does not reach the upper limit), the task is executed, and the worker is released after completing the task. Figure 4 The workerArray (task queue) stores the workers in order according to the entry time of the worker, with the FIFO (First In First Out) feature. At the same time, antsPool has the feature of cleaning expired workers regularly, which will regularly find expired workers from workerArray and put them into workerPool sync.Pool (worker pool) to wait for GC (garbage collection). When the size of the thread pool has been reached, the task is put into the queue to wait.

[0036] When the cache request for a single object is sent successfully, the cache result of the object is updated to complete, the cache content generated according to the cache request is obtained, the cache result list is generated according to the cache content, the cache result list is uploaded to s3, and the selected task state is updated to complete;

[0037] Step four, return to step three until the task queue is empty.

[0038] Overall beneficial effects:

[0039] The application provides a batch data thawing method based on a Blu-ray recording scene, which can be compatible with the aws protocol to batch cache specific objects, so that objects selected by the user can be batch cached, and the system resources during batch sending tasks are managed, so that the server resource usage is always maintained within a reasonable range, and the service will not stop working due to too many tasks in a short time, avoiding errors caused by too many short-time tasks.

[0040] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A batch data unfreezing method based on Blu-ray recording scenarios, characterized in that, include, Step 1: The client calls the AWS protocol interface of the S3 service and initiates a manifest generation request. It transmits the prefix character, the prefix character of the object to be cached, the bucket name of the manifest generation bucket, the bucket name of the object to be cached, and the manifest generation ID to the AWS protocol interface. It filters objects whose names start with the prefix character from the storage bucket, writes the filtered objects into the manifest file, and uploads the manifest file to S3. Step 2: The client calls the AWS protocol interface of the s3control service and initiates a request to create batch tasks. The object name is obtained from the manifest file and used as the target object name of the batch task. The attributes of the task include the task priority, the path of the object in the manifest file, the name of the bucket where the task execution results are stored, and the task status. The created task is added to the task queue. Step 3: Retrieve tasks to be executed from the task queue. Retrieve the object collection from the manifest file for each task to be executed. Retrieve the object name of the object to be cached from the object collection. Create a thread pool. Select a single thread from the thread pool to send a cache request to s3. When the cache request for a single object is successfully sent, update the cache result of the object to complete. Retrieve the cache content generated based on the cache request. Generate a cache result list based on the cache content. Upload the cache result list to s3. Update the status of the selected task to complete. The step of selecting a single thread from the thread pool to send a cache request to s3 includes: Submit the cache request to the thread pool and check if there are any available workers in the thread pool. If a worker is available, assign the cache request task to the worker. After the worker processes the cache request, release the worker back into the cache pool. If no workers are available, then check if the number of workers in the thread pool exceeds the thread pool's capacity. If the thread pool capacity is exceeded, continue checking if there are non-blocking threads in the pool. If so, return null and continue checking if there are non-blocking threads. If not, wait for available workers. If the thread pool capacity is not exceeded, a new worker is created to handle the cache request. After the worker finishes handling the cache request, the worker is released into the cache pool. Step 4: Return to step 3 and continue until there are no tasks to be executed in the task queue.

2. The batch data unfreezing method based on Blu-ray recording scenario according to claim 1, characterized in that, The step of retrieving tasks from the task queue includes selecting the highest priority task from the task list for execution.

3. The batch data unfreezing method based on Blu-ray recording scenario according to claim 1, characterized in that, The creation of the thread pool includes calculating the number of threads in the thread pool based on the number of concurrent processors, the number of processor cores, memory, and network sockets on the user side.

4. The batch data unfreezing method based on Blu-ray recording scenario according to claim 1, characterized in that, Uploading the first object list to S3 includes uploading it to S3 via the AWS chunked upload protocol interface, which includes a chunked upload interface, an upload chunk interface, and a complete chunked upload interface.

Images