File conversion method and device based on double buffer area and storage medium

Abstract

The application discloses a file conversion method and device based on double buffer areas and a storage medium. The method is applied to a server configured with double buffer areas including an input queue and an output queue. The method comprises the following steps: splitting an asynchronous export task into a query subtask and a conversion subtask, configuring a write thread to execute the query subtask and a read thread to execute the conversion subtask; the write thread acquires original file data based on the query subtask, writes the original file data into the input queue, and sends a non-empty signal to the read thread; when receiving the non-empty signal, the read thread judges whether the output queue is empty; when the output queue is empty, the references of the input queue and the output queue are exchanged, the original file data is taken out from the exchanged output queue, and file conversion is performed. According to the application, the task is split and executed in parallel, the queue references are exchanged when the output queue is empty, and the read and write threads are synchronized through the non-empty signal, so that the idle waiting of the threads is reduced, and the file conversion process is accelerated.

Description

Technical Field

[0001] This application relates to the field of file conversion technology, and in particular to a file conversion method, device and storage medium based on dual buffers. Background Technology

[0002] Currently, many business systems need to support users in exporting large-scale data asynchronously. After a user submits an export request, the system performs time-consuming operations such as data querying and file generation in the background, and then provides the generated file for the user to download. In such asynchronous export systems, the query thread (write thread) is responsible for fetching raw data from downstream data sources in pages, while the transformation thread (read thread) is responsible for serializing the raw data into a target format file. Since query operations mainly rely on network I / O and database I / O, while transformation operations mainly rely on CPU computation and disk I / O, their processing speeds often differ, and their performance varies under different data volumes and system loads. Therefore, it is necessary to coordinate the execution rhythm of the two.

[0003] Existing solutions typically use a single cache to connect the query and transformation ends. The query thread writes data to the cache, and the transformation thread reads data from the same cache. However, when the transformation speed is lower than the query speed, the cache easily fills up, causing the query thread to be blocked and unable to fetch more data; when the transformation speed is higher than the query speed, the cache easily becomes empty, causing the transformation thread to idle and wait. Whether the cache is full or empty, there is always one thread in a waiting state, which cannot fully utilize the parallel processing capabilities of system resources, resulting in an increase in the overall task processing time. Summary of the Invention

[0004] The main purpose of this application is to provide a file conversion method, device and storage medium based on dual buffers, which aims to solve the technical problem that the existing method of using a single buffer results in a long time consumption of file data conversion tasks.

[0005] To achieve the above objectives, this application proposes a file conversion method based on a dual-buffer system. The method is applied to a server configured with a dual-buffer system, wherein the dual-buffer system includes an input queue and an output queue. The method includes: The asynchronous export task to be executed is split into a query subtask and a transformation subtask, and a write thread is configured to execute the query subtask and a read thread to execute the transformation subtask. The write thread obtains the original file data based on the query subtask, writes the original file data into the input queue, and sends a non-empty signal to the read thread. When the read thread receives the non-empty signal, it determines whether the output queue is non-empty. When the output queue is empty, the references of the input queue and the output queue are swapped to obtain the swapped output queue; When the output queue after the exchange is not empty, the read thread retrieves the original file data from the output queue after the exchange based on the conversion subtask and performs file conversion.

[0006] In one embodiment, the step of writing the original file data into the input queue includes: Determine whether the output queue is not empty and whether the input queue is not full; When the output queue is not empty and the input queue is not full, the original file data is written to the input queue by the write thread.

[0007] In one embodiment, after the step of determining whether the output queue is not empty and whether the input queue is not full, the method includes: When the output queue is empty or the input queue is full, control the write thread to wait on the non-full condition lock; The original file data is written to the input queue by the write thread until the input queue is not full. The write thread sends a non-null signal to the read thread to wake up the read thread that is waiting on the non-null condition lock.

[0008] In one embodiment, after the step of determining whether the output queue is non-empty when the read thread receives the non-empty signal, the method further includes: When the output queue is not empty, determine whether the input queue is not full; When the input queue is not full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion. When the input queue is full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion.

[0009] In one embodiment, after the step of swapping the references of the input queue and the output queue when the output queue is empty to obtain the swapped output queue, the method further includes: Determine whether the output queue after the exchange is empty; When the output queue after the exchange is empty, control the read thread to wait on a non-empty condition lock; Until the output queue after the exchange is not empty, the read thread retrieves the original file data from the output queue after the exchange based on the conversion subtask and performs file conversion.

[0010] In one embodiment, the step of retrieving the original file data and performing file conversion includes: Extract the original file data and store it in memory; The read thread sends a non-full signal to the write thread to wake up the write thread that is waiting on the non-full condition lock; The original file data is obtained from the memory, and file conversion is performed on the original file data.

[0011] In one embodiment, after the step of retrieving the original file data from the swapped output queue and performing file conversion by the read thread based on the conversion subtask when the swapped output queue is not empty, the method includes: The file conversion stops when the read thread identifies the end marker from the swapped output queue; The end marker is a marker that the write thread writes to the input queue after obtaining all the original file data corresponding to the query subtask.

[0012] In one embodiment, the method further includes: When there are no abnormalities in the write thread and the read thread, the step of obtaining the original file data through the write thread based on the query subtask is executed; When an abnormal situation is detected in the write thread or the read thread, the write thread and the read thread are controlled to stop executing the current operation, and a memory cleanup operation is performed to generate a current abnormal report; The memory cleanup operation includes at least one of the following: releasing the allocated memory buffer, closing the opened file output stream, and deleting temporary files generated during the conversion process.

[0013] Furthermore, to achieve the above objectives, this application also proposes a file conversion device based on a dual-buffer, the device comprising: a memory, a processor, and a file conversion program based on a dual-buffer stored in the memory and executable on the processor, the file conversion program based on a dual-buffer configured to implement the steps of the file conversion method based on a dual-buffer as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a file conversion program based on a dual buffer is stored, and when the file conversion program based on a dual buffer is executed by a processor, it implements the steps of the file conversion method based on a dual buffer as described above.

[0015] This application discloses a file conversion method, device, and storage medium based on a dual-buffer system. The method is applied to a server configured with a dual-buffer system, which includes an input queue and an output queue. The method includes: splitting an asynchronous export task to be executed into a query subtask and a conversion subtask, and configuring a write thread to execute the query subtask and a read thread to execute the conversion subtask; obtaining the original file data based on the query subtask using the write thread, writing the original file data into the input queue, and sending a non-empty signal to the read thread; determining whether the output queue is non-empty when the read thread receives the non-empty signal; swapping the references of the input queue and the output queue when the output queue is empty to obtain the swapped output queue; and retrieving the original file data from the swapped output queue based on the conversion subtask and performing file conversion using the read thread.

[0016] This application can split asynchronous export tasks into independent query subtasks and transformation subtasks, and configure write threads and read threads to execute in parallel, enabling data querying and file transformation to occur simultaneously, avoiding task blocking caused by speed mismatch in serial processing. Simultaneously, by configuring input and output queues to form a dual buffer, and exchanging queue references when the output queue is empty, write threads can continuously execute query operations, and read threads can continuously execute transformation operations, reducing idle waiting time caused by either thread being full or empty, and improving resource utilization. Furthermore, by using non-empty and non-full signals to achieve coordination and synchronization between read and write threads, ensuring orderly data flow, and by replacing physical data movement with queue reference exchange, the additional overhead of data movement is avoided, thereby accelerating the file transformation process and shortening the overall completion time of the asynchronous export task. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the file conversion method based on dual buffers in this application; Figure 2 This is a flowchart illustrating the second embodiment of the file conversion method based on dual buffers in this application; Figure 3 This is a flowchart illustrating the third embodiment of the file conversion method based on dual buffers in this application; Figure 4 A schematic diagram illustrating the entire process of writing threads for this application; Figure 5 This is a schematic diagram of the entire process of the read thread in this application; Figure 6 This is a schematic diagram of the file conversion device based on dual buffers in this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application provides a file conversion method based on dual buffers, referencing... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the file conversion method based on dual buffers according to this application. In this embodiment, the method includes steps S10 to S50: Step S10: Split the asynchronous export task to be executed into a query subtask and a transformation subtask, and configure the write thread to execute the query subtask and the read thread to execute the transformation subtask.

[0024] It should be noted that the method in this embodiment can be applied to asynchronous file export scenarios. For example, when a user submits a request to export large amounts of data files such as reports or order data through a front-end page, the asynchronous file export system can encapsulate the request as an asynchronous export task and execute it in the background without requiring the user to wait synchronously. The executing entity of this embodiment can be a computing electronic device with functions such as data processing, thread scheduling, and file reading and writing, such as the server of the asynchronous file export system, a mainframe computer, or a cloud computing node. Here, the server of the asynchronous file export system (referred to as "server") is used as an example to illustrate this embodiment and the following embodiments.

[0025] The server can be configured with dual buffers, which can include an input queue and an output queue. The input queue is used to receive raw file data written by the write thread, and the output queue is used to provide raw file data to be converted to the read thread. Both queues are implemented using a first-in, first-out (FIFO) data structure to ensure the ordered nature of data writing and reading.

[0026] It should also be noted that the input queue and the output queue can be two instances of the same data structure type. Their capacity is pre-configured when the server starts up to control the upper limit of memory usage and avoid memory overflow caused by excessive data volume.

[0027] In addition, considering that the read and write operations of the two queues involve concurrent access to shared resources, the server can also be configured with read-write locks, which include read locks and write locks. The read lock is a shared lock, and the write lock is a mutex lock, used to control concurrent access to the queues by different threads.

[0028] It should be understood that an asynchronous export task can be a background task that corresponds to a user's data export request. This asynchronous export task can include export conditions, such as time range, data source address, file format, etc.

[0029] In this embodiment, the asynchronous export task can be divided into two logically independent subtasks: a query subtask responsible for retrieving raw data from the data source, and a transformation subtask responsible for converting the raw data into a target file format (such as Excel, PDF, CSV, etc.). To enable the parallel execution of the two subtasks, the server can configure a write thread for the query subtask and a read thread for the transformation subtask.

[0030] In the implementation, after receiving an export request from a user, the server generates an asynchronous export task based on the export conditions and stores it in a task queue. The server retrieves this asynchronous export task from the queue using a task scheduler. Based on the task's type identifier, it determines that the task requires two phases: data query and file conversion. Therefore, the asynchronous export task is split into a query subtask and a conversion subtask, which are then bound to the execution logic of the write thread and read thread, respectively. The write thread acts as the producer thread, responsible for executing the query subtask; the read thread acts as the consumer thread, responsible for executing the conversion subtask. The two threads run in parallel after startup, cooperating to complete the entire export process.

[0031] Step S20: The write thread obtains the original file data based on the query subtask, writes the original file data into the input queue, and sends a non-empty signal to the read thread.

[0032] It should be noted that the non-empty signal can be a wake-up notification sent by the write thread to the read thread, used to inform the read thread that there is data available for consumption in the current dual buffers (i.e., the input queue and the output queue).

[0033] It should be understood that after the write thread starts, the server can initiate pagination query requests to downstream systems (such as databases, API interfaces, file systems, etc.) based on the query conditions (such as data source address, query statement, pagination parameters, etc.) carried in the query subtask, and retrieve the original file data page by page. After each batch of data is retrieved, the write thread obtains that batch of data and writes it as original file data into the input queue.

[0034] During the write process, the write thread can determine whether the input queue is not full by acquiring a read lock. If the input queue is not full, it holds the read lock and performs the write operation. After the write operation is complete, the write thread releases the lock it currently holds and generates a non-empty signal to send to the read thread, notifying the read thread that data is now available for consumption.

[0035] Step S30: When the read thread receives the non-empty signal, it determines whether the output queue is non-empty.

[0036] It should be noted that the read thread, as a consumer of the dual buffer, can be driven by the write thread via a non-null signal. After starting, the read thread is in a waiting-to-be-wake-up state. When it receives a non-null signal from the write thread, the read thread is awakened and begins to perform data consumption operations.

[0037] It should be understood that after receiving a non-empty signal, the read thread first checks the current state of the output queue, that is, determines whether the output queue is not empty. This determination is achieved by reading the current number of elements in the output queue or the queue status flag.

[0038] Understandably, the above judgment result can be used to determine the subsequent execution path of the read thread: if the output queue is not empty, the read thread continues to perform the data retrieval operation; if the output queue is empty, the subsequent queue exchange process is triggered.

[0039] Step S40: When the output queue is empty, swap the references of the input queue and the output queue to obtain the swapped output queue.

[0040] It should be noted that when the read thread detects that the output queue is empty, it indicates that there is no data available for consumption in the current output queue. To avoid the read thread entering an unnecessary idle state while waiting for data, this embodiment can introduce a queue reference swapping mechanism to interchange the references of the input queue and the output queue.

[0041] The swapping of references between the input and output queues involves assigning the memory address that originally pointed to the input queue to the reference variable in the output queue, and vice versa. This swapping operation only involves exchanging the values ​​of the two reference variables; it does not involve the physical movement of data already stored in the queues, meaning it does not perform the operation of taking data elements one by one from one queue and putting them into another.

[0042] It should be understood that after the swap is complete, the read thread can access the input queue (actually before the swap) through the reference variable of the output queue, and the write thread can access the output queue (actually before the swap) through the reference variable of the input queue. The time complexity of this swap operation is constant and is independent of the amount of data stored in the queue.

[0043] In the implementation, after determining that the output queue is empty, the read thread can acquire a write lock, and then, under the protection of the write lock, perform the operation of swapping the values ​​of the input queue reference variables and the output queue reference variables. By acquiring a write lock, the server can ensure that during the reference swapping process, the write thread will not simultaneously operate on the input queue (e.g., write new data to the input queue), thereby avoiding problems such as data ownership confusion or inconsistent reference pointers during the reference swapping process.

[0044] Step S50: When the output queue after the exchange is not empty, the original file data is retrieved from the output queue after the exchange by the read thread based on the conversion subtask and the file conversion is performed.

[0045] It should be noted that after the swap operation is complete, the read thread can check whether there is any consumable data in the new output queue. If the output queue after the swap is not empty, the read thread can retrieve the data from it and perform file conversion processing.

[0046] It should be understood that the above file conversion operations may include writing data rows to an Excel worksheet, formatting data records into a PDF table, or serializing data into a CSV text file. During the conversion process, the read thread can continuously consume the data in the output queue after the exchange through these file conversion operations until the output queue is empty.

[0047] In the specific implementation, after swapping the references of the input queue and the output queue, the read thread can check whether the swapped output queue is not empty. If it is not empty, the read thread retrieves the original file data one by one from the swapped output queue, temporarily stores the original file data in the memory buffer, then releases the aforementioned write lock, and finally writes the original file data sequentially into the target file stream according to the target file format specified in the conversion subtask, thus performing the file conversion operation.

[0048] Furthermore, in order to allow the write and read threads to exit when the asynchronous export task is completed, an end flag can be set to indicate that the original file data has been read completely. Therefore, after step S50, the following steps are also included: Step S60: When the read thread identifies the end marker from the swapped output queue, stop the file conversion.

[0049] It should be noted that, to indicate the completion status of the query subtask, the write thread can write a special end marker to the input queue after retrieving all the raw file data. This end marker is different from the ordinary raw file data and is used to notify the read thread that there is no more data available for consumption from the data source.

[0050] It should be understood that during the execution of the query subtask, after retrieving each batch of data, the write thread can determine whether there is any further data by checking if the query result is empty or if the total number of records has been reached. Once it is confirmed that all original file data has been retrieved, the write thread can write a predefined end-of-line marker object to the input queue after completing the writing of the last batch of data. This end-of-line marker resides in the same queue as the original file data and follows the same first-in, first-out (FIFO) order.

[0051] When the read thread retrieves data from the swapped output queue, it first checks whether each element is an end marker. If it is identified as an end marker, the read thread stops the file conversion operation, stops retrieving subsequent elements from the output queue, and enters the cleanup process. If the element retrieved by the read thread from the output queue is not an end marker, the file conversion operation is performed normally.

[0052] By embedding end markers in the data stream as described above, the server achieves precise connection between query subtasks and transformation subtasks, enabling the read thread to accurately identify the data source exhaustion state and complete the final processing in a timely manner. This avoids the problem of the read thread waiting for a long time or missing the last batch of data because it cannot determine whether all the data has arrived, and ensures the closed-loop execution of the asynchronous export task.

[0053] It should also be noted that after the read thread recognizes the end marker, it can close the target file stream that is currently being written, upload the converted file to the object storage system, thereby releasing local temporary file resources, and synchronously update the status of the asynchronous export task to complete, so that users can obtain the download link through front-end polling.

[0054] Furthermore, to ensure the stability of the asynchronous export task execution process, the device can also be equipped with an exception handling mechanism to monitor whether there are any abnormalities in the read and write threads in real time.

[0055] The abnormal situations mentioned may include, but are not limited to, at least one of the following: a write thread experiences a timeout or connection interruption in the downstream data source while executing a query subtask (e.g., database connection timeout, API interface unresponsive); a read thread encounters insufficient disk space or incorrect file write permissions while performing file conversion (e.g., target disk is full, target directory lacks write permissions); a read thread experiences a network interruption or object storage service authentication failure while performing file upload (e.g., network disconnection during upload, OSS AccessKey expires); or a write or read thread is triggered by an external interrupt signal (e.g., the server is performing graceful shutdown or the task is actively canceled by the user). When any of these abnormal situations occur, the server can consider that an abnormal situation has been detected.

[0056] Specifically, under normal operating conditions where neither the write thread nor the read thread experiences any abnormalities, the server can execute the asynchronous export task according to the process of steps S10 to S50 above, that is, continuously execute the step of obtaining the original file data based on the query subtask through the write thread and the step of performing file conversion through the read thread.

[0057] Upon detecting an anomaly in either the write thread or the read thread, the server immediately triggers an exception handling procedure: the thread experiencing the exception (write thread or read thread) changes the current state of the dual buffers (i.e., the input queue and the output queue) to a stopped state and notifies the other thread of the exception via inter-thread communication. The notified thread, upon sensing the state change, can proactively interrupt its currently executing operation.

[0058] Finally, when both the write thread and the read thread stop executing their current operations, the server can also perform a memory cleanup operation and generate a current exception report, marking the asynchronous export task as failed so that users can query the front end to find out the export failure result.

[0059] The memory cleanup operation may include at least one of the following: releasing the allocated memory buffer (clearing the unconsumed data in the input queue and output queue, and releasing the memory space occupied by the queue), closing the opened file output stream (ensuring that file stream resources are reclaimed in a timely manner and avoiding file handle leakage), and deleting temporary files generated during the conversion process (cleaning up incomplete temporary files left over due to abnormal interruption and releasing disk space).

[0060] It should be understood that through the above exception handling mechanism, the server can quickly interrupt task execution and clean up the environment when an exception occurs in the write thread or read thread, avoiding problems such as system memory leaks, file handle exhaustion, or disk space being filled due to unreleased resources, and ensuring the long-term stable operation of the server.

[0061] This embodiment breaks down the asynchronous export task into independent query and transformation subtasks, which are executed in parallel by write and read threads respectively. This allows the data query and file transformation stages to occur simultaneously without waiting for each other, avoiding task blocking caused by speed mismatch in single-threaded serial processing. Simultaneously, by configuring input and output queues to form a dual buffer and swapping queue references when the output queue is empty, seamless data switching is achieved. This ensures that read threads can continuously consume data and write threads can continuously write data, reducing thread idle waiting time and improving the resource utilization of the dual buffer and the overall efficiency of file transformation.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the file conversion method based on dual buffers in this application.

[0063] In this embodiment, considering that the write thread needs to determine the current state of the input queue and the output queue before writing the original file data to the input queue to determine whether the write operation can be performed, step S20 includes: steps S201~S202: Step S201: Determine whether the output queue is not empty and whether the input queue is not full.

[0064] It should be noted that before writing data to the input queue, the write thread can simultaneously check the status of both queues in the dual buffer. On one hand, it checks whether the output queue is not empty to confirm whether the read thread is currently consuming data. If the output queue is not empty, it means that the read thread is in a normal consumption state, and the write thread can continue to write data to the input queue without immediately triggering a queue swap. On the other hand, checking if the input queue is not full is used to confirm whether there is a free position available for writing in the input queue. If the input queue is not full, the write thread can write data to the input queue. When both of the above conditions are met, the write thread performs the write operation.

[0065] In the implementation, after the write thread obtains a batch of raw file data, it can first acquire a read lock on the current dual buffer. Under the protection of the read lock, it reads the status flag of the output queue and the current number of elements in the input queue. If the status flag of the output queue is not empty and the current number of elements in the input queue is less than a preset capacity threshold, then the condition is met.

[0066] Step S202: When the output queue is not empty and the input queue is not full, the original file data is written to the input queue by the write thread.

[0067] It's important to note that when the output queue is not empty and the input queue is not full, it indicates that the dual buffers are in a normal production-consumption state: the read thread is consuming data from the output queue, and the write thread can safely produce data to the input queue. The two threads do not interfere with each other and do not require additional synchronization control. At this time, data can be directly written to the input queue while holding a read lock; that is, the current batch of raw file data is written sequentially to the tail of the input queue while holding the read lock. Each time data is written, the current element count in the input queue is incremented by one. After all data in this batch of raw files has been written, the write thread releases the read lock.

[0068] In addition, the write thread can also wake up the read thread waiting on the non-null condition lock by sending a non-null signal to the read thread. Sending this non-null signal informs the read thread that there is data available for consumption in the current double buffer. Upon receiving this signal, the read thread switches from the waiting state to the ready state and performs a data retrieval operation based on the state of the output queue.

[0069] Furthermore, if the aforementioned conditions that the output queue is not empty and the input queue is not full are not met, after step S201, the process further includes: steps S203~S205: Step S203: When the output queue is empty or the input queue is full, control the write thread to wait on the non-full condition lock.

[0070] It's important to note that when the output queue is empty or the input queue is full, it indicates that the dual buffer is in an abnormal production-consumption state, and write threads cannot immediately perform write operations. If the output queue is empty, it means that read threads may be waiting because there is no data to consume. If write threads continue to write data to the input queue, data may accumulate in the queue and cannot be consumed in a timely manner. If the input queue is full, it means that there is no free space in the input queue, and write threads cannot continue to write new data to it. In the above situations, write threads can enter a waiting state until the current dual buffer state returns to a writable state.

[0071] In the implementation, after determining that the output queue is empty or the input queue is full, the write thread can first release the read lock previously acquired on the dual buffer, and then acquire a write lock on the current dual buffer. Under the protection of the write lock, it checks whether the input queue is full. This prevents data duplication or omission caused by read and write threads operating on the current dual buffer simultaneously when a queue reference swapping mechanism is subsequently introduced. When the input queue is full, the write thread can wait on a non-full condition lock.

[0072] Step S204: Until the input queue is not full, the original file data is written to the input queue by the write thread.

[0073] Step S205: The write thread sends a non-null signal to the read thread to wake up the read thread that is waiting on the non-null condition lock.

[0074] It should be noted that after a write thread waits on a non-full condition lock, it can be woken up by a read thread sending a non-full signal after consuming the data. When the write thread is woken up, the input queue has released a free position, and the write thread can retry the write operation.

[0075] It should also be noted that a non-full condition lock can be set with a corresponding lock wait time. If a write thread waits on a non-full condition lock and has not received a non-full signal by the time the wait time reaches the corresponding lock wait time, it can release the write lock to avoid indefinite waiting and process deadlock.

[0076] In the implementation, after retrieving data from the output queue, the read thread sends a non-full signal. Upon receiving this signal, the write thread is awakened from the non-full condition lock and checks again whether the input queue is non-full. If there is free space in the input queue, the write thread can hold the write lock and write the original file data into the input queue. After writing is complete, the write thread sends a non-empty signal, waking up the read thread waiting on the non-empty condition lock and notifying it that data is available for consumption. Subsequently, the write thread releases the write lock and continues to perform subsequent data query operations.

[0077] This embodiment adds a status judgment step before the write thread writes data, enabling the write thread to perform the write operation when the dual buffer states are appropriate. This avoids data backlog or write failures caused by blindly writing when the output queue is empty or the input queue is full. When the input queue is full, the write thread actively waits on a non-full condition lock until it receives a non-full signal and determines that the input queue is not full before executing the data write. This achieves coordination between the write and read threads, ensuring precise synchronization between them and guaranteeing the continuity and orderliness of data flow in the dual buffers.

[0078] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the file conversion method based on dual buffers in this application.

[0079] In this embodiment, to specifically illustrate the different execution paths of the read thread upon receiving a non-empty signal, after step S30, the following steps are also included: S41~S43: Step S41: When the output queue is not empty, determine whether the input queue is not full.

[0080] It should be noted that when the read thread determines that the output queue is not empty, it means that there is data available for consumption in the current output queue, and at this time the read thread can perform data retrieval operations.

[0081] It should be understood that, since the input queue and output queue together form the current dual buffer, a write thread may wait on a non-full condition lock because the input queue is full. Before retrieving data, the read thread can further check the current state of the input queue to determine whether it needs to wake up the write thread after retrieving the data.

[0082] In the specific implementation, after determining that the output queue is not empty, the read thread can further determine whether the input queue is not full: read the current number of elements in the input queue and compare it with a preset capacity threshold. If the current number of elements in the input queue is less than the preset capacity threshold, the input queue is determined to be not full; if the current number of elements is equal to the preset capacity threshold, the input queue is determined to be full.

[0083] Step S42: When the input queue is not full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion.

[0084] It should be noted that when the input queue is not full, it indicates that the write thread is not currently blocked on the non-full condition lock and can continue to write data to the input queue. Therefore, the read thread can perform only data retrieval and file conversion operations without needing to wake up the write thread.

[0085] It should be understood that after determining that the input queue is not full, the read thread can use a read lock to retrieve the original file data from the head of the output queue. After retrieving one piece of data, the current number of elements in the output queue is decremented by one, while the current number of elements in the input queue remains unchanged. Then, the read thread can write the retrieved original file data into the target file stream according to the target file format specified in the conversion subtask, thus performing the file conversion operation.

[0086] It should also be noted that the read thread performs a transformation once for each piece of data it retrieves, and repeats the above "retrieve-transform" loop until the output queue is empty or the read thread is interrupted by other conditions.

[0087] Step S43: When the input queue is full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion.

[0088] It's important to note that when the input queue is full, it indicates that a write thread may be waiting for a free space on a non-full condition lock. After a read thread retrieves data from the output queue, a free space is created in the output queue. Since queue swaps may occur subsequently, the storage space corresponding to this free space can be allocated to the input queue after the swap. Therefore, the read thread can first acquire a write lock, and then use the write lock to retrieve data from the output queue to perform file conversion.

[0089] Specifically, the read thread can first hold a write lock to retrieve the original file data from the output queue and store it in a memory buffer. After retrieving the data, it sends a non-full signal to the write thread to wake up the write thread waiting on the non-full condition lock. Then, it releases the aforementioned write lock and finally retrieves the original file data from the memory buffer to perform file conversion.

[0090] In this way, when the input queue is full, the read thread can wake up the write thread in time after retrieving the data, so that the write thread can resume writing data as soon as a free position appears in the input queue, avoiding the write thread waiting for a long time.

[0091] Furthermore, in this embodiment, if the read thread determines that the output queue is not empty and the input queue is not full when performing the aforementioned judgment on whether the input queue is not full, the read thread can also selectively send a non-full signal after retrieving the data, and then perform file conversion on the retrieved file data after sending the non-full signal. Since the input queue itself is not full and the write thread is already running normally, the non-full signal may not necessarily cause the write thread to be awakened (the write thread is not waiting on the non-full condition lock). Therefore, the read thread can omit sending the non-full signal to save unnecessary inter-thread communication overhead. This embodiment does not restrict whether to send a non-full signal in this case.

[0092] Furthermore, when the output queue is determined to be empty in step S30, the read thread can first acquire a write lock and then perform a reference swap between the input queue and the output queue when the output queue is empty. After the swap, the original input queue becomes the new output queue. Considering that the output queue after the swap can also be empty, after step S40, steps S51~S53 are also included: Step S51: Determine whether the output queue after the exchange is empty.

[0093] It should be noted that, considering that the output queue after the swap may contain data (i.e., the original input queue still contains data previously written by the write thread) or may still be empty (i.e., the write thread has not yet written data, or the data in the original input queue has been consumed), the read thread can reconfirm the state of the output queue after the swap in order to determine the next execution path.

[0094] In the specific implementation, after the read thread completes the reference swap between the input queue and the output queue, it can read the current number of elements in the swapped output queue. If the current number of elements is greater than zero, the swapped output queue is determined to be non-empty; if the current number of elements is equal to zero, the swapped output queue is determined to be empty.

[0095] Step S52: When the output queue after the exchange is empty, control the read thread to wait on a non-empty condition lock.

[0096] It should be noted that if the output queue is still empty after the swap, it indicates that there is no data available for consumption in the input queue, meaning that the write thread has not yet written data to the input queue or there is indeed no data in the double buffer. At this time, the read thread cannot perform data retrieval operations and can enter a waiting state until the write thread writes data to the input queue and sends a non-empty signal to wake it up.

[0097] In the specific implementation, after determining that the output queue after the swap is empty, the read thread can perform a waiting operation on the non-empty condition lock until it receives a non-empty signal, at which point the read thread is awakened from the non-empty condition lock.

[0098] Step S53: Until the output queue after the exchange is not empty, the original file data is retrieved from the output queue after the exchange by the read thread based on the conversion subtask and the file conversion is performed.

[0099] It should be noted that after a read thread is woken up on a non-null condition lock, it can re-verify whether there is data in the swapped output queue to avoid invalid operations caused by false wake-ups.

[0100] It should also be noted that non-null condition locks can also be configured with a corresponding lock wait time. If a read thread waits on a non-null condition lock and has not received a non-null signal by the time the wait time has elapsed, it can release the aforementioned write lock, thus preventing indefinite waiting and potential process deadlock.

[0101] It should be understood that after receiving a non-empty signal from the write thread, the read thread is awakened from the non-empty condition lock and checks again whether the swapped output queue is non-empty. If the swapped output queue is not empty, the read thread holds the write lock and retrieves the original file data from the swapped output queue so that it can subsequently perform file conversion operations according to the conversion subtask. If the swapped output queue is still empty, the read thread waits again on the non-empty condition lock until it is awakened again and the queue is confirmed to be non-empty.

[0102] Since the aforementioned output queue was empty, triggering the swap, and the read thread had already acquired the write lock and completed the swap operation in step S40, the read thread can perform the retrieval operation under the protection of the write lock in this step. The retrieved data is the data that the write thread wrote to the input queue before the swap.

[0103] Furthermore, after a read thread retrieves data from the swapped output queue, the output queue can release free space. Since subsequent queue swaps can occur again, the storage space corresponding to the free space after another swap can be allocated to the input queue. Therefore, after retrieving data, a read thread can send a non-full signal to notify write threads that may be waiting on a non-full condition lock that there may be free space in the input queue.

[0104] In the specific implementation, after retrieving the original file data from the swapped output queue, the read thread can first store the original file data in a memory buffer and send a non-full signal to the write thread to wake up the write thread waiting on the non-full condition lock. Then, the aforementioned write lock is released, and finally the original file data is retrieved from the memory buffer to perform file conversion.

[0105] Upon receiving the non-full signal, the write thread can be woken up from the non-full condition lock, reacquire the write lock, and check if the input queue is non-full. If there is free space in the input queue, the write thread writes the retrieved raw file data into the input queue; if the input queue is still full, the write thread continues to wait. This allows the write thread to promptly detect potential free spaces in the input queue, maximizing the data flow efficiency of the current dual buffers.

[0106] This embodiment further determines the state of the input queue when the output queue is not empty, and decides whether to send a non-full signal to wake up the write thread based on whether the input queue is full. This allows the write thread to resume data writing as soon as an empty position appears in the input queue, avoiding prolonged waiting due to the write thread's inability to detect changes in the input queue state, thus improving the data flow efficiency of the dual buffers. Simultaneously, the read thread reconfirms whether the output queue is empty after a queue swap, only performing a retrieval operation if data is actually present. This avoids invalid operations caused by the output queue remaining empty after the swap, reducing unnecessary resource consumption. It also enables the write thread to promptly detect potential empty positions in the input queue and resume data writing, maximizing the data flow efficiency of the dual buffers and ensuring the continuous parallel execution of the query and transformation subtasks.

[0107] Furthermore, this can be referenced here. Figure 4 and Figure 5 This application describes the file conversion method based on dual buffers. Figure 4 A schematic diagram illustrating the entire process of writing threads for this application. Figure 5 This is a schematic diagram of the entire process of the read thread in this application.

[0108] like Figure 4 As shown, after the write thread starts, it first retrieves the raw file data of the current batch from the downstream system, then acquires a read lock and checks whether the output queue is not empty and the input queue is not full. If the conditions are met, it holds the read lock and writes the data to the input queue, releases the read lock, and checks whether there is a next batch of data. If there is, it continues to retrieve the next batch of data and repeats the above process. If the conditions are not met, it releases the read lock and acquires a write lock, checks whether the input queue is full. If it is not full, it holds the write lock and writes the data to the input queue, sends a non-empty signal to wake up the read thread, and releases the write lock. If the input queue is full, it waits on the non-full condition lock until it receives a non-full signal and is woken up, at which point it holds the write lock and writes the data to the input queue. When all batches of raw file data have been written, the write thread writes an end marker to the input queue and sends a non-empty signal to wake up the read thread, after which the write thread terminates.

[0109] like Figure 5 As shown, after the read thread starts, it first checks whether the output queue is not empty. If the output queue is not empty, it further checks whether the input queue is not full. If the input queue is not full, it retrieves data from the output queue and then checks whether there is an end marker in the retrieved data; if the input queue is full, it acquires a write lock, retrieves data from the output queue while holding the write lock, sends a non-full signal to wake up the write thread, releases the write lock, and then checks whether there is an end marker in the retrieved data.

[0110] If the output queue is empty, a write lock is acquired, the references to the input and output queues are swapped, and then it is checked whether the swapped output queue is empty. If it is empty, the thread waits on a non-empty condition lock until a non-empty signal is received, then swaps the references to the input and output queues again, and returns to the previous step of checking whether the swapped output queue is empty, until the output queue is non-empty; if it is non-empty, the thread holds the write lock, retrieves data from the swapped output queue, sends a non-full signal to wake up the write thread, and after releasing the write lock, checks whether an end marker exists in the retrieved data.

[0111] After determining whether an end marker exists in the retrieved data, if the result is that no end marker exists, file conversion can be performed on the retrieved data, and after the conversion is completed, the process can return to the step of determining whether the output queue is not empty to continue the read thread process; if the result is that an end marker exists, file conversion can be stopped and the read thread can be terminated.

[0112] This application enables data querying and file conversion to occur simultaneously by executing write and read threads in parallel, avoiding task blocking. Through a queue reference swapping mechanism, the buffer is quickly switched when the output queue is empty, reducing thread idle waiting and improving resource utilization. Furthermore, the combination of read-write locks and condition locks ensures precise synchronization of read and write threads, guaranteeing orderly data flow, improving file conversion efficiency, and ultimately shortening the completion time of asynchronous export tasks.

[0113] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the file conversion method based on dual buffers in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0114] This application also provides a file conversion device based on a dual-buffer, the file conversion device based on a dual-buffer includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the file conversion method based on a dual-buffer in the first embodiment described above.

[0115] The following is for reference. Figure 6 , Figure 6 This is a schematic diagram of the file conversion device based on dual buffers according to this application. The file conversion device based on dual buffers in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and fixed terminals such as digital TVs and desktop computers. Figure 6 The file conversion device based on dual buffers shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0116] like Figure 6As shown, a dual-buffered file conversion device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the dual-buffered file conversion device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the dual-buffered file conversion device to communicate wirelessly or wiredly with other devices to exchange data. Although dual-buffered file conversion devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0117] The file conversion device based on dual buffers provided in this application, employing the file conversion method based on dual buffers in the above embodiments, can solve the technical problems of file conversion methods based on dual buffers. Compared with the prior art, the beneficial effects of the file conversion device based on dual buffers provided in this application are the same as those of the file conversion method based on dual buffers provided in the above embodiments, and other technical features in this file conversion device based on dual buffers are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0118] This application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the file conversion method based on dual buffers in the above embodiments.

[0119] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0120] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described file conversion method based on dual buffers, thereby solving the technical problems of the file conversion method based on dual buffers. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the file conversion method based on dual buffers provided in the above embodiments, and will not be repeated here.

[0121] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other elements in the process, method, article, or system that includes that element.

[0122] The above embodiment numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and do not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A file conversion method based on dual buffers, characterized in that, The method is applied to a server configured with dual buffers, wherein the dual buffers include an input queue and an output queue, and the method includes: The asynchronous export task to be executed is split into a query subtask and a transformation subtask, and a write thread is configured to execute the query subtask and a read thread to execute the transformation subtask. The write thread obtains the original file data based on the query subtask, writes the original file data into the input queue, and sends a non-empty signal to the read thread. When the read thread receives the non-empty signal, it determines whether the output queue is non-empty. When the output queue is empty, the references of the input queue and the output queue are swapped to obtain the swapped output queue; When the output queue after the exchange is not empty, the read thread retrieves the original file data from the output queue after the exchange based on the conversion subtask and performs file conversion.

2. The method as described in claim 1, characterized in that, The step of writing the original file data into the input queue includes: Determine whether the output queue is not empty and whether the input queue is not full; When the output queue is not empty and the input queue is not full, the original file data is written to the input queue by the write thread.

3. The method as described in claim 2, characterized in that, After the step of determining whether the output queue is not empty and whether the input queue is not full, the following steps are included: When the output queue is empty or the input queue is full, control the write thread to wait on the non-full condition lock; The original file data is written to the input queue by the write thread until the input queue is not full. The write thread sends a non-null signal to the read thread to wake up the read thread that is waiting on the non-null condition lock.

4. The method as described in claim 1, characterized in that, After the step of determining whether the output queue is non-empty when the read thread receives the non-empty signal, the method further includes: When the output queue is not empty, determine whether the input queue is not full; When the input queue is not full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion. When the input queue is full, the read thread retrieves the original file data from the output queue based on the conversion subtask and performs file conversion.

5. The method as described in claim 1, characterized in that, After the step of swapping the references of the input queue and the output queue when the output queue is empty to obtain the swapped output queue, the method further includes: Determine whether the output queue after the exchange is empty; When the output queue after the exchange is empty, control the read thread to wait on a non-empty condition lock; Until the output queue after the exchange is not empty, the read thread retrieves the original file data from the output queue after the exchange based on the conversion subtask and performs file conversion.

6. The method as described in claim 4 or 5, characterized in that, The steps of retrieving the original file data and performing file conversion include: Extract the original file data and store it in memory; The read thread sends a non-full signal to the write thread to wake up the write thread that is waiting on the non-full condition lock; The original file data is obtained from the memory, and file conversion is performed on the original file data.

7. The method as described in claim 1, characterized in that, After the step of retrieving the original file data from the swapped output queue and performing file conversion by the read thread based on the conversion subtask when the swapped output queue is not empty, the process includes: The file conversion stops when the read thread identifies the end marker from the swapped output queue; The end marker is a marker that the write thread writes to the input queue after obtaining all the original file data corresponding to the query subtask.

8. The method as described in claim 1, characterized in that, The method further includes: When there are no abnormalities in the write thread and the read thread, the step of obtaining the original file data through the write thread based on the query subtask is executed; When an abnormal situation is detected in the write thread or the read thread, the write thread and the read thread are controlled to stop executing the current operation, and a memory cleanup operation is performed to generate a current abnormal report; The memory cleanup operation includes at least one of the following: releasing the allocated memory buffer, closing the opened file output stream, and deleting temporary files generated during the conversion process.

9. A file conversion device based on dual buffers, characterized in that, The device includes: a memory, a processor, and a dual-buffered file conversion program stored on the memory and executable on the processor, the dual-buffered file conversion program being configured to implement the steps of the dual-buffered file conversion method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium stores a file conversion program based on a dual-buffer, which, when executed by a processor, implements the steps of the file conversion method based on a dual-buffer as described in any one of claims 1 to 8.

Images