A data reading method, apparatus, electronic device, and storage medium

Abstract

This invention provides a data reading method, apparatus, electronic device, and storage medium. The method includes: receiving a data reading request; determining whether the data file corresponding to the identifier of the data file to be read has been completely read; if not, determining whether the current cache contains readable target data; if not, determining the number of data objects to be read in the target cache queue; if the number is less than a preset threshold, asynchronously loading the data objects to be read from the data file to be read into the target cache queue, and returning to execute the step of determining whether the current cache contains readable target data; if the number is not zero, updating the data in the current cache according to the status information of the first data object to be read in the target cache queue, and returning to execute the step of determining whether the current cache contains readable target data. This invention can improve the efficiency of data reading.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data reading method, apparatus, electronic device, and storage medium. Background Technology

[0002] When a client needs to read local or remote files, it typically does so via file streams, processing a finite amount of data at a time. When the target file is large, the number of file accesses required increases. Each read operation involves an I / O (Input / Output) transfer, which takes time, resulting in a long read time and low efficiency. For example, if the target file is 1MB and each read operation involves 1024 bytes, it requires 1024 file accesses and 1024 I / O transfers to read the entire file.

[0003] In related technologies, a technical solution to improve data reading efficiency is to set up a cache of a certain storage size. When a client needs to read a local or remote file, the target file data corresponding to the cache size is first read into the cache, and then the data is read from the cache. This reduces the number of I / O transfers, thereby reducing data reading time and improving data reading efficiency. Specifically, for example, if the target file size is 1MB, and each read operation is 1 byte, and the cache size is 1024 bytes, then 1024 bytes of target file data are read into the cache at a time, and then the data is read byte by byte from the cache. Without a cache, the number of I / O transfers would be 1024 × 1024, while with a cache, the number of I / O transfers is reduced to 1024, significantly reducing the time spent on I / O transfers.

[0004] However, the data reading in the aforementioned technologies suffers from synchronous blocking. When the buffer is empty or all the data in the buffer has been read, it is necessary to wait for the buffer to read (fill) new data, that is, it is necessary to wait for the buffer to read 1024 bytes of data before further data reading can be carried out. When the target file data is large, such as 1G, it is also necessary to wait for the time consumed by 1024 I / O transfers before the target file data can be read. The data reading efficiency is low. Summary of the Invention

[0005] The purpose of this invention is to provide a data reading method, apparatus, electronic device, and storage medium to improve data reading efficiency. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a data reading method, the method comprising:

[0007] Receive a data read request, wherein the data read request contains an identifier of the data file to be read;

[0008] Determine whether the data file corresponding to the identifier of the data file to be read has been completely read;

[0009] If the data file to be read has not been completely read, determine whether the current cache area corresponding to the data file to be read contains readable target data, wherein the readable target data is a portion of the data file to be read.

[0010] If the current cache does not contain readable target data, then the number of data objects to be read contained in the target cache queue is determined, wherein the data objects to be read are part of the data of the data file to be read, and the target cache queue is the cache queue corresponding to the current cache.

[0011] If the number is less than a preset threshold, the data objects to be read in the data file to be read are asynchronously loaded into the target cache queue, and the process returns to the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0012] If the number is not zero, update the data in the current cache area according to the status information of the first data object to be read in the target cache queue, and return to the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0013] Optionally, the step of asynchronously loading the data objects to be read from the data file to the target cache queue when the number is less than a preset threshold includes:

[0014] If the number is less than a preset threshold, the number of tasks to be executed corresponding to asynchronous data loading is determined according to a preset strategy. The preset strategy is determined based on the size of the target cache queue and the number of data objects to be read contained in the target cache queue.

[0015] The task to be executed in the data file to be read is loaded into several data objects to be read, and the loaded data objects to be read are stored sequentially in the target cache queue.

[0016] Optionally, the preset strategy is:

[0017] The minimum value between the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, is determined as the number of tasks to be executed.

[0018] Optionally, the step of updating the data in the current cache based on the state information of the first data object to be read in the target cache queue when the number is not zero includes:

[0019] If the number is not zero, determine the status information of the first data object to be read in the target cache queue;

[0020] If the status information is not a loading failure message, the first data object to be read is updated in the current cache area, and the first data object to be read is deleted from the target cache queue.

[0021] Optionally, the method further includes:

[0022] If the status information indicates a loading failure, return a message indicating a data reading failure and clear the current cache.

[0023] Optionally, the step of updating the first data object to be read to the current cache and deleting the first data object to be read from the target cache queue when the status information is not a loading failure message includes:

[0024] If the status information indicates successful loading, the first data object to be read is updated in the current cache area, and the first data object to be read is deleted from the target cache queue.

[0025] If the status information is "no data loaded", wait for the data in the first data object to be read to be loaded, then update the first data object to be read to the current cache area, and delete the first data object to be read from the target cache queue.

[0026] Optionally, after determining whether the current cache corresponding to the data file to be read contains readable target data, the method further includes:

[0027] If the current cache contains readable target data, then the readable target data is read from the current cache and the read target data is returned, and the step of determining whether the current cache corresponding to the data file to be read contains readable target data is returned.

[0028] Optionally, after determining whether the data file corresponding to the identifier of the data file to be read has been completely read, the method further includes:

[0029] Once the data file to be read has been read, clear the current cache area.

[0030] Secondly, embodiments of the present invention provide a data reading device, the device comprising:

[0031] The receiving module is used to receive data read requests, wherein the data read requests contain an identifier of the data file to be read;

[0032] The first judgment module is used to determine whether the data file to be read corresponding to the identifier of the data file to be read has been read completely.

[0033] The second judgment module is used to determine whether the current cache area corresponding to the data file to be read contains readable target data when the first judgment module determines that the data file to be read has not been read completely. The readable target data is a part of the data file to be read.

[0034] The determining module is used to determine the number of data objects to be read in the target cache queue when the second determining module determines that the current cache does not contain readable target data, wherein the data objects to be read are part of the data of the data file to be read, and the target cache queue is the cache queue corresponding to the current cache.

[0035] The data loading module is used to asynchronously load the data objects to be read from the data file to be read into the target cache queue when the number is less than a preset threshold, and to trigger the second judgment module to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0036] The data update module is used to update the data in the current cache area according to the status information of the first data object to be read in the target cache queue when the number is not zero, and to trigger the second judgment module to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0037] Optionally, the data loading module is specifically used for:

[0038] If the number is less than a preset threshold, the number of tasks to be executed corresponding to asynchronous data loading is determined according to a preset strategy. The preset strategy is determined based on the size of the target cache queue and the number of data objects to be read contained in the target cache queue.

[0039] The task to be executed in the data file to be read is loaded into several data objects to be read, and the loaded data objects to be read are stored sequentially in the target cache queue.

[0040] Optionally, the preset strategy is:

[0041] The minimum value between the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, is determined as the number of tasks to be executed.

[0042] Optionally, the data update module includes:

[0043] The determination submodule is used to determine the status information of the first data object to be read in the target cache queue when the number is not zero;

[0044] The first update submodule is used to update the first data object to be read into the current cache area and delete the first data object to be read from the target cache queue when the status information is not a loading failure information.

[0045] Optionally, the data update module further includes:

[0046] The second update submodule is used to return a data reading failure message and clear the current cache when the status information is a loading failure message.

[0047] Optionally, the first update submodule is specifically used for:

[0048] If the status information indicates successful loading, the first data object to be read is updated in the current cache area, and the first data object to be read is deleted from the target cache queue.

[0049] If the status information is "no data loaded", wait for the data in the first data object to be read to be loaded, then update the first data object to be read to the current cache area, and delete the first data object to be read from the target cache queue.

[0050] Optionally, the device further includes:

[0051] The data reading module is used to read the readable target data from the current cache area and return the read target data when the second judgment module determines that the current cache area contains readable target data, and to trigger the second judgment module to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0052] Optionally, the device further includes:

[0053] The clearing module is used to clear the current cache area when the first judgment module determines that the data file to be read has been read.

[0054] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0055] Memory, used to store computer programs;

[0056] When a processor executes a program stored in memory, it implements the steps of the data reading method described in the first aspect above.

[0057] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data reading method described in the first aspect.

[0058] Beneficial effects of the embodiments of the present invention:

[0059] This invention provides a data reading method, apparatus, electronic device, and storage medium. It sets a target cache queue corresponding to the current cache area, asynchronously loads multiple data objects to be read into the target cache queue, and achieves asynchronous non-blocking data reading. Furthermore, even when the current cache area does not contain readable target data, data is directly read from the target cache queue. Compared to existing technologies, this only requires waiting for the time consumed by one I / O transfer during the first data read to complete the data reading of the data file to be read, reducing client data reading waiting time, decreasing the overall data file reading time, and improving data reading efficiency.

[0060] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating a data reading method provided in an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of a cache distribution provided in an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of a data reading process provided in an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the structure of a data reading device provided in an embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] To address the problems of long waiting times for I / O transmissions and low data reading efficiency in existing data reading processes, embodiments of the present invention provide a data reading method, apparatus, electronic device, and storage medium.

[0069] An embodiment of the present invention provides a data reading method, which may include:

[0070] Receive a data read request, wherein the data read request contains an identifier of the data file to be read;

[0071] Determine whether the data file corresponding to the identifier of the data file to be read has been completely read;

[0072] If the data file to be read has not been completely read, determine whether the current cache area corresponding to the data file to be read contains readable target data, wherein the readable target data is a portion of the data file to be read.

[0073] If the current cache does not contain readable target data, then the number of data objects to be read contained in the target cache queue is determined, wherein the data objects to be read are part of the data of the data file to be read, and the target cache queue is the cache queue corresponding to the current cache.

[0074] If the number is less than a preset threshold, the data objects to be read in the data file to be read are asynchronously loaded into the target cache queue, and the process returns to the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0075] If the number is not zero, update the data in the current cache area according to the status information of the first data object to be read in the target cache queue, and return to the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0076] This invention provides a data reading method that sets a target cache queue corresponding to the current cache area, asynchronously loads multiple data objects to be read into the target cache queue, and realizes asynchronous non-blocking data reading. Thus, even if the current cache area does not contain readable target data, data is directly read from the target cache queue. Compared with the prior art, it only needs to wait for the time consumed by one I / O transmission during the first data reading to complete the data reading of the data file to be read, reducing the client's data reading waiting time, reducing the overall data file reading time, and improving the efficiency of data reading.

[0077] The following is a detailed description of a data reading method provided by an embodiment of the present invention:

[0078] The data reading method provided in this embodiment of the invention can be applied to client devices or server devices, or it can be implemented through a streaming reading acceleration component of a client device or server device, wherein the streaming reading acceleration component is a data reading component implemented through the data reading method provided in this embodiment of the invention.

[0079] like Figure 1 As shown, this embodiment of the invention provides a reading processing method, which may include the following steps:

[0080] S101, Receive data read request.

[0081] This embodiment of the invention is illustrated using a client device as an example. When the client needs to read a local or remote file (which may be a data file on a server device), it receives a data read request sent by the user. This data read request may include the identifier of the data file to be read, the size of the data file to be read, and the amount of data to be read each time. The data file to be read is either a local or remote file. For example, the amount of data to be read each time may be one byte, or 10 bytes, etc.

[0082] S102, determine whether the data file corresponding to the identifier of the data file to be read has been read completely.

[0083] If the data file corresponding to the identifier of the data file to be read has not been read completely, step S103 can be executed to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0084] Once the data file corresponding to the identifier of the data file to be read has been read, the current cache area corresponding to the data file to be read can be cleared.

[0085] S103, determine whether the current cache corresponding to the data file to be read contains readable target data.

[0086] In this embodiment of the invention, a corresponding current cache area can be set for the data file to be read. The current cache area can correspond to a corresponding cache queue, i.e., a target cache queue. The target cache queue can be a pre-generated cache queue of a certain length, all used to cache the data to be read in the data file. The current cache area can be a memory region, etc. The readable target data is a portion of the data in the data file to be read. For example, such as... Figure 2 As shown, the size of the data file to be read is 1KB, and part of the data to be read is cached in the current cache area. The length of the target cache queue can be set according to actual needs, for example, it can be 5, 10, or 20, etc.

[0087] Upon receiving a data read request for a file to be read, and before the file has been fully read, it can be further determined whether the current cache corresponding to the file contains readable target data. This readable target data refers to the readable data of the file to be read cached in the current cache; that is, determining whether the current cache contains readable data of the file to be read. For example, the readable target data can be determined using the difference between the length of all the data to be read cached in the current cache and the position of the currently read data within the length of all the data to be read. A difference of zero indicates that the current cache does not contain readable target data, while a difference that is not zero indicates that the current cache contains readable target data.

[0088] Specifically, if the current cache does not yet have any readable data of the data file to be read, or if the data of the data file to be read cached in the current cache has been read completely and there is no more readable data, then step S104 can be executed.

[0089] If the data in the cached data file to be read has not been completely read, that is, if it still contains readable target data, read the readable target data from the current cache, return the read target data, and return to the step of determining whether the current cache corresponding to the data file to be read contains readable target data.

[0090] If the data in the current buffer is not yet fully read, the system can continue reading readable target data from the current buffer. For example, if the current buffer contains 1024 bytes of data to be read from the data file, and one byte is read at a time, and the current buffer has read the 100th byte, then the current buffer still contains readable target data from the 101st to the 1023rd byte. In this case, the system reads the readable target data corresponding to the 101st byte from the current buffer, returns the read target data, and returns to the previous step of checking whether the current buffer contains readable target data, until all readable target data in the current buffer is read, or all data in the entire data file is read.

[0091] S104, if the current cache does not contain readable target data, determine the number of data objects to be read contained in the target cache queue.

[0092] If the current cache does not contain readable target data, it means that the data to be read from the data file to be read has not yet been cached in the current cache, or the data to be read from the data file to be read cached in the current cache has been read. At this time, the number of data objects to be read contained in the target cache queue can be further determined. The data objects to be read are part of the data of the data file to be read, and the target cache queue is the cache queue corresponding to the current cache.

[0093] The target cache queue can be a first-in, first-out queue responsible for storing the data objects to be read. The target cache queue can store the data objects to be read, along with their corresponding start and end positions, and status information. The start and end positions of the data objects can be the start and end positions of the data objects within the data file to be read, and the status information can be the loading status information of the data objects.

[0094] S105, if the number is less than the preset threshold, asynchronously load the data objects to be read from the data file to be read into the target cache queue, and return to execute the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0095] If the current cache does not contain readable target data, and the number of data objects to be read in the target cache queue is less than a preset threshold, the remaining data objects to be read in the data file to be read can be asynchronously loaded into the target cache queue, and the process returns to step S103 to determine whether the current cache corresponding to the data file to be read contains readable target data. Specifically, asynchronous loading of data objects to be read can be implemented using thread pool technology or multi-threaded parallel technology.

[0096] S106, if the number is not zero, update the data in the current cache area according to the status information of the first data object to be read in the target cache queue, and return to the step of determining whether the current cache area corresponding to the data file to be read contains readable target data.

[0097] If the current cache does not contain readable target data, but the number of data objects to be read in the target cache queue is not zero, it indicates that there is no readable target data in the current cache, but there are still data objects to be read cached in the target cache queue. In this case, the status information of the first data object to be read in the target cache queue can be determined first, and then the data in the current cache can be updated according to the status information of the first data object to be read in the target cache queue. Then, the execution step S103 is returned to determine whether the current cache corresponding to the data file to be read contains readable target data.

[0098] This invention provides a data reading method that sets a target cache queue corresponding to the current cache area, asynchronously loads multiple data objects to be read into the target cache queue, and realizes asynchronous non-blocking data reading. Thus, even if the current cache area does not contain readable target data, data is directly read from the target cache queue. Compared with the prior art, it only needs to wait for the time consumed by one I / O transmission during the first data reading to complete the data reading of the data file to be read, reducing the client's data reading waiting time, reducing the overall data file reading time, and improving the efficiency of data reading.

[0099] As an optional implementation of this invention, if the number of data objects in step S105 is less than a preset threshold, the data objects to be read in the data file to be read are asynchronously loaded into the target cache queue. This can be achieved in the following way:

[0100] If the number is less than a preset threshold, the number of tasks to be executed corresponding to asynchronous data loading is determined according to a preset strategy. The preset strategy is determined based on the size of the target cache queue and the number of data objects to be read contained in the target cache queue.

[0101] Load several data objects to be read from the data file to be executed, and store the loaded data objects sequentially in the target cache queue.

[0102] Preferably, the preset strategy can be: the minimum value between the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, is determined as the number of tasks to be executed. The preset threshold can be the same as or different from the number of tasks to be executed.

[0103] In practical applications, when the number of data objects to be read in the target cache queue is less than a preset threshold, thread pool technology can be used to determine the number of tasks to be executed corresponding to asynchronous data loading according to a preset strategy. Then, the number of data objects to be read in the data file to be read is loaded and the loaded data objects to be read are stored sequentially in the target cache queue.

[0104] Specifically, if the number of data objects to be read in the target cache queue is less than a preset threshold, an asynchronous data loading task can be sent to the thread pool.

[0105] In this embodiment of the invention, a thread pool can be pre-created, which may contain a certain number of threads for preloading data from a data file to be read and storing the data objects to be read from the loaded data file in a target cache queue. The number of threads in the thread pool can be set according to actual needs.

[0106] When initially reading data from the current cache, both the current cache and the target cache queue are empty. At this point, asynchronous data loading is initiated, sending an asynchronous data loading task to the thread pool. This process requires waiting for the time consumed by one I / O transfer to retrieve the data object to be read. Once the data to be read from the current cache has been read (i.e., the current cache does not contain any readable target data, and the number of data objects to be read in the target cache queue is less than a preset threshold), an asynchronous data loading task is sent to the thread pool to preload the data objects to be read. Since the target cache queue still contains data objects to be read, these objects can be updated in the current cache. The client can then directly read the data from the target cache file without waiting.

[0107] Furthermore, the thread pool, based on the asynchronous data loading task, determines the number of tasks to be executed corresponding to the asynchronous data loading according to a preset strategy, starts threads, loads the number of data objects to be read from the data file to be read, and stores the loaded data objects to be read sequentially in the target cache queue.

[0108] For example, the target cache queue size is 10, and the number of threads in the thread pool is 5. During the initial data read, the number of data objects to be read in the target cache queue is 0. The preset threshold is the minimum of the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, which is Min(10-0, (0+1)*2)=2. Since 0 is less than 2, the thread pool needs to start an asynchronous data loading task. The number of tasks to be executed is the minimum of the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, which is also Min(10-0, (0+1)*2)=2. That is, any two threads in the thread pool can be used to load two data objects to be read from the data file to be read and store them sequentially in the target cache queue. At this time, there are two data objects to be read in the target cache queue.

[0109] When a data object to be read in the target cache queue is updated to the current cache and read (i.e., a data object to be read is consumed), the number of data objects to be read in the target cache queue is 1. The preset threshold becomes Min(10-1, (1+1)*2)=4. Since 1 is less than 4, the thread pool needs to start an asynchronous data loading task. The number of tasks to be executed also becomes Min(10-1, (1+1)*2)=4. That is, any 4 threads in the thread pool can be used to load 4 data objects to be read from the data file to be read and store them sequentially in the target cache queue. At this time, there are 5 data objects to be read in the target cache queue.

[0110] During further data reading, when a data object to be read in the target cache queue is updated to the current cache area and read, the number of data objects to be read in the target cache queue is 4. The preset threshold becomes Min(10-4, (4+1)*2)=6. Since 4 is less than 6, the thread pool needs to start an asynchronous data loading task. The number of tasks to be executed also becomes Min(10-4, (4+1)*2)=6. That is, 5 threads in the thread pool can be used to load 5 data objects to be read in the data file to be read. Then, 1 thread in the thread pool is used to load 1 data object to be read in the data file to be read and store them sequentially in the target cache queue. At this time, there are 10 data objects to be read in the target cache queue.

[0111] For example, if the size of the data file to be read is 1G and the size of each data object to be read is 1024 bytes, then multiple 1024-byte data objects to be read can be asynchronously loaded using threads in the thread pool and stored in the target cache queue according to their position in the data file to be read. Then the target cache queue contains multiple data objects to be read, and when the current cache does not contain readable target data, data can be read from the target cache queue, reducing data waiting time.

[0112] Asynchronous data loading allows multiple threads to load (i.e., fetch) data from a file based on the start and end positions of each data object to be read. The loaded data objects are then written to a target cache queue, and the information for each data object in the target cache queue is updated. The start and end positions of each data object can be set according to actual needs. For example, the size of the data object can be set to 1024 bytes or 2048 bytes. If it's 1024 bytes, the start and end positions of a data object in the file can be 0 and 1023, or 1024 and 2047; if it's 2048 bytes, the start and end positions can be 0 and 2047, or 2048 and 4095, etc.

[0113] In this embodiment of the invention, thread pool technology is used to asynchronously load multiple data objects to be read into the target cache queue, thereby achieving asynchronous non-blocking data reading. Then, even if the current cache does not contain readable target data, data can be read directly from the target cache queue. Compared with the prior art, it is only necessary to wait for the time consumed by one I / O transmission during the first data reading to complete the data reading of the data file to be read, which reduces the client's data reading waiting time, reduces the overall data file reading time, and improves the efficiency of data reading.

[0114] As an optional implementation of this invention, if the number of data objects in step S106 is not zero, updating the data in the current cache based on the status information of the first data object to be read in the target cache queue can be achieved in the following way:

[0115] If the number is not zero, determine the status information of the first data object to be read in the target cache queue;

[0116] If the status information is not a loading failure message, update the first data object to be read to the current cache area and delete the first data object to be read from the target cache queue.

[0117] If the current cache does not contain readable target data, but the number of data objects to be read in the target cache queue is not zero, it indicates that there is no readable target data in the current cache, but there are still data objects to be read in the target cache queue. In this case, the status information of the first data object to be read in the target cache queue can be determined first.

[0118] The status information of the first data object to be read can include: no data loaded, successful loading, and failed loading. "No data loaded" indicates that the data in the first data object has not yet been loaded, meaning the first data object is empty, and the thread needs to be waited for to load the data. "Successful loading" indicates that the data in the first data object has been loaded. "Failed loading" indicates that the data in the first data object failed to load. In practical applications, the thread can be configured to retry in case of loading failure. If loading fails after a set number of retries, it indicates that the data in the first data object has indeed failed to load.

[0119] As an optional implementation of this invention, when the status information is not a loading failure message, updating the first data object to be read to the current cache area and deleting the first data object to be read from the target cache queue may include:

[0120] If the status information indicates successful loading, update the first data object to be read to the current cache area and delete the first data object to be read from the target cache queue;

[0121] If the status information is "no data loaded", wait for the data in the first data object to be read to be loaded, then update the first data object to be read to the current cache area, and delete the first data object to be read from the target cache queue.

[0122] As an optional implementation of this invention, if the status information of the first data object to be read is a loading failure message, the data reading failure message is returned and the current cache is cleared.

[0123] If the status information of the first data object to be read is a loading failure message, it indicates that the data of the first data object to be read has indeed failed to be loaded. At this time, the data reading failure message can be returned to the client and the current cache area can be cleared.

[0124] In this embodiment of the invention, when the current cache does not contain readable target data, but the number of data objects to be read in the target cache queue is not zero, the data objects to be read in the target cache queue can be updated to the current cache. This allows the client to directly read the target data from the data file to be read in the current cache without waiting. Furthermore, when the number of data objects to be read in the target cache queue is less than a preset threshold, asynchronous loading of the data objects to be read allows the reading of the target data in the data file to be read and the loading of the data objects to be read in the data file to be read to be synchronized, thereby improving data reading efficiency.

[0125] As an optional implementation of this invention, when the data file to be read has been read, the thread pool can be closed and the current cache area can be cleared.

[0126] Knowing the size of the data file to be read, and recognizing that each data object has a start and end position, and that the readable target data in the current buffer is obtained by updating the current buffer with the data object to be read, allows us to determine the current position of the readable target data when reading it from the current buffer. This enables us to ascertain whether the data file has been completely read. Once the data file has been read completely, the thread pool can be shut down and the current buffer cleared.

[0127] For example, such as Figure 3 As shown, the data reading method of this embodiment can be implemented through a streaming reading acceleration component. When a client needs to read a remote file, it can read readable target data from the data file to be read in the current cache through the client data reading interface. If the current cache contains readable target data, it reads the readable target data from the current cache and returns the read target data. If the current cache does not contain readable target data, it determines the data object to be read contained in the target cache queue. Figure 3 If the number of data objects to be read in the target cache queue is less than a preset threshold, an asynchronous data loading task is sent to the thread pool. Based on the asynchronous data loading task, the thread pool determines the number of tasks to be executed corresponding to the asynchronous data loading according to a preset strategy, starts a thread, loads the number of data objects to be read in the data file to be read, and stores the loaded data objects to be read sequentially in the target cache queue.

[0128] If the number of data objects to be read in the target cache queue is not zero, determine the status information of the first data object to be read in the target cache queue. If the status information of the first data object to be read is not a loading failure message, update the first data object to be read to the current cache area and delete the first data object to be read. If the status information of the first data object to be read is a loading failure message, return the data reading failure message and clear the current cache area. If the data file to be read is read completely, shut down the thread pool and clear the current cache area.

[0129] Corresponding to the above method embodiments, the present invention also provides corresponding device embodiments.

[0130] like Figure 4 As shown, an embodiment of the present invention provides a data reading device, which may include:

[0131] The receiving module 201 is used to receive a data read request, which contains the identifier of the data file to be read.

[0132] The first judgment module 202 is used to determine whether the data file corresponding to the identifier of the data file to be read has been read completely.

[0133] The second judgment module 203 is used to determine whether the current cache corresponding to the data file to be read contains readable target data when the first judgment module determines that the data file to be read has not been read completely. The readable target data is a part of the data file to be read.

[0134] The determination module 204 is used to determine the number of data objects to be read in the target cache queue when the second determination module determines that the current cache does not contain readable target data. The data objects to be read are part of the data in the data file to be read, and the target cache queue is the cache queue corresponding to the current cache.

[0135] The data loading module 205 is used to asynchronously load the data objects to be read from the data file to be read into the target cache queue when the number is less than a preset threshold, and to trigger the second judgment module to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0136] The data update module 206 is used to update the data in the current cache area according to the status information of the first data object to be read in the target cache queue when the number is not zero, and to trigger the second judgment module to determine whether the current cache area corresponding to the data file to be read contains readable target data.

[0137] This invention provides a data reading device that sets a target cache queue corresponding to the current cache area, asynchronously loads multiple data objects to be read into the target cache queue, and realizes asynchronous non-blocking data reading. Thus, even if the current cache area does not contain readable target data, data can be read directly from the target cache queue. Compared with the prior art, it only needs to wait for the time consumed by one I / O transmission during the first data reading to complete the data reading of the data file to be read, reducing the client's data reading waiting time, reducing the overall data file reading time, and improving the efficiency of data reading.

[0138] Optionally, the data loading module 205 described above is specifically used for:

[0139] If the number is less than a preset threshold, the number of tasks to be executed corresponding to asynchronous data loading is determined according to a preset strategy. The preset strategy is determined based on the size of the target cache queue and the number of data objects to be read contained in the target cache queue.

[0140] Load several data objects to be read from the data file to be executed, and store the loaded data objects sequentially in the target cache queue.

[0141] Optionally, the above-mentioned preset strategy is:

[0142] The minimum of the difference between the size of the target cache queue and the number of data objects to be read in the target cache queue, and twice the number of data objects to be read in the target cache queue plus 1, is determined as the number of tasks to be executed.

[0143] Optionally, the aforementioned data update module 206 includes:

[0144] The determination submodule is used to determine the status information of the first data object to be read in the target cache queue when the number is not zero.

[0145] The first update submodule is used to update the first data object to be read into the current cache area and delete the first data object to be read from the target cache queue when the status information is not a loading failure message.

[0146] Optionally, the data update module 206 mentioned above further includes:

[0147] The second update submodule is used to return a data reading failure message and clear the current cache when the status information is a loading failure message.

[0148] Optionally, the first update submodule mentioned above is specifically used for:

[0149] If the status information indicates successful loading, update the first data object to be read to the current cache area and delete the first data object to be read from the target cache queue.

[0150] If the status information is "no data loaded", wait for the data in the first data object to be read to be loaded, then update the first data object to be read to the current cache area, and delete the first data object to be read from the target cache queue.

[0151] Optionally, the above-mentioned device further includes:

[0152] The data reading module is used to read readable target data from the current cache when the second judgment module determines that the current cache contains readable target data, and return the read target data, as well as to trigger the second judgment module to determine whether the current cache corresponding to the data file to be read contains readable target data.

[0153] Optionally, the above-mentioned device further includes:

[0154] The clearing module is used to clear the current cache area when the first judgment module determines that the data file to be read has been read.

[0155] This invention also provides a proxy server device, such as... Figure 5 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0156] Memory 303 is used to store computer programs;

[0157] When the processor 301 executes the program stored in the memory 303, it implements the steps of the above data reading method to achieve the same technical effect.

[0158] The communication bus mentioned in the above electronic devices can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0159] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0160] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0161] The processors mentioned above can be general-purpose processors, including CPUs (Central Processing Units), NPs (Network Processors), etc.; they can also be DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0162] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-described data reading methods to achieve the same technical effect.

[0163] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the data reading methods in the above embodiments to achieve the same technical effect.

[0164] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, DSL (Digital Subscriber Line)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs (Digital Versatile Discs)), or semiconductor media (e.g., SSDs (Solid State Disks)).

[0165] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device / electronic device embodiments are basically similar to the system embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0167] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data reading method characterized by, The method comprises: receiving a data reading request, the data reading request containing an identifier of a data file to be read; determining whether the data file to be read corresponding to the identifier has been read completely; if the data file to be read has not been read completely, determining whether the current cache area corresponding to the data file to be read contains readable target data, the readable target data being part of the data file to be read; if the current cache area does not contain the readable target data, determining the number of data objects to be read contained in a target cache queue, the data objects to be read being part of the data file to be read, and the target cache queue being a cache queue corresponding to the current cache area; if the number is less than a preset threshold, loading the data objects to be read in the data file to be read into the target cache queue asynchronously, and returning to the step of determining whether the current cache area corresponding to the data file to be read contains the readable target data; if the number is not zero, updating the data in the current cache area according to the state information of the first data object to be read in the target cache queue, and returning to the step of determining whether the current cache area corresponding to the data file to be read contains the readable target data.

2. The method of claim 1, wherein, The step of loading the data objects to be read in the data file to be read into the target cache queue asynchronously if the number is less than the preset threshold comprises: if the number is less than the preset threshold, determining the number of tasks to be executed corresponding to asynchronous data loading according to a preset strategy, the preset strategy being determined according to the size of the target cache queue and the number of data objects to be read contained in the target cache queue; loading the number of data objects to be read in the data file to be read, and storing the loaded data objects to be read in the target cache queue in sequence.

3. The method of claim 2, wherein, The preset strategy is: determining the minimum of the difference between the size of the target cache queue and the number of data objects to be read contained in the target cache queue and twice the number of data objects to be read contained in the target cache queue plus 1 as the number of tasks to be executed.

4. The method of claim 1, wherein, The step of updating the data in the current cache area according to the state information of the first data object to be read in the target cache queue if the number is not zero comprises: if the number is not zero, determining the state information of the first data object to be read in the target cache queue; if the state information is not loading failure information, updating the first data object to be read in the current cache area, and deleting the first data object to be read in the target cache queue.

5. The method of claim 4, wherein, The method further comprises: if the state information is loading failure information, returning information of data reading failure, and clearing the current cache area.

6. The method of claim 4, wherein, The step of updating the first to-be-read data object to the current cache area and deleting the first to-be-read data object in the target cache queue in the case that the state information is not the loading failure information comprises: The step of updating the first to-be-read data object to the current cache area and deleting the first to-be-read data object in the target cache queue in the case that the state information is the loading success information comprises: The step of updating the first to-be-read data object to the current cache area and deleting the first to-be-read data object in the target cache queue in the case that the state information is the unloading data information comprises:

7. The method of claim 1, wherein, After the step of judging whether the current cache area corresponding to the to-be-read data file contains the readable target data, the method further comprises: If the current cache area contains the readable target data, reading the readable target data from the current cache area and returning the read target data, and returning to the step of judging whether the current cache area corresponding to the to-be-read data file contains the readable target data.

8. The method of claim 1, wherein, After the step of judging whether the to-be-read data file corresponding to the identification of the to-be-read data file has been read completely, the method further comprises: In the case that the to-be-read data file has been read completely, emptying the current cache area.

9. A data reading device, characterized by The device comprises: A receiving module is configured to receive a data reading request, wherein the data reading request contains the identification of a to-be-read data file; A first judging module is configured to judge whether the to-be-read data file corresponding to the identification of the to-be-read data file has been read completely; A second judging module is configured to, in the case that the first judging module judges that the to-be-read data file has not been read completely, judge whether the current cache area corresponding to the to-be-read data file contains the readable target data, wherein the readable target data is part of the data of the to-be-read data file; A determining module is configured to, in the case that the second judging module judges that the current cache area does not contain the readable target data, determine the number of to-be-read data objects contained in the target cache queue, wherein the to-be-read data object is part of the data of the to-be-read data file, and the target cache queue is the cache queue corresponding to the current cache area; A data loading module is configured to, in the case that the number is less than a preset threshold, load the to-be-read data object in the to-be-read data file into the target cache queue asynchronously, and trigger the second judging module to judge whether the current cache area corresponding to the to-be-read data file contains the readable target data; A data updating module is configured to, in the case that the number is not zero, update the data in the current cache area according to the state information of the first to-be-read data object in the target cache queue, and trigger the second judging module to judge whether the current cache area corresponding to the to-be-read data file contains the readable target data.

10. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. a memory for storing a computer program; a processor for implementing the method steps of any one of claims 1-8 when executing the program stored in the memory.

11. A computer readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method steps of any one of claims 1-8.

Images