A data reading method and device, electronic equipment and storage medium

Abstract

This invention discloses a data reading method, apparatus, electronic device, and storage medium. The method includes: determining the file to which the data to be read belongs and the storage segment of the data to be read based on a received current data reading request; if the file data in the file to which the data belongs has been cached in file memory, reading first file sub-data corresponding to the storage segment from the file memory, identifying the first file sub-data as the data to be read, and sending the data to be read to a data request node; wherein, the file memory includes: file data in the file to which the data belongs, cached based on historical data reading requests, and the file data includes at least one file sub-data corresponding to a storage segment. The technical solution of this invention reduces the number of accesses to the local disk and lowers the local disk load by directly reading from the file memory, thereby improving the efficiency and stability of data computation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data reading method, apparatus, electronic device and storage medium. Background Technology

[0002] In big data distributed computing scenarios, the shuffle operation on data is indispensable. The essence of the shuffle operation is to divide the data obtained by the Map node using a partitioner and distribute the data to the corresponding Reducer node.

[0003] In existing distributed computing scenarios, when a Reducer node obtains Shuffle data based on the Shuffle service, it needs to access data from each storage segment corresponding to each Map node on its local disk to complete the Shuffle operation. However, in the process of implementing this invention, it was found that the existing technology has at least the following technical problems: in big data computing scenarios, the amount of access to the local disk is large; and when obtaining Shuffle data from each storage segment, random reads are prone to occur, resulting in high disk load and affecting the working efficiency and stability of the distributed computing engine. Summary of the Invention

[0004] This invention provides a data reading method, apparatus, electronic device, and storage medium to reduce the number of accesses to the local disk, reduce the local disk load, and improve the efficiency and stability of data computing.

[0005] According to one aspect of the present invention, a data reading method is provided, comprising:

[0006] Based on the received current data read request, determine the file to which the data to be read belongs and the storage area of ​​the data to be read;

[0007] If the file data in the file has been cached in the file memory, read the first file sub-data corresponding to the storage area from the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node;

[0008] The file memory includes: file data in the file to which it belongs, cached based on historical data read requests, and the file data includes file sub-data corresponding to at least one region.

[0009] According to another aspect of the present invention, a data reading device is provided, the device comprising:

[0010] The file ownership determination module is used to determine the file to which the data to be read belongs and the storage area of ​​the data to be read based on the received current data read request;

[0011] The data to be read sending module is used to read the first file sub-data corresponding to the storage area from the file memory when the file data in the file has been cached in the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node;

[0012] The file memory includes: file data in the file to which it belongs, cached based on historical data read requests, and the file data includes file sub-data corresponding to at least one storage segment.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data reading method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data reading method described in any embodiment of the present invention.

[0018] The technical solution of this invention, based on the received current data read request, determines the file to which the data to be read belongs and the storage area of ​​the data to be read. If the file data in the file to which the data belongs has been cached in the file memory, the first file sub-data corresponding to the storage area is read from the file memory and sent as the data to be read to the data request node. Since the file data can be cached in the file memory through historical data read requests, when executing the current data read request after the historical data read request, the operation of directly reading the file memory is performed, which reduces the number of accesses to the local disk, reduces the local disk load, and achieves the effect of improving the efficiency and stability of data computing.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a data reading method provided in the prior art;

[0022] Figure 2 This is a flowchart of a data reading method provided according to an embodiment of the present invention;

[0023] Figure 3 This is a flowchart of another data reading method provided according to an embodiment of the present invention;

[0024] Figure 4 This is an architecture diagram of a data reading process provided according to an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram illustrating a process for acquiring data to be read according to an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of a data reading method apparatus provided according to an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the data reading method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "etc.", and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Figure 1 This is a schematic diagram of a data reading method provided in the prior art; such as Figure 1 As shown, in a distributed computing scenario, there can be M Map nodes and P Reducer nodes. During the Shuffle operation, the Map nodes generate Shuffle data and write it to the local disk according to different storage areas. The number of storage areas corresponding to each Map node is N; where M, P, and N are positive integers. When a Reducer node retrieves Shuffle data stored on its local disk, it sends a request to the Shuffle service deployed on the Map nodes. The Shuffle service accesses the local disk to obtain the Shuffle data. However, the above data reading method has the following problems: retrieving data stored on the local disk requires a maximum of M×N accesses to the local disk, resulting in a large number of local disk accesses in big data computing scenarios; and random reads are prone to occur when retrieving Shuffle data from each storage area, leading to high disk load and affecting the working efficiency and stability of the distributed computing engine.

[0031] To address the current issues in Shuffle data reading, this technical solution provides a data reading method. The distributed computing engine in this solution may include MapReduce, Spark, etc. The Spark engine itself has an ESS (External Shuffle Service) based on Spark on Yarn mode, which is deployed on all distributed computing nodes.

[0032] It should be noted that in a distributed computing scenario, there may be at least one Map node and at least one Reducer node. Map nodes execute Shuffle operations, generating Shuffle data corresponding to the operation. This Shuffle data is then written to local disks according to different storage shards. A data file and an index file are generated on the Map node. The data file stores the Shuffle data, and the index file provides storage information for the Shuffle data. After the Shuffle data is written to the local disk, its metadata can be sent to the service driver. This metadata may include at least one of the following: the identifier of the application performing the Shuffle operation, the Shuffle operation identifier, and the identifier of the storage shard corresponding to the Shuffle data. The service driver can be the Spark Driver. Reducer nodes can obtain the desired Shuffle data metadata by sending a request to the server driver. Based on this metadata, they generate a data read request and send it to the Map node to retrieve the Shuffle data using the Shuffle service. For example, the Shuffle service could be a data retrieval service provided by the ESS service.

[0033] Based on the solution in this embodiment, the Shuffle service can be started based on the acquired data read request, and the Shuffle data generated by the Shuffle operation can be directly obtained from the file memory. Therefore, while obtaining the Shuffle data, there is no need to access the local disk, reducing the load on the local disk caused by the data acquisition operation.

[0034] Figure 2 This is a flowchart of a data reading method according to an embodiment of the present invention. This embodiment is applicable to reading Shuffle data, and the method can be executed by a data reading device, which can be implemented in hardware and / or software.

[0035] like Figure 2 As shown, the method in this embodiment may specifically include:

[0036] S110. Based on the received current data read request, determine the file to which the data to be read belongs and the storage area of ​​the data to be read.

[0037] The data to be read can be the shuffled data generated by the shuffle operation; the current data read request can be a request sent by the Reducer node to the Map node to obtain the data to be read. The current data read request may include information such as the data identifier of the data to be read and the storage area identifier corresponding to the data. The file to which it belongs is the file containing the data to be read.

[0038] In this embodiment, the current data read request may also include the file identifier of the file to which the data to be read belongs, so as to determine the file to which the data to be read belongs based on the file identifier; or, the data identifier of the data to be read may be determined based on the current data read request, and the file identifier of the file to which the data belongs may be determined by parsing the data identifier according to the preset parsing rules, thereby determining the file to which the data to be read belongs.

[0039] S120. If the file data in the file has been cached in the file memory, read the first file sub-data corresponding to the storage area from the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node.

[0040] The file memory includes: file data cached based on historical data read requests, and the file data includes file sub-data corresponding to at least one storage segment. The first file sub-data is the file sub-data stored in the storage segment of the data to be read.

[0041] In practice, it can be pre-determined whether the file data in the file has been cached, i.e., whether the file data has been cached in the file memory. If it has been cached in memory, the first file sub-data corresponding to the storage area can be directly read from the file memory as the data to be read. If it has not been cached in memory, the data corresponding to the storage area can be obtained by accessing the local disk as the data to be read; alternatively, the file data can be cached in the file memory first, and then the data corresponding to the storage area can be obtained from the file memory as the data to be read.

[0042] In this embodiment, after acquiring the data to be read, the data can be sent to the data request node. Optionally, in a distributed computing scenario, the data request node includes a distributed computing node, the current data read request includes a computing task execution request sent by the distributed computing node, and the data to be read includes the shuffled data required to execute the computing task. In this embodiment, the specific implementation of determining the file to which the data to be read belongs based on the received current data read request may include: when a computing task execution request is received from at least one distributed computing node, determining the file to which the shuffled data to be read belongs based on the computing task execution request.

[0043] For example, the data request node can be a Reducer node used for distributed computing. In a specific implementation, computing task execution requests sent by different distributed computing nodes can be received simultaneously. For each received computing task execution request, the corresponding shuffling data identifier of the shuffling data to be read can be determined, thereby determining the file to which the shuffling data to be read belongs based on the shuffling data identifier.

[0044] In this embodiment, when the file data in the file has been cached in the file memory, the shuffle data to be read can be read directly from the file memory, thereby being used for distributed computing, reducing the number of reads from the local disk, and helping to improve the computing efficiency during distributed computing.

[0045] In specific implementation, in order to improve the space utilization of file memory and reduce the waste of space resources, after reading the first file sub-data corresponding to the storage area from the file memory, the following steps are also included: determining the current storage duration of the first file sub-data; if the current storage duration is greater than or equal to the preset storage duration, then clearing the first file sub-data in the file memory corresponding to the file; and updating the current remaining storage of the file memory based on the file memory occupied by the file data, the total storage of the file memory, and the memory occupied by the first file sub-data.

[0046] Specifically, after the first file sub-data is read, the current storage duration of the first file sub-data in the file memory can be detected randomly or periodically, and the current storage duration can be compared with the preset storage duration. The preset storage duration can be determined based on the frequency and / or the time interval between readings of data in the file memory.

[0047] For example, the higher the frequency, the longer the preset storage duration can be set, and the lower the frequency, the shorter the preset storage duration can be set; or, the longer the time interval, the shorter the preset storage duration can be set, and the shorter the time interval, the longer the preset storage duration can be set. This makes it easier to clear data that is not frequently read in a timely manner to reduce the space occupied by the file memory.

[0048] If the current storage duration is greater than or equal to the preset storage duration, it indicates that the first file sub-data has been stored in file memory for a relatively long time. To save file memory space resources, the first file sub-data that has been stored in file memory for a relatively long time can be cleared to reduce space waste. If the current storage duration is less than the preset storage duration, it indicates that the first file sub-data has been stored in file memory for a relatively short time. To avoid the need for local disk access operations when the first file sub-data is read again later, the first file sub-data can be stored in file memory.

[0049] For example, if the first file sub-data will only be read once, the preset storage time can be set to 0 minutes. That is, after the first file sub-data is read, the first file sub-data can be cleared directly to reduce the waste of file memory resources without affecting data reading.

[0050] Furthermore, after clearing the first file's sub-data, the current remaining storage capacity of the file can be updated to understand the remaining file memory resources when storing new file data. Specifically, the difference between the memory occupied by the file and the memory occupied by the first file's sub-data can be determined, and the current remaining storage capacity of the file can be determined by subtracting this difference from the total storage capacity.

[0051] In this embodiment, by clearing the first file sub-data whose current storage duration is greater than or equal to the preset storage duration, the memory resources of the file memory are saved, which helps to improve the space utilization of the file memory. Furthermore, after clearing the data, the current remaining storage amount of the file memory is updated in a timely manner, which makes it easier to accurately understand the remaining resource status of the file memory.

[0052] In this embodiment, the method further includes: identifying a third file sub-data that has been read less than a preset threshold number of times within a preset time period in the file memory, and performing a data cleaning operation on the third file sub-data.

[0053] It should be noted that those skilled in the art can determine the preset duration based on the average time interval between the reading of each file sub-data within a historical time period; and determine the preset number threshold based on the average number of times the file sub-data is read within that time interval.

[0054] To conserve storage space in file memory, infrequently accessed file sub-data can be promptly cleared. In practice, the number of reads of each file sub-data stored in file memory within a preset time period can be detected; for example, the preset time period could be two hours. If a third file sub-data is detected with fewer reads than a preset threshold within the preset time period, it indicates that the third file sub-data is read infrequently. To reduce storage space usage, a data cleanup operation can be performed on this third file sub-data, releasing the space resources occupied by it in file memory. If the number of reads of each file sub-data within the preset time period is greater than or equal to the preset threshold, it indicates that each file sub-data is frequently accessed and can be retained in file memory for easy retrieval, reducing read pressure on the local disk.

[0055] This embodiment detects the number of times each file sub-data in the file memory is read within a preset time period. Based on the frequency of reading the file sub-data, it promptly performs data cleanup operations on the file sub-data stored in the file memory to reduce the waste of file memory space resources.

[0056] The technical solution of this invention, based on the received current data read request, determines the file to which the data to be read belongs and the storage area of ​​the data to be read. If the file data in the file to which the data belongs has been cached in the file memory, the first file sub-data corresponding to the storage area is read from the file memory and sent as the data to be read to the data request node. Since the file data can be cached in the file memory through historical data read requests, when executing the current data read request after the historical data read request, the operation of directly reading the file memory is performed, which reduces the number of accesses to the local disk, reduces the local disk load, and achieves the effect of improving the efficiency and stability of data computing.

[0057] Figure 3 This is a flowchart of another data reading method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further includes: when file data in the belonging file is not cached in file memory, acquiring file data from the belonging file, caching the file data in the file memory corresponding to the belonging file; reading first file sub-data corresponding to a storage area in the file memory, determining the first file sub-data as the data to be read, and sending the data to be read to the data request node. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0058] S210. Based on the received current data read request, determine the file to which the data to be read belongs and the storage area of ​​the data to be read.

[0059] S220. If the file data in the file has been cached in the file memory, read the first file sub-data corresponding to the storage area from the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node.

[0060] The file memory includes: file data in the file to which the historical data read request is cached, and the file data includes file sub-data corresponding to at least one storage segment.

[0061] S230. If the file data in the file is not cached in the file memory, obtain the file data in the file and cache the file data in the file memory corresponding to the file; read the first file sub-data corresponding to the storage area in the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node.

[0062] Specifically, when file data is not cached in file memory, in order to ensure that the data to be read corresponding to the current read request can be obtained normally, the first file sub-data corresponding to the determined storage area can be obtained directly by accessing the local disk as the data to be read, so as to complete the process of obtaining the data to be read.

[0063] Alternatively, to facilitate subsequent retrieval of the data to be read, and reduce the access pressure on the local disk during subsequent reading processes, the file data can be obtained by accessing the local disk. Then, according to the correspondence between file sub-data on the local disk and storage areas, each file sub-data within the file data can be stored separately in the file memory. Furthermore, based on the storage area of ​​the data to be read, the first file sub-data corresponding to that storage area can be determined in the file memory, and this first file sub-data can be sent to the data request node as the data to be read.

[0064] In this embodiment, if the file data in the file is not cached in the file memory, the file data can be obtained through a single access operation to the local disk and cached in the file memory. Thus, when subsequent sub-data of the file needs to be read, it can be read directly from the file memory, greatly reducing the number of accesses to the local disk.

[0065] In practice, caching file data in the file memory corresponding to the file can be achieved by: determining the current remaining storage and total storage of the file memory; if the file data meets the preset conditions, then caching the file data in the file memory.

[0066] The preset conditions include that the file's memory usage is less than or equal to a preset memory threshold, and the sum of the file's memory usage and the current remaining storage is less than or equal to the total storage.

[0067] Specifically, the sum of the file's memory usage and the remaining storage space can be determined. If this sum is greater than the total storage space, it indicates that the file's memory cannot meet the file data's memory requirements. Furthermore, a preset memory threshold can be set, which represents the maximum amount of file data allowed to be stored in the file's memory. When the file's memory usage exceeds the preset memory threshold, it can be determined that the file's memory cannot meet the file data's memory requirements.

[0068] To ensure file data can be successfully cached in file memory, the following two conditions must be met simultaneously: 1. The file's memory usage is less than or equal to a preset memory threshold; 2. The sum of the file's memory usage and the current remaining storage is less than or equal to the total storage. When both conditions are met, the file data can be cached in file memory, and the current remaining storage in file memory can be updated based on the file's memory usage.

[0069] Furthermore, if the file data does not meet either of the above two conditions, it means that the file memory cannot cache the file data. In this case, the data to be read can be obtained directly by accessing the local disk based on the current data read request.

[0070] In this embodiment, by determining whether the file data meets the preset conditions, it is possible to quickly and efficiently determine whether the file memory can successfully cache the file data, thereby improving the effectiveness and accuracy of caching file data.

[0071] Optionally, before obtaining the file data in the file, the method further includes: if no file information corresponding to the file is found in the Bloom filter corresponding to the file, then determining the number of times the file data was requested in the historical time period before receiving the data read request; if the number of requests is less than a preset threshold, then determining that the file data in the file has not been cached in the file memory.

[0072] The Bloom filter stores file information corresponding to file data that has been requested more than or equal to a preset threshold but has not been stored in the file memory. For example, the preset threshold can be 1.

[0073] In practical implementation, a Bloom filter can be pre-built. When determining whether file data is cached in file memory, the pre-built Bloom filter can be used to determine whether file information corresponding to the file can be found in the filter. The file information can be: the filename and / or file identifier of the file, converted into an array according to the storage format of the Bloom filter.

[0074] When determining whether file data has been stored in file memory, if no file information corresponding to the file is found in the Bloom filter, it means that the number of times the file data has been requested is less than a preset threshold, and no record has been generated in the Bloom filter yet; or, if the number of times the file data has been requested is equal to or greater than the preset threshold, the file data has been stored in file memory. For example, if file data is requested for the first time, no record is generated in the Bloom filter, and it is not stored in file memory. Therefore, when it is determined that no file information corresponding to the file is found in the Bloom filter, it can be further determined whether the number of times the file data has been requested in the historical time period is less than the preset threshold. If it is less than the preset threshold, it is determined that the file data in the file has not been cached in file memory.

[0075] In this embodiment, a Bloom filter can be used to quickly and accurately determine whether file data has been stored in file memory, increasing the convenience of the query process and improving the efficiency of obtaining the data to be read.

[0076] Furthermore, if no file information corresponding to the file is found in the Bloom filter corresponding to the file, in order to accurately determine the cache status of the file data, the following may also be included: if the number of requests is greater than or equal to a preset threshold, then determine whether the cache operation record stored in advance includes the cache information of the file; if the cache operation record includes the cache information, then determine that the file has been cached in the file memory.

[0077] Specifically, for file data that has undergone caching, the information of the corresponding file can be recorded in the cache operation record. For example, the filename and / or file identifier of the file can be stored in the cache operation record. If no file information corresponding to the file is found in the Bloom filter corresponding to the file, and the number of requests is greater than or equal to a preset threshold, to avoid determination errors caused by concurrent data read requests, the cache operation record can be used to determine whether the file data has been cached in file memory.

[0078] It should be noted that concurrent data read requests are two or more data read requests whose request time interval is less than a preset time interval. These concurrent data read requests can be considered to be generated almost simultaneously, and are used to retrieve the same concurrent sub-data from a file. For example, the preset threshold can be set to 1. If the interval between the request times of the first and second data read requests is less than the preset interval value, then both the first and second data read requests can be identified as concurrent data read requests, used to read the same concurrent sub-data from a file. The first and second data read requests can be considered to pass through a Bloom filter simultaneously. Since both are the first requests for the file sub-data, no corresponding file information is found in the Bloom filter. Furthermore, for read requests that do not find file information, it is necessary to determine whether the number of requests in the historical time period is greater than or equal to the preset threshold. For the first data read request, since this is the first request, the number of requests in the historical time period is 0. If the concurrent file data corresponding to the concurrent file sub-data requested by the first data read request has not been cached in file memory, then we can continue to determine whether the concurrent file data meets the preset conditions and whether to perform a caching operation on the concurrent file data. For the second data read request, which is now the second request for the concurrent file sub-data, the number of requests equals the preset threshold 1. As explained above, in the case where there are no concurrent data read requests, it can be determined that the concurrent file data corresponding to the second data read request has been stored in file memory. However, to prevent errors in determining the caching status of concurrent file data due to the concurrent file data corresponding to the first data read request not having completed the caching operation, we can determine whether the concurrent file data has been cached in file memory based on the cache operation record.

[0079] Specifically, the cache information can include the cache time, cache data volume, file identifier, etc. If the cache operation record includes the cache information of the file, it is determined that the file data corresponding to the file has been cached in the file memory; if it does not include it, it means that the file data has not been cached in the file memory.

[0080] This embodiment determines whether file data has been cached by caching operation records, which can improve the accuracy and effectiveness of determining whether file data has been cached and avoid determination failure due to concurrent data read requests.

[0081] In this embodiment, the method further includes: if the cache operation record does not include cache information; or if file information is found in the Bloom filter corresponding to the file, then the second file sub-data corresponding to the storage area stored in the local disk is read; the second file sub-data is determined as the data to be read, and the data to be read is sent to the data request node.

[0082] Specifically, if the cache operation record does not include cache information, it means that the file data has not been cached. If the Bloom filter corresponding to the file can find the file information corresponding to the file data, it also means that the file data has not been cached. In order to successfully obtain the data to be read, the local disk can be accessed to obtain the second file sub-data corresponding to the storage area as the data to be read, and then the data to be read can be sent to the requesting node.

[0083] In this embodiment, for file data that has not been stored on the local disk, the local disk can be accessed to ensure that the data to be read corresponding to the current read data request can be successfully obtained.

[0084] In specific implementation, if the file data does not meet the preset conditions, the operation of caching the file data to the file memory will not be performed. In order to facilitate subsequent querying of the cached status of the file data, this embodiment also includes: if the file data does not meet the preset conditions, generating file information corresponding to the file based on the file data, and storing the file information in the Bloom filter corresponding to the file.

[0085] Specifically, for file data that does not meet preset conditions, the corresponding file can be directly determined based on the file data, and file information corresponding to the corresponding file in the Bloom filter can be generated. This file information is then stored in the Bloom filter to update the file information of file data that has not been cached. This improves the accuracy of the Bloom filter in querying file data that has not been cached in file memory and enhances query convenience.

[0086] The embodiments of the data reading method have been described in detail above. In order to enable those skilled in the art to further understand the technical solution of this method, specific application scenarios are given below.

[0087] Figure 4 This is an architecture diagram of a data reading process provided according to an embodiment of the present invention; Figure 4 Taking a scenario with one Map node and P Reducer nodes as an example, where each Map node corresponds to N storage partitions (M, P, and N are positive integers), the shuffle data reading process in a distributed computing scenario is explained. Each Reducer node can send a shuffle request to the Map node. The Map node, based on the Shuffle service deployed on it, can send the acquired shuffle data to the corresponding Reducer node. For example... Figure 4As shown, a DataFile and an Index File are generated on the Map node. The DataFile stores the Shuffle data, and the Index File provides storage information for the Shuffle data. When a shuffle request is received, the file data corresponding to the requested shuffle data is cached in a pre-built file memory through a single local disk access, organized by storage slice. Subsequent shuffle requests can directly read the shuffle data of the corresponding storage slice from the file memory. For example, the second shuffle request retrieves the shuffle data from storage slice 1, the second shuffle request retrieves the shuffle data from storage slice 2, and so on, until the Nth shuffle request retrieves the shuffle data from storage slice N. After reading the shuffle data, it is cleared from the file memory until the file memory is empty.

[0088] In this embodiment, after the Map node receives a shuffle request from the Reducer node for reading shuffle data, the process of obtaining the data to be read can be found in [reference needed]. Figure 5 It includes the following steps:

[0089] S310, Receive the current shuffle request

[0090] The current shuffle request may include information such as: the application identifier (AppId) of the requested data, the shuffle file identifier (ShuffleId), and the starting and ending segment numbers of the storage segment containing the shuffle data to be acquired. Based on the received current shuffle request, the shuffle file name of the shuffle file to which the shuffle data belongs is determined.

[0091] S320. Determine whether shuffle file information exists in the Bloom filter corresponding to the currently requested shuffle file.

[0092] The Bloom filter can be defined according to the format AppId.ShuffleId.BloomFilter, which corresponds to the shuffle file.

[0093] For example, the Bloom filter is built based on the shuffle file name when the first data read request for the shuffle file does not meet the preset conditions for caching to file memory. It is used to store the file information of the shuffle file that does not meet the preset conditions. This ensures that if the shuffle file does not meet the preset conditions for caching to file memory during the first data read request, the Bloom filter can quickly identify the shuffle file for the next data read request.

[0094] Specifically, if the Bloom filter includes the file information of the shuffle file, then it no longer checks whether the preset conditions are met, and S390 can be executed to directly obtain the file sub-data according to the original processing logic in the prior art; wherein, the original processing logic is to obtain the shuffle data corresponding to the first data read request or the next data read request by accessing the local disk.

[0095] S330. If the Bloom filter does not include the file information of the shuffle file, determine whether the current shuffle request is the first request for the shuffle data. If it is the first request, execute S340; if it is not the first request, execute S350.

[0096] S340. Determine whether the shuffle file meets the preset conditions for caching to file memory. The preset conditions may be: if the size of the shuffle file meets the following two conditions, it means that the shuffle file meets the preset conditions, and then execute S360; if it does not meet either of the following conditions, it means that the shuffle file does not meet the preset conditions, and then execute S370.

[0097] Condition 1: The memory usage of the shuffle file data is less than or equal to a preset memory threshold; the preset memory threshold can be 64MB.

[0098] Condition 2: The sum of the memory occupied by the file and the current remaining storage in the file's memory is less than or equal to the total storage of the file's memory; where the total storage can be 400MB.

[0099] S350: Determine whether the shuffle file's data has been cached in file memory by checking the cache operation record. If it has been cached, proceed to S380; otherwise, proceed to S390. For example, the cached data can be determined using the ConcurrentHashMap<filename, shuffleCache> data storage structure within the ESS service.

[0100] S360. Cache the file data in the file memory and update the used memory amount of the file memory; execute S380. For example, the data storage structure after storing the shuffle file data can be ConcurrentHashMap<filename, shuffleCache>.

[0101] S370. Store the file information of the shuffle file into a Bloom filter and obtain the shuffle data according to the original processing logic.

[0102] S380. Obtain the file sub-data of the storage segment corresponding to the current shuffle request, use it as shuffle data, and clean up the shuffle data from the file memory. Update the used memory amount of the file memory based on the cleanup result, and feed back the obtained shuffle data to the Reducer node that sent the request.

[0103] For example, the corresponding shuffle data can be obtained from the ConcurrentHashMap<filename,shuffleCache> memory data structure of the ESS service based on the start and end segment numbers of the storage segment of the shuffle request.

[0104] S390. Obtain shuffle data according to the original quantity logic.

[0105] In this embodiment, by caching file data in memory, the total number of accesses to the local disk can be effectively reduced to N, thereby reducing the amount of access to the local disk; avoiding random reads, reducing disk load, improving task runtime efficiency, ensuring the working efficiency and stability of the distributed computing engine, and saving service resources.

[0106] Figure 6 This is a schematic diagram of a data reading method apparatus according to an embodiment of the present invention. This apparatus is used to execute the data reading method provided in any of the above embodiments. This apparatus and the data reading methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the data reading apparatus can be found in the embodiments of the above data reading methods. Figure 6 As shown, the device includes:

[0107] The file ownership determination module 10 is used to determine the file to which the data to be read belongs and the storage area of ​​the data to be read based on the received current data read request;

[0108] The data to be read sending module 11 is used to read the first file sub-data corresponding to the storage area from the file memory when the file data in the file has been cached in the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node;

[0109] The file memory includes: file data in the file to which the historical data read request is cached, and the file data includes file sub-data corresponding to at least one storage segment.

[0110] In addition to any of the optional technical solutions in the embodiments of the present invention, the invention may also include:

[0111] The file data caching module is used to retrieve the file data from the file in the file and cache it in the file memory corresponding to the file in ...

[0112] The first file sub-data reading module is used to read the first file sub-data corresponding to the storage area in the file memory, identify the first file sub-data as the data to be read, and send the data to be read to the data request node.

[0113] Based on any optional technical solution in the embodiments of the present invention, the optional file data caching module includes:

[0114] The total storage determination unit is used to determine the current remaining storage and total storage of the file memory;

[0115] The file data caching unit is used to cache the file data in the file memory if the file data meets the preset conditions.

[0116] The preset conditions include that the file's memory usage is less than or equal to a preset memory threshold, and the sum of the file's memory usage and the current remaining storage is less than or equal to the total storage.

[0117] Based on any optional technical solution in the embodiments of the present invention, the file data caching module may optionally further include:

[0118] The request count determination unit is used to determine the number of times the file data was requested in the historical time period before receiving the data read request if no file information corresponding to the file is found in the Bloom filter corresponding to the file before the file data in the file is retrieved; if the number of requests is less than a preset threshold, it is determined that the file data in the file has not been cached in the file memory.

[0119] The Bloom filter is used to store file information corresponding to file data that has been requested more than or equal to a preset threshold and has not been stored in the file memory.

[0120] Based on any optional technical solution in the embodiments of the present invention, the file data caching module may optionally further include:

[0121] The file information generation unit is used to generate file information corresponding to the file based on the file data if the file data does not meet the preset conditions, and to store the file information in the Bloom filter corresponding to the file.

[0122] Based on any optional technical solution in the embodiments of the present invention, the file data caching module may optionally further include:

[0123] The cache information determination unit is used to determine whether the cache operation record stored in advance includes the cache information of the file to which the file belongs if the number of requests is greater than or equal to a preset threshold; if the cache operation record includes cache information, it is determined that the file to which the file belongs has been cached in the file memory.

[0124] Based on any optional technical solution in the embodiments of the present invention, the file data caching module may optionally further include:

[0125] The second file sub-data reading unit is used to read the second file sub-data corresponding to the storage area stored on the local disk if the cache operation record does not include cache information, or if file information is found in the Bloom filter corresponding to the file.

[0126] The data to be read sending unit is used to determine the sub-data of the second file as the data to be read and send the data to be read to the data request node.

[0127] In addition to any of the optional technical solutions in the embodiments of the present invention, the invention may also include:

[0128] The current storage duration determination module is used to determine the current storage duration of the first file sub-data after reading the first file sub-data corresponding to the storage area from the file memory. If the current storage duration is greater than or equal to the preset storage duration, the first file sub-data is cleared from the file memory corresponding to the file.

[0129] The current remaining storage update module is used to update the current remaining storage of the file memory based on the file's memory usage, the total storage of the file memory, and the memory usage of the first file's sub-data.

[0130] In addition to any of the optional technical solutions in the embodiments of the present invention, the invention may also include:

[0131] The third file sub-data cleanup module is used to identify third file sub-data in file memory that has been read less than a preset threshold number of times within a preset time period, and to perform data cleanup operations on the third file sub-data.

[0132] Based on any optional technical solution in the embodiments of the present invention, optionally, the data request node includes a distributed computing node, the current data read request includes a computing task execution request sent by the distributed computing node, and the data to be read includes the shuffle data to be read when executing the computing task;

[0133] The file determination module 10 includes:

[0134] The file determination unit is used to determine the file to which the shuffle data to be read belongs based on the computing task execution request when it receives a computing task execution request sent by at least one distributed computing node.

[0135] The technical solution of this invention, based on the received current data read request, determines the file to which the data to be read belongs and the storage area of ​​the data to be read. If the file data in the file to which the data belongs has been cached in the file memory, the first file sub-data corresponding to the storage area is read from the file memory and sent as the data to be read to the data request node. Since the file data can be cached in the file memory through historical data read requests, when executing the current data read request after the historical data read request, the operation of directly reading the file memory is performed, which reduces the number of accesses to the local disk, reduces the local disk load, and achieves the effect of improving the efficiency and stability of data computing.

[0136] It is worth noting that in the embodiments of the above data reading device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0137] Figure 7 This is a schematic diagram of the structure of an electronic device implementing the data reading method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 7As shown, the electronic device 20 includes at least one processor 21 and a memory, such as a read-only memory (ROM) 22 or a random access memory (RAM) 23, communicatively connected to the at least one processor 21. The memory stores computer programs executable by the at least one processor. The processor 21 can perform various appropriate actions and processes based on the computer program stored in the ROM 22 or loaded from storage unit 28 into the RAM 23. The RAM 23 can also store various programs and data required for the operation of the electronic device 20. The processor 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output (I / O) interface 25 is also connected to the bus 24.

[0139] Multiple components in electronic device 20 are connected to I / O interface 25, including: input unit 26, such as keyboard, mouse, etc.; output unit 27, such as various types of monitors, speakers, etc.; storage unit 28, such as disk, optical disk, etc.; and communication unit 29, such as network card, modem, wireless transceiver, etc. Communication unit 29 allows electronic device 20 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 21 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 21 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 21 performs the various methods and processes described above, such as data reading methods.

[0141] In some embodiments, the data reading method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 28. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 20 via ROM 22 and / or communication unit 29. When the computer program is loaded into RAM 23 and executed by processor 21, one or more steps of the data reading method described above may be performed. Alternatively, in other embodiments, processor 21 may be configured to perform the data reading method by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0147] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0148] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data reading method characterized by, include: Based on the received current data read request, determine the file to which the data to be read belongs and the storage area of ​​the data to be read; Determine whether the file data in the file to which the file belongs has been cached in the file memory; If the file data in the file has been cached in the file memory, read the first file sub-data corresponding to the storage area from the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node; The file memory includes: file data in the file to which it belongs, cached based on historical data read requests, wherein the file data includes file sub-data corresponding to at least one storage segment; Determining whether the file data in the file to which the file belongs has been cached in file memory includes: If no file information corresponding to the file is found in the Bloom filter corresponding to the file, then the number of times the file data was requested in the historical time period before the data read request was received is determined. If the number of requests is less than a preset threshold, it is determined that the file data in the file to which the request belongs has not been cached in the file memory; The Bloom filter is used to store file information corresponding to file data that has been requested more than or equal to the preset threshold and has not been stored in the file memory.

2. The method of claim 1, wherein, Also includes: If the file data in the associated file is not cached in the file memory, retrieve the file data from the associated file and cache the file data in the file memory corresponding to the associated file; Read the first file sub-data corresponding to the storage area in the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node.

3. The method according to claim 2, characterized in that, The step of caching the file data in the file memory corresponding to the file to which it belongs includes: Determine the current remaining storage and total storage of the file; If the file data meets the preset conditions, the file data is cached in the file memory; The preset conditions include that the file's memory usage is less than or equal to a preset memory threshold, and that the sum of the file's memory usage and the current remaining storage is less than or equal to the total storage.

4. The method according to claim 3, characterized in that, Also includes: If the file data does not meet the preset conditions, then file information corresponding to the file to which the file belongs is generated based on the file data, and the file information is stored in the Bloom filter corresponding to the file to which the file belongs.

5. The method according to claim 3, characterized in that, The step of determining whether the file data in the file to which the file belongs has been cached in the file memory also includes: If the number of requests is greater than or equal to the preset threshold, then determine whether the cached operation record stored in advance includes the cached information of the file to which the request belongs; If the cache operation record includes the cache information, then it is determined that the file to which the cache belongs has been cached in the file memory.

6. The method according to claim 5, characterized in that, Also includes: If the cache operation record does not include the cache information; Alternatively, if the file information is found in the Bloom filter corresponding to the file, the second file sub-data corresponding to the storage area stored on the local disk is read. The second file sub-data is identified as the data to be read, and the data to be read is sent to the data request node.

7. The method according to claim 1, characterized in that, After reading the first file sub-data corresponding to the storage area from the file memory, the method further includes: Determine the current storage duration of the first file sub-data. If the current storage duration is greater than or equal to a preset storage duration, then clear the first file sub-data from the file memory corresponding to the file to which it belongs. Based on the file memory usage of the file data, the total storage capacity of the file memory, and the memory usage of the first file sub-data, update the current remaining storage capacity of the file memory.

8. The method according to claim 1, characterized in that, Also includes: In the file memory, a third file sub-data that has been read less than a preset threshold number within a preset time period is identified, and data cleaning operations are performed on the third file sub-data.

9. The method according to any one of claims 1-8, characterized in that, The data request node includes a distributed computing node, the current data read request includes a computing task execution request sent by the distributed computing node, and the data to be read includes the shuffle data to be read when executing the computing task; The step of determining the file to which the data to be read belongs based on the received current data read request includes: When a computing task execution request is received from at least one distributed computing node, the file to which the shuffle data to be read belongs is determined based on the computing task execution request.

10. A data reading device, characterized in that, include: The file ownership determination module is used to determine the file to which the data to be read belongs and the storage area of ​​the data to be read based on the received current data read request; The data to be read sending module is used to read the first file sub-data corresponding to the storage area from the file memory when the file data in the file has been cached in the file memory, determine the first file sub-data as the data to be read, and send the data to be read to the data request node; The file memory includes: file data in the file to which it belongs, cached based on historical data read requests, wherein the file data includes file sub-data corresponding to at least one storage segment; The device further includes a file data caching module for determining whether the file data in the file has been cached in the file memory; The file data caching module includes: The request count determination unit is used to determine the number of times the file data was requested in the historical time period before the data read request was received if no file information corresponding to the file is found in the Bloom filter corresponding to the file to which the file belongs; if the number of requests is less than a preset threshold, it is determined that the file data in the file to which the file belongs has not been cached in the file memory. The Bloom filter is used to store file information corresponding to file data that has been requested more than or equal to the preset threshold and has not been stored in the file memory.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data reading method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data reading method according to any one of claims 1-9.

Images