Data writing method, apparatus, and computing device

Abstract

This invention discloses a data writing method, apparatus, and computing device. The method includes: monitoring file update events of a directory to be tracked in a storage system; updating information in the target memory according to the file information of the updated target file corresponding to the file update event; deleting the directory tracking file corresponding to the directory to be tracked that is already stored in the storage system; generating an updated directory tracking file corresponding to the directory to be tracked based on the information in the target memory and uploading it to the storage system; and when a management requirement for the directory to be tracked is detected, reading the directory tracking file corresponding to the directory to be tracked from the storage system and cleaning up invalid file data under the directory to be tracked based on the read directory tracking file. Through the above method, the performance of data writing to the storage system can be improved, and the generation of invalid data can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and in particular to a data writing method, device and computing equipment. BACKGROUND

[0002] HDFS (Hadoop Distributed File System) is the default file-based storage system of the big data ecosystem, and many big data computing engines are designed and implemented based on its API. Unlike HDFS, object storage is a way to store data. Under the trend of separating storage and computing, many enterprises try to use object storage to build a database solution, which directly promotes more and more big data computing engines to use object storage as a storage system. Big data computing engines can use HDFS semantics to access object storage, build computing and analysis platforms, and meet multi-dimensional scene analysis needs.

[0003] When big data computing engines use HDFS semantics to write data into object storage, in order to prevent the generation of dirty data, the intermediate results of computing and analysis are generally written into a temporary directory, and after all analysis results are written to disk, the temporary directory is renamed to the final directory. Due to the limitations of object storage, the functions and performance are very limited, and once the data is written into the object storage, the data object cannot be changed. Therefore, the HDFS semantic directory renaming operation based on object storage will traverse all the files under the directory to copy and delete them. The complexity of the directory renaming operation is positively correlated with the size and number of files under the directory. The more files and the larger the files, the higher the complexity of the renaming operation of the directory, and the greater the performance impact. In addition, the renaming operation using HDFS semantics will also cause atomicity problems, that is, the operation is divided into two operations of copying and deleting, and it is also easy to cause inconsistent user data views. SUMMARY

[0004] In view of the above problems, the present application is proposed in order to provide a data writing method, device and computing equipment which can overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present application, a data writing method is provided, the method comprising:

[0006] Monitoring a file update event of a to-be-tracked directory in a storage system;

[0007] Updating information in a target memory according to file information of an update target file corresponding to the file update event;

[0008] Deleting a directory tracking file corresponding to the to-be-tracked directory which has been stored in the storage system;

[0009] According to the information in the target memory, an updated directory tracking file corresponding to the to-be-tracked directory is generated and uploaded to the storage system;

[0010] When the management demand for the to-be-tracked directory is monitored, the directory tracking file corresponding to the to-be-tracked directory is read from the storage system, and invalid file data under the to-be-tracked directory is cleaned according to the read directory tracking file.

[0011] Optionally, updating the information in the target memory according to the file information of the updated target file corresponding to the file update event further includes:

[0012] If a file addition event of the to-be-tracked directory is monitored, the file information of the added file corresponding to the file addition event is added to the target memory;

[0013] If a file deletion event of the to-be-tracked directory is monitored, the file information of the to-be-deleted file corresponding to the file deletion event is deleted from the target memory.

[0014] Optionally, the directory tracking file further includes generation time information; reading the directory tracking file corresponding to the to-be-tracked directory from the storage system further includes:

[0015] If the storage system stores a plurality of directory tracking files corresponding to the to-be-tracked directory, the directory tracking file with the generation time information closest to the current time information is read from the plurality of directory tracking files.

[0016] Optionally, the directory tracking file further includes file expiration time information of each file under the to-be-tracked directory; cleaning the invalid file data under the to-be-tracked directory according to the read directory tracking file further includes:

[0017] According to the file expiration time information of each file contained in the read directory tracking file, the data block of the expired file under the to-be-tracked directory is deleted.

[0018] Optionally, cleaning the invalid file data under the to-be-tracked directory according to the read directory tracking file further includes:

[0019] According to the directory tracking file read this time and the directory tracking file read last time, the to-be-cleaned file under the to-be-tracked directory is determined, and the data block of the to-be-cleaned file is deleted.

[0020] Optionally, the directory tracking file further includes file size information of each file under the to-be-tracked directory; the method further includes:

[0021] According to the file size information of each file contained in the read directory tracking file, the file data amount under the to-be-tracked directory is counted.

[0022] Optionally, the method further includes: determining the various information items contained in the directory tracking file based on the directory management requirements dimension;

[0023] Based on the information in the target memory, generating the updated directory tracking file corresponding to the directory to be tracked also includes:

[0024] Based on the information corresponding to each information item contained in the target memory, an updated directory tracking file is generated for the directory to be tracked.

[0025] According to another aspect of the present invention, a data writing apparatus is provided, the apparatus comprising:

[0026] The monitoring module is suitable for monitoring file update events in a directory to be tracked in a storage system;

[0027] The information update module is suitable for updating the file information of the target file and the information in the target memory according to the file update event.

[0028] The file deletion module is suitable for deleting directory tracking files corresponding to the directories to be tracked that are already stored in the storage system;

[0029] The file generation module is suitable for generating updated directory tracking files corresponding to the directories to be tracked based on the information in the target memory.

[0030] The upload module is suitable for uploading updated directory tracking files to the storage system;

[0031] The acquisition module is suitable for reading the directory tracking file corresponding to the directory to be tracked from the storage system when a management need for the directory to be tracked is detected.

[0032] The data cleaning module is suitable for cleaning up invalid file data in the directory to be tracked based on the directory files read.

[0033] Optionally, the information update module is further adapted to: if a file addition event is detected in the directory to be tracked, add the file information of the newly added file corresponding to the file addition event to the target memory; if a file deletion event is detected in the directory to be tracked, delete the file information of the file to be deleted corresponding to the file deletion event from the target memory.

[0034] Optionally, the directory tracking file also includes generation time information; the acquisition module is further adapted to: if the storage system stores multiple directory tracking files corresponding to the directory to be tracked, read the directory tracking file whose generation time information is closest to the current time information among the multiple directory tracking files.

[0035] Optionally, the directory tracking file also includes: file expiration time information of each file in the directory to be tracked; the data cleaning module is further adapted to: delete the data blocks of expired files in the directory to be tracked based on the file expiration time information of each file contained in the directory tracking file.

[0036] Optionally, the data cleaning module is further adapted to: determine the files to be cleaned in the directory to be tracked based on the directory tracking file read this time and the directory tracking file read last time, and delete the data blocks of the files to be cleaned.

[0037] Optionally, the directory tracking file also includes: file size information of each file in the directory to be tracked; the device also includes: a statistics module, adapted to count the amount of file data in the directory to be tracked based on the file size information of each file contained in the directory tracking file.

[0038] Optionally, the device further includes: an information management module, adapted to determine each information item contained in the directory tracking file according to the directory management requirements; and a file generation module, further adapted to: generate an updated directory tracking file corresponding to the directory to be tracked based on the information corresponding to each information item contained in the target memory.

[0039] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0040] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above data writing method.

[0041] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the data writing method described above.

[0042] According to the data writing method, apparatus, and computing device of the present invention, the following steps are taken: First, monitor file update events of the target directory in the storage system. Then, update the information in the target memory based on the file information of the updated target file corresponding to the file update event. Next, delete the directory tracking file corresponding to the target directory already stored in the storage system. Then, generate an updated directory tracking file corresponding to the target directory based on the information in the target memory and upload it to the storage system. When a management requirement for the target directory is detected, read the directory tracking file corresponding to the target directory from the storage system and clean up invalid file data in the target directory based on the read directory tracking file. Through this method, data is directly written to the target directory, and the file information of the target directory is identified by a custom directory tracking file. The directory tracking file is dynamically generated continuously following changes in the target directory's files, and the directory tracking file is obtained and parsed to clean up dirty data in the target directory. This avoids the renaming operation during data writing to the object storage process and also prevents dirty data from being generated in the target directory, greatly improving data writing performance and avoiding inconsistencies in the user's directory view.

[0043] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1 A flowchart of the data writing method provided in an embodiment of the present invention is shown;

[0046] Figure 2 A flowchart of a data writing method provided by another embodiment of the present invention is shown;

[0047] Figure 3 A schematic diagram of the data writing device provided in an embodiment of the present invention is shown;

[0048] Figure 4a A schematic diagram of the system architecture in another embodiment of the present invention is shown;

[0049] Figure 4b A schematic diagram of the structure of a directory file tracking system according to another embodiment of the present invention is shown;

[0050] Figure 5a A schematic diagram of the directory tracking file format is shown in one embodiment of the present invention;

[0051] Figure 5b This diagram illustrates the format of an extended directory tracking file in one embodiment of the present invention.

[0052] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0053] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0054] Figure 1 A flowchart of a data writing method provided by an embodiment of the present invention is shown. This method can be applied to any device with computing capabilities. Figure 1 As shown, the method includes the following steps:

[0055] Step S110: Monitor file update events of the directory to be tracked in the storage system.

[0056] The storage system can consist of object storage. When writing data to the object storage, it will monitor the addition of new files and deletion of existing files in the directory to be tracked, and monitor the addition or deletion of files in the directory to be tracked.

[0057] Step S120: Update the information in the target memory according to the file information of the target file corresponding to the file update event.

[0058] When an update to a file in the target directory is detected, such as writing a new file to the target directory or deleting a file in the target directory, the information in the target memory is updated based on the file information of the updated target file.

[0059] Step S130: Delete the directory tracking file corresponding to the directory to be tracked that is already stored in the storage system.

[0060] Delete the directory tracking file corresponding to the directory to be tracked that is already stored in the object storage. The directory tracking file is used to identify the file content under the directory to be tracked.

[0061] Step S140: Based on the information in the target memory, generate an updated directory tracking file corresponding to the directory to be tracked and upload it to the storage system.

[0062] After updating the file information in the target memory, an updated directory tracking file is generated based on the file information in the target memory, and the updated directory tracking file is uploaded to object storage.

[0063] Step S150: When a management requirement for a directory to be tracked is detected, the directory tracking file corresponding to the directory to be tracked is read from the storage system, and invalid file data under the directory to be tracked is cleaned up according to the read directory tracking file.

[0064] For example, when a scheduled time is reached, if there is a need to manage a directory to be tracked, the corresponding directory tracking file for the directory to be tracked is read from the storage system. Based on the read directory tracking file, invalid files under the directory to be tracked are identified, and the data blocks of the invalid files are deleted, thereby deleting the dirty data under the directory to be tracked.

[0065] In existing technologies, during offline computation and analysis using Spark based on object storage, result files are named with the .temp extension or as is. Once the computation task is complete, the result file is immediately renamed to the final file. If the analysis result file is very large at this point, the file renaming operation undoubtedly has a significant impact on Spark analysis performance.

[0066] According to the data writing method of the present invention, data is directly written to the target directory, and the file information of the target directory is identified by a custom directory tracking file. The directory tracking file is dynamically generated continuously as the files in the target directory change, and the directory tracking file is obtained and parsed to clean up dirty data in the target directory. This method avoids the renaming operation in the data writing object storage process, and also avoids the generation of dirty data in the target directory. The data writing performance is greatly improved, and the inconsistency of the user's directory view is also avoided.

[0067] Figure 2 A flowchart of a data writing method according to another embodiment of the present invention is shown, which can be applied to any device with computing capabilities. Figure 2 As shown, the method includes the following steps:

[0068] Step S210: Based on the tracking registration request, register and generate the directory to be tracked in the storage system.

[0069] For example, a big data engine initiates a tracking registration request, registers and generates a directory to be tracked in the storage system, and then new files can be added or files can be deleted from the directory.

[0070] Step S220: Monitor file update events of the directory to be tracked in the storage system.

[0071] Detect file updates in the directory to be tracked. If a file addition event is detected in the directory to be tracked, proceed to step S230; if a file deletion event is detected in the directory to be tracked, proceed to step S240.

[0072] Step S230: Add the file information of the newly added file corresponding to the file addition event to the target memory.

[0073] Among them, the file addition event can be a file addition command or a file addition operation. When a new file is detected in the directory to be tracked, the file information of the new file is added to the target memory. The file information can include: file name and file size. That is, the directory tracking file contains the file name and file size information of each file in the directory.

[0074] Step S240: Remove the file information of the file to be deleted corresponding to the file deletion event from the target memory.

[0075] Among them, the file deletion event can be a file deletion command. When a file deletion command is detected in the directory to be tracked, the file information of the corresponding file to be deleted is removed from the target memory.

[0076] Step S250: Delete the directory tracking file corresponding to the directory to be tracked that is already stored in the storage system.

[0077] Delete the directory tracking files corresponding to the directory to be tracked that are already stored in the storage system.

[0078] Step S260: Based on the information in the target memory, generate an updated directory tracking file corresponding to the directory to be tracked and upload it to the storage system.

[0079] Whenever a new file is added or a file is deleted in the directory to be tracked, the file information of the new file is appended to the target memory or the file information of the file to be deleted is deleted from the target memory. After the data in the target memory is updated, a new directory tracking file is generated based on the information in the target memory, and the directory tracking file corresponding to the directory to be tracked in the storage system is replaced with the new directory tracking file.

[0080] Step S270: When a management requirement for a directory to be tracked is detected, if the storage system stores multiple directory tracking files corresponding to the directory to be tracked, read the directory tracking file whose generation time information is closest to the current time information among the multiple directory tracking files, and clean up invalid file data under the directory to be tracked based on the read directory tracking file.

[0081] Read the latest directory tracking file corresponding to the directory to be tracked from the storage system, parse the read directory tracking file, identify invalid files in the directory to be tracked based on the parsing results, and clean up the data blocks of the invalid files.

[0082] In this embodiment, whenever a file in the directory to be tracked changes, a new directory tracking file is generated to replace the existing directory tracking file in the storage system. Therefore, theoretically, there is always only one directory tracking file for the directory to be tracked in the storage system, which is the directory tracking file generated by the most recent file update of the directory to be tracked.

[0083] However, in practical applications, there may be situations where the storage system stores two directory tracking files corresponding to the directory to be tracked. For example, if the system crashes before deleting the old directory tracking file when a new one is generated, two directory tracking files will exist after the system restarts. In this case, it is necessary to analyze which directory tracking file should be read. Specifically, when the directory tracking file is generated, its generation time can be included in the file. Then, the directory tracking file with the generation time closest to the current time can be read. For example, if the generation timestamp is included in the directory tracking file, the directory tracking file with the largest timestamp can be read. In this way, the directory tracking file that accurately identifies the files in the directory to be tracked can be read.

[0084] In an optional approach, the directory tracking file also includes file expiration time information for each file in the directory. Therefore, cleaning up invalid file data in the directory to be tracked based on the read directory tracking file further includes: deleting the data blocks of expired files in the directory to be tracked based on the file expiration time information of each file contained in the read directory tracking file. Specifically, based on the expiration time of each file contained in the directory tracking file, expired files are identified, and the data blocks of the expired files are deleted.

[0085] In one alternative approach, when file deletion operations are detected in the directory to be tracked, file deletion is temporarily suspended. Instead, file data blocks are deleted during directory management based on the directory tracking file. Specifically, based on the currently read directory tracking file and the previously read directory tracking file, the files to be cleaned up in the directory to be tracked are identified, and the data blocks of these files are deleted. By comparing the names of each file contained in the currently read directory tracking file with the names of each file contained in the previously read directory tracking file, the filenames of those files that are included in the previously read directory tracking file but not in the currently read directory tracking file are identified. The file corresponding to this filename is the file to be cleaned up, and its data blocks are then deleted.

[0086] In an alternative approach, the size of the files in the directory can also be calculated based on the directory tracking file. Specifically, the amount of file data in the directory to be tracked can be calculated based on the file size information of each file contained in the directory tracking file.

[0087] In existing technologies, obtaining the file data volume of a directory requires iteratively traversing and retrieving the size of all files in the directory within object storage. The complexity of obtaining the directory data volume under object storage is positively correlated with the number of subdirectories and files within the directory. That is, the more subdirectories and files there are, the greater the network throughput, but the greater the latency, complexity, and performance of obtaining the directory data volume. The method of this invention eliminates the need to query the size of each file in the directory to be tracked individually; it only requires parsing the directory tracking files of the directory to be tracked to statistically determine the file data volume of the directory.

[0088] In one alternative approach, to enable more granular management of the directory, the directory tracking file also includes information for directory authentication management, specifically including: permission identifiers, owner identifiers, and / or group identifiers. Specifically, the identity of the initiator of the management request is authenticated based on the authentication identifiers, and if the authentication is successful, the initiator is allowed to obtain other file information in the directory tracking file. Information for directory quota management can also be added to the directory tracking file, specifically including: maximum capacity and / or maximum number of files, so that the initiator of the management request can understand the capacity of the directory to be tracked.

[0089] Existing technologies cannot manage directory operations based on object storage. Object storage allows for coarse-grained management at the bucket level during bucket creation, such as directly restricting read / write access to another sub-account or coarsely limiting the bucket size. Furthermore, once a bucket is created, these permission settings cannot be modified. This is inadequate for fine-grained HDFS semantic directory management, such as directory authentication, quotas, and file expiration policies. This invention, by extending authentication, quota, and file expiration information into the directory tracking file, can also implement directory authentication, quota, and file expiration policies.

[0090] In another optional approach, the information recorded in the directory tracking file can be customized according to management needs. Based on the dimensions of directory management requirements, the various information items included in the directory tracking file are determined. Then, based on the information corresponding to each information item contained in the target memory, an updated directory tracking file for the directory to be tracked is generated. Information items include: file name, file size, file expiration time, permission identifier, owner identifier, maximum number of files, maximum capacity, timestamp, etc. For example, based on the type of directory to be tracked, the multiple dimensions of information items that the directory tracking file needs to record are determined. Some fixed information is pre-added to the target memory, while some dynamically changing information is updated in the target memory each time a file is updated. Through this method, users can customize the file information recorded in the directory tracking file to meet the initiator's personalized directory tracking needs.

[0091] According to the data writing method of this invention, data is directly written to the directory, a directory tracking file for identifying directory file information is dynamically generated, and dirty data in the directory is cleaned up periodically by acquiring and parsing the directory tracking file. This can avoid directory renaming operations (i.e., avoid copying and deleting large amounts of data blocks in object storage), while avoiding the generation of dirty data in the directory and the time-consuming migration of large amounts of data blocks in object storage. This significantly improves the performance of writing massive amounts of data to object storage, solves the problem of operation atomicity, and avoids inconsistencies in user directory views. Furthermore, by replacing multiple subdirectory network queries with a single acquisition and parsing of the directory tracking file, the performance of querying directory data of various sizes is also greatly improved.

[0092] Figure 3 A schematic diagram of the data writing device provided in an embodiment of the present invention is shown, as follows: Figure 3 As shown, the device includes:

[0093] Monitoring module 31 is suitable for monitoring file update events in a directory to be tracked in a storage system;

[0094] Information update module 32 is adapted to update the information in the target memory according to the file information of the target file corresponding to the file update event;

[0095] File deletion module 33 is suitable for deleting directory tracking files corresponding to the directory to be tracked that are already stored in the storage system;

[0096] The file generation module 34 is adapted to generate an updated directory tracking file corresponding to the directory to be tracked based on the information in the target memory.

[0097] Upload module 35 is suitable for uploading updated directory tracking files to the storage system;

[0098] The acquisition module 36 is adapted to read the directory tracking file corresponding to the directory to be tracked from the storage system when a management requirement for the directory to be tracked is detected.

[0099] The data cleaning module 37 is suitable for cleaning up invalid file data in the directory to be tracked based on the directory tracking files read.

[0100] In an alternative approach, the information update module 32 is further adapted to: if a file addition event is detected in the directory to be tracked, add the file information of the newly added file corresponding to the file addition event to the target memory; if a file deletion event is detected in the directory to be tracked, delete the file information of the file to be deleted corresponding to the file deletion event from the target memory.

[0101] In an alternative approach, the directory tracking file also includes generation time information; the acquisition module 36 is further adapted to: if the storage system stores multiple directory tracking files corresponding to the directory to be tracked, read the directory tracking file whose generation time information is closest to the current time information among the multiple directory tracking files.

[0102] In an alternative approach, the directory tracking file further includes: file expiration time information of each file in the directory to be tracked; the data cleaning module 37 is further adapted to: delete the data blocks of expired files in the directory to be tracked based on the file expiration time information of each file contained in the directory tracking file.

[0103] In an alternative manner, the data cleaning module 37 is further adapted to: determine the files to be cleaned in the directory to be tracked based on the directory tracking file read this time and the directory tracking file read last time, and delete the data blocks of the files to be cleaned.

[0104] In one alternative approach, the directory tracking file further includes: file size information of each file in the directory to be tracked; the device further includes: a statistics module adapted to count the amount of file data in the directory to be tracked based on the file size information of each file contained in the directory tracking file.

[0105] In an alternative embodiment, the apparatus further includes: an information management module adapted to determine each information item contained in the directory tracking file according to the directory management requirements dimension; and a file generation module 34 further adapted to: generate an updated directory tracking file corresponding to the directory to be tracked based on the information corresponding to each information item contained in the target memory.

[0106] Figure 4a A schematic diagram of the system architecture in another embodiment of the present invention is shown, as follows: Figure 4a As shown, the directory file tracking system is embedded within the big data computing engine, and it includes a directory operation interface module, a directory tracking module, and a directory monitoring center.

[0107] The directory operation interface module is responsible for registering the directories to be tracked with the directory supervision center. It is the only directory operation entry point for the big data computing engine for the directory to be tracked.

[0108] The directory tracking module is responsible for managing directory tracking files. When a file in the directory to be tracked changes, the corresponding data in memory, the tracking file on the local disk, and the object storage will all change.

[0109] The directory monitoring center is responsible for monitoring and managing the tracking directory, cleaning up dirty data files, and can monitor multiple tracking directories.

[0110] Figure 4bA schematic diagram of the structure of a directory file tracking system according to another embodiment of the present invention is shown, such as... Figure 4b As shown, the monitoring module, information update module, file generation module, file deletion module, upload module, and information management module in the data writing device are located in the directory tracking module of the directory file tracking system, while the acquisition module, data cleaning module, and statistics module in the data writing device are located in the directory supervision center of the directory tracking system.

[0111] Figure 5a This diagram illustrates the format of a directory tracking file in one embodiment of the present invention. Figure 5b The diagram illustrates the format of an extended directory tracking file in one embodiment of the present invention. In comparison, the extended directory tracking file contains more dimensions of file information, which is used to achieve fine-grained management based on HDFS semantic directory operations.

[0112] First, the big data computing engine registers the directories to be tracked with the directory monitoring center through the directory operation interface module. Second, the directory operation interface module generates directory tracking files T1 in memory and on the local disk through the directory tracking module. Directory tracking file T1 contains information such as the timestamp of its generation, file name, and file size.

[0113] After the directory tracking file T1 is generated, the directory tracking module uploads the local directory tracking file T1 to object storage.

[0114] When a new file is generated in the directory to be tracked, the new file name and file size are immediately appended to memory, the memory data is re-diskmed to the directory tracking file T2, the directory tracking file T2 is then uploaded to object storage, and the directory tracking file T1 is deleted.

[0115] When a file is deleted from the directory to be tracked, the file name and file size are immediately removed from memory. Then, the updated data in memory is diskized to the local directory tracking file T1. The directory tracking file T1 is then uploaded to object storage, and the directory tracking file T2 is deleted.

[0116] In other words, a directory tracking file reflects the contents of the corresponding directory at a given moment. When the directory changes (i.e., files are added or removed), the directory tracking file also changes; that is, the directory tracking file reflecting the directory contents is switched, from directory tracking file T1 to directory tracking file T2, or directory tracking file T2 to directory tracking file T1. The reason why the old directory tracking file is not directly modified is that data in object storage does not support modification. Therefore, a new directory tracking file is dynamically generated to replace the old one as the files in the directory are updated.

[0117] The directory monitoring center obtains directory tracking files through the directory tracking module. The logic for obtaining the directory tracking file for a specified directory is as follows: First, it determines whether the directory tracking file exists. If it exists, it further determines whether there are two directory tracking files. If there are two directory tracking files, namely directory tracking file T1 and directory tracking file T2, then the timestamps of the two tracking files are compared, and the directory tracking file with the larger timestamp is read. If there is only one tracking file in the object storage, regardless of whether it is directory tracking file T1 or directory tracking file T2, that directory tracking file is selected and appended to the directory monitoring center. The directory monitoring center dynamically and periodically manages the files under the directory based on the obtained directory tracking file.

[0118] The big data computing engine perceives directory content by tracking files in the directory. Even if dirty data is generated in the destination directory, the big data computing engine will not be aware of the existence of dirty data until the directory file tracking system cleans up the dirty data periodically. This embodiment of the solution ensures data writing performance while also avoiding the generation of dirty data.

[0119] For example, in existing technology, three files—Child_A / File_A, Child_B / File_B, and File_C—are first written to a temporary directory, and then the temporary directory is renamed to the Parent directory. This renaming operation involves copying and deleting object storage data blocks, which has a significant impact on the performance of writing massive amounts of data to object storage.

[0120] Using the method of this embodiment, three files are directly written to the target directory Parent: / Parent / Child_A / File_A, / Parent / Child_B / File_B, and / Parent / File_C. After these three files are written, a directory tracking file T1 is generated. Directory tracking file T1 contains a generation timestamp, the filenames and sizes of Child_A / File_A, Child_A / File_B, and Child_A / File_C. If a deletion operation of File_C under the Parent directory is detected at a certain point, it is not necessary to immediately delete the File_C data block. Only the data in memory needs to be modified, and the result is atomically output to the object storage as directory tracking file T2. Directory tracking file T2 contains a generation timestamp, the filenames and sizes of Child_A / File_A and Child_A / File_B. The directory monitoring center will obtain directory tracking file T2 and clean up the expired file File_C. To count the amount of file data in the directory Parent, it is not necessary to query the subdirectories Child_A and Child_B under Parent separately; only the directory tracking file T2 needs to be parsed and read. Any changes to files in the directory will cause the tracking file to be regenerated, and the tracking file name will alternate between T1 and T2.

[0121] Currently, the industry has made some optimizations to the renaming operation based on object storage systems, which can be broadly categorized into two methods. The first method involves adding a database-like system between the object storage system and the computing analysis engine to manage the mapped directory file names. This method requires the database system to adapt to the existing big data computing engine and object storage business system, necessitating the development of multiple connection devices. This is not only complex but also impacts system robustness. The second method conceptually proposes refactoring the object storage service client to add support for renaming operations, similar to server-side support for modifying object keywords. However, this method is more complex, only a conceptual optimization, and has low operability.

[0122] The data writing method of this invention can improve the performance of directory file-related operations in big data computing engines. For example, HBase based on HDFS uses a large number of table temporary directories, which are mainly used to store intermediate results during the Flush and Compaction processes. Taking Flush as an example, the KV data in MemStore is first generated into HFiles in the .tmp directory before being moved to the corresponding actual file directory. Renaming table temporary directories in HBase using object storage involves a large number of data block copies and deletions, and the operations are not atomic. HBase data directory tracing technology can solve these problems. KV data in MemStore is written to the actual file directory, and after all data writing operations are completed, the filenames are appended to the directory tracing file. At this point, HBase can merge these Store HFiles. Similarly, when Spark SQL saves Hive query results to object storage, it first writes to the tmp directory and then renames the files in that directory to the final directory. Directory file tracing technology can also solve the performance problems of Spark on Hive and the atomicity of file diskization.

[0123] The system in this invention fully utilizes existing computing and storage resources. It reuses an object storage system for maintaining the state of directory tracking files, and the dynamic directory tracking logic is embedded within a big data computing engine. Specifically, custom directory tracking files are dynamically and alternately generated in the object storage system, while file retrieval and parsing are handled within the big data computing engine. While addressing performance and security issues, the system maintains the original business system architecture unchanged. Furthermore, it can simultaneously track directory files in multiple directories. The system is compact, has low dependence on computing resources, and offers high flexibility.

[0124] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the data writing method in any of the above method embodiments.

[0125] Figure 6 The diagram shows a structural schematic of an embodiment of the computing device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0126] like Figure 6 As shown, the computing device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0127] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608. Communication interface 604 is used to communicate with other network elements such as clients or other servers. The processor 602 executes program 610, specifically performing the relevant steps in the above-described embodiment of the data writing method for computing devices.

[0128] Specifically, program 610 may include program code that includes computer operation instructions.

[0129] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0130] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0131] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0132] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0133] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0134] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0135] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0136] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0137] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A data writing method, characterized in that, The method includes: Monitor file update events in the directory to be tracked in the storage system; Update the information in the target memory according to the file information of the target file corresponding to the file update event; Delete the directory tracking file corresponding to the directory to be tracked that is already stored in the storage system; Based on the information in the target memory, an updated directory tracking file corresponding to the directory to be tracked is generated and uploaded to the storage system; When a management requirement for the directory to be tracked is detected, the directory tracking file corresponding to the directory to be tracked is read from the storage system, and invalid file data under the directory to be tracked is cleaned up according to the read directory tracking file.

2. The method according to claim 1, characterized in that, The step of updating the file information of the target file according to the file update event further includes: If a file addition event is detected in the directory to be tracked, the file information of the newly added file corresponding to the file addition event is added to the target memory; If a file deletion event is detected in the directory to be tracked, the file information of the file to be deleted corresponding to the file deletion event is deleted from the target memory.

3. The method according to claim 1, characterized in that, The directory tracking file also includes generation time information; the step of reading the directory tracking file corresponding to the directory to be tracked from the storage system further includes: If the storage system stores multiple directory tracking files corresponding to the directory to be tracked, read the directory tracking file whose generation time information is closest to the current time information among the multiple directory tracking files.

4. The method according to claim 1, characterized in that, The directory tracking file also includes: file expiration time information for each file in the directory to be tracked; the step of cleaning up invalid file data in the directory to be tracked based on the read directory tracking file also includes: Based on the file expiration time information of each file contained in the directory tracking file, delete the data blocks of expired files in the directory to be tracked.

5. The method according to claim 1, characterized in that, The step of cleaning up invalid file data in the directory to be tracked based on the read directory tracking file also includes: Based on the directory tracking file read this time and the directory tracking file read last time, determine the files to be cleaned in the directory to be tracked, and delete the data blocks of the files to be cleaned.

6. The method according to claim 1, characterized in that, The directory tracking file further includes: file size information of each file in the directory to be tracked; the method further includes: Based on the file size information of each file contained in the directory tracking file, the amount of file data in the directory to be tracked is counted.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: determining the various information items contained in the directory tracking file based on the directory management requirements dimension; The step of generating an updated directory tracking file corresponding to the directory to be tracked based on the information in the target memory further includes: Based on the information corresponding to each information item contained in the target memory, an updated directory tracking file corresponding to the directory to be tracked is generated.

8. A data writing device, characterized in that, The device includes: The monitoring module is suitable for monitoring file update events in a directory to be tracked in a storage system; The information update module is adapted to update the information in the target memory according to the file information of the target file corresponding to the file update event; The file deletion module is adapted to delete the directory tracking files corresponding to the directory to be tracked that are already stored in the storage system; The file generation module is adapted to generate an updated directory tracking file corresponding to the directory to be tracked based on the information in the target memory. An upload module is adapted to upload the updated directory tracking file to a storage system; The acquisition module is adapted to read the directory tracking file corresponding to the directory to be tracked from the storage system when a management need for the directory to be tracked is detected. The data cleaning module is adapted to clean up invalid file data in the directory to be tracked based on the directory tracking files read.

9. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the data writing method as described in any one of claims 1-7.

10. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the data writing method as described in any one of claims 1-7.

Images