Distributed report file management method, system, electronic device and storage medium

Abstract

The present invention discloses a distributed report file management method, system, electronic device and storage medium, and relates to the field of file system management technology. The method includes: obtaining a report file directory creation request through a client and sending it to a file directory management service; constructing a current report file directory based on the current report file directory information through the file directory management service, and generating a current directory log; performing a disk flushing operation on the current directory log, solidifying the current directory log into the corresponding file storage node service, and feeding back the file storage node service to the client; uploading the current report file to the current storage node directory of the file storage node service based on the current report file directory through the client. The technical solution of the embodiment of the present invention reduces the difficulty of operation and maintenance during actual use, and improves the availability, consistency and security of the report files stored in the distributed report file management system.

Description

Technical Field

[0001] The present invention relates to the technical field of file system management, and in particular to a distributed report file management method, system, electronic equipment and storage medium. Background Art

[0002] In the era of digital transformation, business systems are shifting from isolated, isolated systems to integrated, collaborative, and shared ones. Demand for WYSIWYG functionality is increasing, and the need for business system convenience is becoming increasingly demanding. As a foundational component of business systems, report file management systems play a crucial role in low-code business system implementations. The process of defining report templates and generating and displaying reports generates a large number of report files, creating challenges for managing these files within the report file management system.

[0003] Existing technologies primarily manage report files through open source frameworks. However, these frameworks are cumbersome, have redundant functions, and present operational and maintenance difficulties. For example, when data issues arise, it's difficult to locate the cause, and it's also difficult to ensure the availability, consistency, and security of report files stored in the report file management system. Summary of the Invention

[0004] The present invention provides a distributed report file management method, system, electronic device and storage medium, which reduce the difficulty of operation and maintenance during actual use and improve the availability, consistency and security of report files stored in the distributed report file management system.

[0005] According to one aspect of the present invention, a distributed report file management method is provided, the method comprising:

[0006] Obtain the report file directory creation request through the client and send it to the file directory management service;

[0007] By means of the file directory management service, a current report file directory is constructed according to the current report file directory information included in the report file directory creation request, and a current directory log is generated;

[0008] Through the file directory management service, a disk flushing operation is performed on the current directory log, the current directory log is solidified to the corresponding file storage node service, and the file storage node service is fed back to the client;

[0009] The client uploads the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory.

[0010] According to another aspect of the present invention, a distributed report file management system is provided, the system comprising: a client, a file directory management service and the file storage node service; wherein,

[0011] The client is used to obtain a report file directory creation request and send it to the file directory management service;

[0012] The file directory management service is used to build the current report file directory according to the current report file directory information included in the report file directory creation request, and generate a current directory log;

[0013] The file directory management service is used to perform a disk flushing operation on the current directory log, solidify the current directory log to the corresponding file storage node service, and feed back the file storage node service to the client;

[0014] The client is configured to upload the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the distributed report file management method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed report file management method according to any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention is to obtain a report file directory creation request through a client and send it to a file directory management service. The file directory management service constructs a current report file directory based on the current report file directory information contained in the report file directory creation request, generates a current directory log, performs a disk flushing operation on the current directory log, solidifies the current directory log into a corresponding file storage node service, and feeds back the file storage node service to the client. The client uploads the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory. The report file directory is managed by the file directory management service, and the report file is stored accordingly by the file storage node service. This realizes distributed management of report files, avoids the problem of functional redundancy of the report file management system, realizes lightweight management of distributed report files, and facilitates the operation and maintenance of the report file management system during actual use. For example, when a problem occurs with the data, the file directory management service and the file storage node service cooperate with each other to quickly locate the cause of the data occurrence, thereby improving the availability, consistency and security of the report files stored in the report file management system.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of a distributed report file management method provided in accordance with the first embodiment of the present invention;

[0024] Figure 2 This is a flow chart of a distributed report file management method provided in accordance with the second embodiment of the present invention;

[0025] Figure 3 This is a flow chart of a distributed report file management method provided in accordance with the third embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a distributed report file management system provided according to a fourth embodiment of the present invention;

[0027] Figure 5The present invention is a schematic diagram of the structure of an electronic device for implementing the distributed report file management method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Example 1

[0031] Figure 1 This is a flow chart of a distributed report file management method provided in Example 1 of the present invention. This embodiment of the present invention is applicable to distributed management of report files. The method can be executed by a distributed report file management system, which can be implemented in hardware and / or software and can be configured in an electronic device that carries the distributed report file management function.

[0032] See also Figure 1 The distributed report file management method shown includes:

[0033] S110. Obtain a report file directory creation request through the client and send it to the file directory management service.

[0034] The client can be used to obtain a report file directory creation request issued by the report file uploader and receive the corresponding current report file. The report file directory creation request can be used to request the creation of a report file directory to implement the management of the report file. The report file directory creation request can be initiated by the report file uploader. Exemplarily, the report file directory creation request can include the current report file directory information and the current report file to be uploaded. The file directory management service can be used to generate the current report file directory, the current directory log, and solidify the current directory log to the corresponding file storage node service. Optionally, there can be multiple file directory management services.

[0035] Specifically, the client can obtain the report file directory creation request sent by the report file uploader, and send the report file directory creation request to the file directory management service.

[0036] S120 . Build the current report file directory according to the current report file directory information included in the report file directory creation request through the file directory management service, and generate a current directory log.

[0037] The current report file directory information can be used to generate the current report file directory. Exemplarily, the current report file directory information may include the report file directory path where the current report file is to be stored and the number of report files in each report folder contained in the report file directory path. The report file directory path refers to the storage path of the report file directory in a file directory management service (e.g., a directory knowledge manager). For example, the current report file directory information may include "Create report folder X2 within report folder X1 in the file directory management service to store X3 current report files."

[0038] The current report file directory can be the directory corresponding to the current report file in the file directory management service. The current report file directory can be managed by the file directory management service, for example, allowing the creation, viewing, and deletion of the current report file directory. The current report file directory can be displayed in the client. The current directory log can be used to record information about the current report file directory. The current directory log can be used as a reference for loading and replaying to generate directory knowledge snapshots.

[0039] Optionally, the file directory management service can establish a file read-write lock. Optionally, the file directory management service can establish a file read-write lock when receiving a report file directory creation request. The file directory management service can close the file read-write lock after the current directory log is solidified to the corresponding file storage node service.

[0040] Specifically, the current report file directory can be constructed based on the current report file directory information included in the report file directory creation request through a file read-write lock. The current directory log can be generated based on the current report file directory information included in the report file directory creation request through the file directory management service.

[0041] Exemplarily, the existence of a directory knowledge manager can be detected through a file read-write lock. When it is detected that a directory knowledge manager does not exist, a directory knowledge manager is created through a file read-write lock. When it is detected that a directory knowledge manager exists, the directory knowledge manager is called through a file read-write lock, and based on the current report file directory information, each directory knowledge node in the directory knowledge manager is queried to see whether a corresponding target directory knowledge node exists. When it is queried through a file read-write lock that a corresponding target directory knowledge node exists, the current main directory in the file directory management service is determined as the current report file directory. When it is queried through a file read-write lock that a corresponding target directory knowledge node does not exist, a target directory knowledge node is created in the file directory management service based on the current report file directory information, and the target directory knowledge node is added to the directory knowledge manager, and the current main directory in the file directory management service is updated to generate the current report file directory. By introducing a file read-write lock, the hierarchical structure of the report file directory is enriched.

[0042] The existence of a directory knowledge manager can be determined by checking whether the current primary directory in the file directory management service is empty. If the current primary directory in the file directory management service is empty, the directory knowledge manager does not exist; if the current primary directory in the file directory management service is not empty, the directory knowledge manager exists. The directory knowledge node can be information related to the report file directory, such as a report folder identifier, which can be used to identify the report folder.

[0043] S130: Perform a disk flushing operation on the current directory log through the file directory management service, solidify the current directory log into the corresponding file storage node service, and feed back the file storage node service to the client.

[0044] The file storage node service can be a file management system in the server. The file storage node service can be the actual storage space for report files. In comparison, the file directory management service can correspond to a virtual container in the memory; the file storage node service corresponds to the physical storage space in the server. There is a one-to-one correspondence between the file storage node service and the file directory management service. There is a mapping relationship between the report file directory in the file directory management service and the actual physical address in the file storage node service. Optionally, the number of file directory management services can be multiple; accordingly, the number of file storage node services can also be multiple. Optionally, each file storage node service can periodically synchronize the file system it manages and register the file system it manages with the corresponding file directory management service. In this way, the consistency between the file storage node service and its corresponding file directory management service can be improved, thereby improving the stability of the file directory management service and the corresponding file storage node service.

[0045] The flush operation can be used to write directory logs from the in-memory file directory management service to the server's file storage node service. This batch writes directory logs from in-memory to the server's file storage node service. This ensures content consistency between the file directory management service and the file storage node service, improving the reliability of report file management while ensuring stable memory and server performance.

[0046] Specifically, through the file directory management service, a disk flushing operation can be performed based on the current directory log, the current directory log can be solidified in the corresponding file storage node service, and the file storage node service can be fed back to the client. Exemplarily, a disk flushing operation can be performed based on the current directory log through the directory log manager of the file directory management service, and the current directory log can be solidified in the corresponding file storage node service.

[0047] S140 . Upload the current report file included in the report file directory creation request to the current storage node directory of the file storage node service through the client based on the current report file directory.

[0048] There is a mapping relationship between the current storage node directory and the current report file directory. The current report file directory is located in the file directory management service; the current storage node directory is located in the file storage node service. The current storage node directory can be the directory corresponding to the physical address of the current report file directory in the file storage node service. The current report file can be the report file uploaded by the report file uploader. For example, the current report file can be a small report file generated during the report template definition and report generation and display process.

[0049] Specifically, the report file uploader may upload the current report file included in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory through the client.

[0050] The technical solution of the embodiment of the present invention is to obtain a report file directory creation request through a client and send it to a file directory management service. The file directory management service constructs a current report file directory based on the current report file directory information contained in the report file directory creation request, generates a current directory log, performs a disk flushing operation on the current directory log, solidifies the current directory log into a corresponding file storage node service, and feeds back the file storage node service to the client. The client uploads the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory. The report file directory is managed by the file directory management service, and the report file is stored accordingly by the file storage node service. This realizes distributed management of report files, avoids the problem of functional redundancy of the report file management system, realizes lightweight management of distributed report files, and facilitates the operation and maintenance of the report file management system during actual use. For example, when a problem occurs with the data, the file directory management service and the file storage node service cooperate with each other to quickly locate the cause of the data occurrence, thereby improving the availability, consistency and security of the report files stored in the report file management system.

[0051] Example 2

[0052] Figure 2A flow chart of a distributed report file management method provided for the second embodiment of the present invention. Based on the above embodiments, the embodiment of the present invention concretizes the "file directory management service" into a "directory knowledge manager and directory log manager", and concretizes "through the file directory management service, performing a disk flushing operation on the current directory log, and solidifying the current directory log into the corresponding file storage node service" into "through the directory log manager, upon receiving the current directory log record request sent by the directory knowledge manager, obtaining the current directory log transaction number, and incrementing the current directory log transaction number, while creating a local transaction variable for the current thread, storing the incremented current directory log transaction number into the local transaction variable, and writing the incremented current directory log transaction number and the current directory log into the dual-handle log buffer; wherein, the dual-handle The hand log buffer is constructed by the directory log manager. After the double-handle log buffer is written, the directory log manager detects whether the memory buffer capacity of the double-handle log buffer has reached the write-to-disk threshold. When the double-handle log buffer detects that the memory buffer capacity of the double-handle log buffer has reached the write-to-disk threshold, the current thread is used to perform a disk flush operation on the self-incremented current directory log transaction number and current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and current directory log are solidified in the corresponding file storage node service. This achieves structural refinement of the file directory management service, takes into account the availability of the double-handle log buffer, and can also improve the performance stability of the file directory management service and the file storage node service. It should be noted that for parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0053] See also Figure 2 The distributed report file management method shown includes:

[0054] S210: Obtain a report file directory creation request through the client and send it to the file directory management service.

[0055] S220 . Build the current report file directory according to the current report file directory information included in the report file directory creation request through the file directory management service, and generate a current directory log.

[0056] Specifically, the Directory Knowledge Manager in the File Directory Management Service can construct the current report file directory based on the current report file directory information included in the report file directory creation request and generate a current directory log. The Directory Knowledge Manager calls the Directory Log Manager and sends a current directory log record request to the Directory Log Manager.

[0057] S230. Through the directory log manager, when receiving the current directory log record request sent by the directory knowledge manager, the current directory log transaction number is obtained, and the current directory log transaction number is incremented. At the same time, a local transaction variable of the current thread is created, and the incremented current directory log transaction number is stored in the local transaction variable, and the incremented current directory log transaction number and the current directory log are written into the dual-handle log buffer.

[0058] A file directory management service can include a directory knowledge manager and a directory log manager. Optionally, a file directory management service can include both a directory knowledge manager and a directory log manager. Optionally, before receiving a report file directory creation request from a client and sending it to the file directory management service, the directory knowledge manager and directory log manager can be created when the file directory management service starts.

[0059] The Directory Knowledge Manager can be used to provide directory management services for report file directories to clients and generate report file directory logs. For example, directory management services for report file directories include viewing, creating, and deleting report file directories. When creating a Directory Knowledge Manager, a mapping relationship between the report file directories in the Directory Knowledge Manager and the physical addresses of the File Storage Node Service can also be created. Furthermore, a mapping relationship between the report file directories in the Directory Knowledge Manager and the storage node directories of the File Storage Node Service can be created.

[0060] The directory log manager can be used to build a two-handle log buffer. Optionally, the directory log manager can be built at the same time as the directory knowledge manager is created. Optionally, the directory log manager can be used to read the directory log and load and replay the directory log in memory to generate a corresponding directory knowledge snapshot. Optionally, when creating the directory log manager, the directory log manager can build a two-handle log buffer. The directory log manager can read the historical directory log from the file storage node service corresponding to the file directory management service in which it resides. The directory log manager can load and replay the historical directory log in memory to build a historical directory knowledge snapshot corresponding to the historical directory log. The historical directory log includes the historical log record transaction number and the historical report file directory information corresponding to the historical log record transaction number. Loading and replaying is to perform adjustment operations on the historical report file directory corresponding to the historical log record transaction number in memory based on the historical directory log, and reproduce the historical report file directory results corresponding to the historical directory log. The historical directory knowledge snapshot can be the final historical report file directory results generated after the load and replay operation on the historical directory log. Since the file directory management service and the file storage node service are independent, the file directory management service is in memory and the file storage node service is in the hardware server. By loading and replaying the historical directory logs through the directory log manager corresponding to the file directory management service, the consistency between the file directory management service and the file storage node service can be ensured.

[0061] Optionally, while creating the directory knowledge manager, a junk service cleaner can be constructed and scheduled. The directory knowledge manager schedules the junk service cleaner, including: periodically scheduling the junk service cleaner through the directory knowledge manager. Obtain all directory knowledge nodes in the directory knowledge manager through the junk service cleaner, construct a deletion node container, and start the node scanner. Perform a hierarchical cross-scan of the directory knowledge nodes at each level in the directory knowledge manager through the node scanner of the directory knowledge manager. Detect whether there is a directory knowledge node to be deleted whose directory is empty and in the deletion state through the junk service cleaner. If there is a directory knowledge node to be deleted, insert the directory knowledge node to be deleted into the deletion node container through the junk service cleaner. After the directory knowledge nodes at each level in the directory knowledge manager are scanned, iteratively delete the node container through the junk service cleaner. Send the directory knowledge node to be deleted to the corresponding file storage node service and directory knowledge manager through the junk service cleaner. Delete the first directory to be deleted corresponding to the directory node to be deleted in the file storage node service through the file storage node service. The directory knowledge manager deletes the second directory to be deleted corresponding to the directory node to be deleted in the directory knowledge manager, generates a directory deletion log, and writes it into the directory log manager and the file storage node service.

[0062] The dual-handle log buffer is constructed by the directory log manager. The dual-handle log buffer can be used to perform a disk flushing operation on the current directory log, solidifying the current directory log to the corresponding file storage node service. The current directory log record request can be used to request the directory log manager to perform a disk flushing operation on the current directory log. After the directory knowledge manager generates the current directory log, the directory knowledge manager can send a current directory log record request containing the current directory log to the directory log manager. The current directory log transaction number can be used to identify the directory log that the directory log manager is processing. It can be understood that each time the directory log manager processes a directory log, the directory log record transaction number will be increased accordingly. The current directory log transaction number can be used to identify the processing progress of the directory log in the directory log manager.

[0063] The directory log manager can have multiple threads. A directory log request can be processed by a thread. The current thread can be the thread executing the current directory log request. A local transaction variable can be used to store the directory log transaction number of the directory log being processed. The current thread can be associated with the local transaction variable.

[0064] Specifically, through the directory log manager, when the management service log channel of the directory log manager receives the current directory log record request sent by the directory knowledge manager, the current directory log transaction number in the directory log manager can be obtained, and the current directory log transaction number can be incremented. At the same time, a local transaction variable of the current thread can be created, and the incremented current directory log transaction number can be stored in the local transaction variable, and the incremented current directory log transaction number and the current directory log can be written to the dual-handle log buffer. Optionally, through the directory log manager, when the current directory log record request sent by the directory knowledge manager is received, a management service log channel corresponding to the directory log manager can be established, and the current directory log can be sent to the corresponding management service log channel. Among them, the management service log channel can be used to realize the transmission of the current directory log from the directory knowledge manager to the directory log manager.

[0065] Optionally, when the directory log manager receives the current directory log record request sent by the directory knowledge manager, it can first detect whether the directory log manager has a disk flushing operation. When the directory log manager has a disk flushing operation in progress, the directory log manager waits for the disk flushing operation to be completed. When the directory log manager has no disk flushing operation in progress or when the directory log manager has completed the disk flushing operation, the directory log manager obtains the current directory log transaction number and increments the current directory log transaction number. At the same time, a local transaction variable of the current thread can be created, the incremented current directory log transaction number can be stored in the local transaction variable, and the incremented current directory log transaction number and the current directory log can be written to the dual-handle log buffer.

[0066] Exemplarily, when it is detected that there is a disk-flush status mark in the directory log manager, it is determined that there is a disk-flush operation in the directory log manager. When it is detected that there is no disk-flush status mark in the directory log manager, it is determined that there is no disk-flush operation in the directory log manager. When it is detected that there is a disk-flush status mark in the directory log manager, and after the disk-flush operation time has passed, it is detected that there is no disk-flush status mark in the directory log manager, it is determined that the disk-flush operation of the directory log manager is completed. Among them, the existence of a disk-flush operation in the log manager means that the directory log manager corresponding to the file directory management service is writing or performing a disk-flush operation, and it is necessary to wait for the disk-flush operation to end before executing the disk-flush operation of the current directory log.

[0067] S240: After the dual-handle log buffer is written, the directory log manager detects whether the memory buffer capacity of the dual-handle log buffer reaches the disk write threshold.

[0068] The memory buffer capacity of the double-handle log buffer can be used to represent the used capacity of the double-handle log buffer. The write disk threshold can be used to measure whether the available capacity of the double-handle log buffer is sufficient. Detecting whether the memory buffer capacity of the double-handle log buffer has reached the write disk threshold can be understood as determining whether the available capacity of the double-handle log buffer is sufficient. If the memory buffer capacity of the double-handle log buffer has reached the write disk threshold, it can be understood that the storage space of the double-handle log buffer is insufficient; if the memory buffer capacity of the double-handle log buffer has not reached the write disk threshold, it can be understood that the storage space of the double-handle log buffer is sufficient.

[0069] Specifically, through the directory log manager, after the dual-handle log buffer is written, that is, after the self-incremented current directory log transaction number and the current directory log have been written into the dual-handle log buffer, the memory buffer capacity of the dual-handle log buffer can be tested to determine whether the memory buffer capacity of the dual-handle log buffer reaches the write disk threshold.

[0070] S250. Through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread is used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and the current directory log are solidified to the corresponding file storage node service, and the file storage node service is fed back to the client.

[0071] When the dual-handle log buffer's memory buffer reaches the disk write threshold, a flush operation is performed on the auto-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable in the dual-handle log buffer, effectively freeing up storage space in the dual-handle log buffer. Feedback is sent to the file storage node service, which can be understood as a prompt to the client to upload the current report file to the file storage node service.

[0072] Specifically, through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread can be used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and the current directory log are solidified to the corresponding file storage node service, and the file storage node service is fed back to the client.

[0073] For example, through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread can be used to obtain the current directory log transaction number executed by the current thread from the local transaction variable, and obtain the synchronization practice number. The synchronization practice number and the current directory log transaction number can be compared. When the synchronization transaction number is less than the current directory log transaction number, the current thread can be used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log, and the self-incremented current directory log transaction number and the current directory log are solidified from the dual-handle log buffer to the corresponding file storage node service; when the synchronization transaction number is greater than or equal to the current directory log transaction number, the current thread does not need to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log.

[0074] The synchronization transaction number is used to identify the overall flushing progress of the file directory management service. This means that the synchronization transaction number identifies the directory log that the file directory management service is currently flushing. If the synchronization transaction number is greater than or equal to the current directory log transaction number, it means that another thread has already performed a flush on the directory log corresponding to the current directory log transaction number. If the synchronization transaction number is less than the current directory log transaction number, it means that the flush has not yet been performed on the directory log corresponding to the current directory log transaction number.

[0075] Optionally, after obtaining the current directory log transaction number executed by the current thread from the local transaction variable through the dual-handle log buffer using the current thread, it is possible to detect whether other threads in the dual-handle log buffer are performing disk flushing operations; when other threads are performing disk flushing operations, the current thread is controlled to wait for a preset write time; when no other threads are performing disk flushing operations, or after the current thread waits for a preset write time, the synchronization transaction number is obtained. Exemplarily, it is possible to determine whether other threads are performing disk flushing operations by detecting whether there are disk flushing status identifiers corresponding to other threads in the dual-handle log buffer. In this way, repeated disk flushing of the same current directory log by different threads can be avoided. The preset write time can be the upper limit of the write time of a single directory log. For example, the preset write time can be 1s.

[0076] Optionally, before using the current thread to perform a disk flushing operation on the auto-incremented current directory log transaction number and the current directory log, and before the auto-incremented current directory log transaction number and the current directory log are solidified from the dual-handle log buffer to the corresponding file storage node service, the current thread can be used to update the synchronization transaction number to the current directory log transaction number and set the dual-handle log buffer to a disk flushing state. Exemplarily, the disk flushing state flag in the dual-handle log buffer can be updated.

[0077] Optionally, after performing a flush operation on the current directory log transaction number and the current directory log after the auto-increment corresponding to the local transaction variable, the flush start transaction number and the flush end transaction number corresponding to the current directory log can be encapsulated into current transaction information and saved to the directory knowledge manager. Among them, the current transaction information can be used to determine the log record transaction number that has been flushed and the log record transaction number that has not been flushed. The flush start transaction number can be the directory log transaction number before the flush. The flush end transaction number can be the directory log transaction number after the flush is completed. After the new flush operation is completed, it can be determined whether the flush operation has been performed on a certain directory log based on the stored transaction information.

[0078] In an optional embodiment of the present invention, the dual-handle log buffer includes a working buffer and a synchronization buffer; accordingly, through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread is used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and the current directory log are solidified to the corresponding file storage node service, including: through the dual-handle log buffer, when receiving the current directory log, detecting whether the used storage capacity of the working buffer exceeds the preset buffer storage capacity; through the working buffer, when the used storage capacity of the working buffer exceeds the preset buffer storage capacity, forwarding the current directory log in the working buffer to the synchronization buffer; through the synchronization buffer, using the current thread, performing a disk flushing operation on the self-incremented current directory log transaction number and the current directory log, and solidifying the self-incremented current directory log transaction number and the current directory log to the corresponding file storage node service.

[0079] The working buffer can be used to receive directory logs. When performing a disk flushing operation, the working buffer can provide a buffering effect. The synchronization buffer can be used to perform a disk flushing operation. The working buffer and the synchronization buffer can be independent of each other and do not affect each other. The used storage capacity of the working buffer can be used to characterize the used capacity of the working buffer. The preset buffer storage capacity can be used to measure whether the available capacity of the working buffer is sufficient. Detecting whether the used storage capacity of the working buffer exceeds the preset buffer storage capacity can be understood as judging whether the available capacity of the working buffer is sufficient. Among them, if the used storage capacity of the working buffer exceeds the preset buffer storage capacity, it can be understood that the storage space of the working buffer is insufficient; if the used storage capacity of the working buffer does not exceed the preset buffer storage capacity, it can be understood that the storage space of the working buffer is sufficient.

[0080] This solution concretizes the dual-handle log buffer into a working buffer and a synchronization buffer, and adopts the working buffer to receive and store the current directory log. When the used storage capacity of the working buffer exceeds the preset buffer storage capacity, the current directory log in the working buffer is forwarded to the synchronization buffer. Through the synchronization buffer, the self-incremented current directory log transaction number and the current directory log are flushed, and the self-incremented current directory log transaction number and the current directory log are solidified into the corresponding file storage node service, realizing the independent implementation of the current directory log receiving and storage process and the flushing operation process, which can improve the stability of the flushing operation. Moreover, the available capacity of the working buffer is also taken into account to ensure the stability of the working buffer receiving and storage process.

[0081] S260 . Upload the current report file included in the report file directory creation request to the current storage node directory of the file storage node service through the client based on the current report file directory.

[0082] The technical solution of the embodiment of the present invention realizes the structural refinement of the file directory management service by concretizing the file directory management service into a directory knowledge manager and a directory log manager. Through the directory log manager, when receiving the current directory log record request sent by the directory knowledge manager, the current directory log transaction number is obtained and the current directory log transaction number is incremented. At the same time, a local transaction variable of the current thread is created, and the current directory log transaction number after the increment is stored in the local transaction variable, and the current directory log transaction number after the increment and the current directory log are written into the dual-handle log buffer. After the dual-handle log buffer is written, the memory buffer amount of the dual-handle log buffer is detected. When the write disk threshold is reached, through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer has reached the write disk threshold, the current thread is used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and the current directory log are solidified to the corresponding file storage node service. Compared with directly performing a disk flushing operation on a single current directory log, when the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the disk flushing operation is performed on the current directory log. This takes into account the availability of the dual-handle log buffer and can also improve the performance stability of the file directory management service and the file storage node service.

[0083] Example 3

[0084] Figure 3A flowchart of a distributed report file management method provided for the third embodiment of the present invention. Based on the above embodiments, the embodiment of the present invention further adds "periodically obtaining batch directory logs from the directory log manager of each file directory management service through the file directory calculation service; loading and replaying batch directory logs in memory through the file directory calculation service, updating the first complete directory knowledge snapshot to update the virtual file system, and storing the batch directory logs in the server corresponding to the file directory calculation service" after "uploading the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory through the client." The file directory calculation service is introduced to realize the aggregation of directory logs in the file directory management service and improve the integrity of the directory logs. It should be noted that for the parts not described in detail in the embodiment of the present invention, please refer to the description of other embodiments.

[0085] See also Figure 3 The distributed report file management method shown includes:

[0086] S310: Obtain a report file directory creation request through the client and send it to the file directory management service.

[0087] S320: Build the current report file directory according to the current report file directory information included in the report file directory creation request through the file directory management service, and generate a current directory log.

[0088] S330: Through the file directory management service, perform a disk flushing operation on the current directory log, solidify the current directory log into the corresponding file storage node service, and feed back the file storage node service to the client.

[0089] S340: Upload the report file included in the report file directory creation request to the current storage node directory of the file storage node service through the client based on the current report file directory.

[0090] S350: Periodically obtain batches of directory logs from the directory log managers of each file directory management service through the file directory calculation service.

[0091] There can be multiple file directory management services. Accordingly, there can also be multiple file storage node services. There is a one-to-one correspondence between file directory management services and file storage node services. The content stored in each file directory management service (or each file storage node service) may differ. To address the performance issues associated with the directory log manager, which must both receive and store directory logs and build directory knowledge snapshots corresponding to the directory logs, a file directory calculation service is established. Within this service, a memory-based virtual file system is built to aggregate the directories stored by each file directory management service and the files stored by each file storage node service. The file directory calculation service can be used to aggregate the directories stored by each file directory management service and each file storage node service. Batch directory logs can be obtained from the directory logs of each file directory management service for a single data acquisition cycle. Optionally, batch directory logs can be acquired using a 1-second time slice scheduling cycle (i.e., a single data acquisition cycle). Optionally, batch directory logs can be loaded and replayed based on the directory operation type corresponding to the batch directory logs. Directory operation types include creating a new report file directory, creating a new report folder, deleting a report folder, or renaming a report folder.

[0092] Optionally, batch directory logs can be periodically obtained from the directory log manager of each file directory management service through the computing service log channel between the file directory computing service and each file directory management service. Optionally, a computing service log channel can be established between the file directory computing service and the file directory management service through a single file directory management service, waiting for the file directory computing service to batch obtain log records from the file directory management service through the computing service log channel. The directory service log channel is used to connect a single file directory management service with the file directory computing service. The directory service log channel corresponds to the file directory management service. The directory service log channel can enable the corresponding file directory management service to transmit the current directory logs in batches to the file directory computing service.

[0093] Optionally, you can use the directory log grabber of the File Directory Computing Service to periodically retrieve batches of directory logs from the directory log managers of each file directory management service. Optionally, after the File Directory Computing Service is started, you can build a directory log grabber through the File Directory Computing Service.

[0094] S360. Load and replay the batch directory logs in memory through the file directory computing service, update the first complete directory knowledge snapshot to update the virtual file system, and store the batch directory logs in the server corresponding to the file directory computing service.

[0095] The first complete directory knowledge snapshot can be the directory result obtained by loading and replaying the batch directory log. The virtual file system can be the file management system corresponding to the in-memory file directory calculation service. The batch directory log is stored on the server corresponding to the file directory calculation service to facilitate querying and replaying the batch directory log.

[0096] Specifically, through the file directory computing service, batch directory logs can be loaded and replayed in memory, the directory knowledge snapshot of the virtual file system in the file directory computing service can be updated to obtain the first complete directory knowledge snapshot, and then the virtual file system is updated, and the batch directory logs are stored in the server corresponding to the file directory computing service.

[0097] The technical solution of the embodiment of the present invention introduces a file directory calculation service, realizes the aggregation of directory logs in each file directory management service, and realizes the management of report files corresponding to each file storage node service through the virtual file system in the file directory calculation service, further improving the availability, consistency, security and integrity of the report files stored in the report file management system.

[0098] In an optional embodiment of the present invention, it also includes: through the file directory computing service, when the file directory computing service is started, scanning and reading all directory logs in the server corresponding to the file directory computing service; through the file directory computing service, loading and replaying all directory logs in the memory, generating a second complete directory knowledge snapshot, and restoring the virtual file system.

[0099] All directory logs may be all directory logs stored in the server where the file directory calculation service is located. The second complete directory knowledge snapshot may be a directory result obtained by loading and replaying all directory logs.

[0100] Specifically, when the file directory computing service starts, it can scan and read all directory logs from the server corresponding to the file directory computing service. The file directory computing service loads and replays all directory logs in memory, generating a second complete directory knowledge snapshot and restoring the virtual file system. Optionally, when the file directory computing service starts, it can set the file directory computing service to a recovery state.

[0101] This solution achieves the recovery of the virtual file system in the file directory computing service by scanning and reading all directory logs in the server corresponding to the file directory computing service when the file directory computing service is started, loading and replaying all directory logs in the memory, and generating a second complete directory knowledge snapshot. This ensures the integrity of the virtual file system maintained in the file directory computing service. Moreover, when the file directory computing service fails, such as when the file directory management service suddenly loses power, the virtual file system in the file directory computing service can be locally recovered, thereby improving the fault tolerance of the virtual file system in the file directory computing service.

[0102] In an optional embodiment of the present invention, it also includes: through the file directory computing service, periodically obtaining a first complete directory knowledge snapshot from the virtual file system, and storing it in the server corresponding to the file directory computing service; accordingly, through the file directory computing service, when the file directory computing service is started, scanning and reading all directory logs in the server corresponding to the file directory computing service, including: through the file directory computing service, when the file directory computing service is started, scanning and reading all directory logs and the first complete directory knowledge snapshot in the server corresponding to the file directory computing service; through the file directory computing service, loading and replaying all directory logs in the memory, constructing a second complete directory knowledge snapshot, and restoring the virtual file system, including: through the file directory computing service, filtering the target directory log obtained after the first complete directory knowledge snapshot from all directory logs, and loading and replaying the target directory log to generate a third complete directory knowledge snapshot, and restoring the virtual file system.

[0103] The data acquisition period for the first complete directory knowledge snapshot can be the same as or different from the data acquisition period for batch directory logs. The target directory log can be a directory log whose data acquisition period is after the first complete directory knowledge snapshot among all directory logs. The third complete directory knowledge snapshot can be the directory result obtained by loading and replaying the target directory log based on the first complete directory knowledge snapshot.

[0104] Optionally, the first complete directory knowledge snapshot can be periodically obtained from the virtual file system through the computing service log channel between the file directory computing service and each file directory management service, and stored in the server corresponding to the file directory computing service.

[0105] Optionally, a directory knowledge snapshot acquisition subservice may be constructed through the file directory computing service. The directory knowledge snapshot acquisition subservice may periodically acquire a first complete directory knowledge snapshot from the virtual file system and store it in a server corresponding to the file directory computing service.

[0106] Specifically, when the file directory calculation service is started, all directory logs and the first complete directory knowledge snapshot in the server corresponding to the file directory calculation service can be scanned and read. Based on the data acquisition period of each directory log, target directory logs with a data acquisition period after the first complete directory knowledge snapshot are selected from all directory logs. Based on the first complete directory knowledge snapshot, the target directory log can be loaded and replayed to generate a third complete directory knowledge snapshot to restore the virtual file system.

[0107] This solution uses a file directory computing service to periodically obtain a first complete directory knowledge snapshot from the virtual file system and store it on the server corresponding to the file directory computing service, facilitating rapid query and acquisition of the first complete directory knowledge snapshot. Upon startup, the file directory computing service scans and reads all directory logs and the first complete directory knowledge snapshot from the server corresponding to the file directory computing service. This solution utilizes the periodically generated local log snapshots (i.e., the first complete directory knowledge snapshot) and the latest batch log files (i.e., all directory logs) stored on the server corresponding to the file directory computing service. The solution then filters the target directory logs obtained after the first complete directory knowledge snapshot from all directory logs and replays the target log files to generate the latest complete directory knowledge snapshot (i.e., the third complete directory knowledge snapshot), thereby restoring the virtual file system in memory and improving the recovery speed of the virtual file system in the file directory computing service. Furthermore, by filtering the target directory logs obtained after the first complete directory knowledge snapshot from all directory logs, the solution avoids the situation where the first complete directory knowledge snapshot is not the latest directory knowledge snapshot when the data acquisition period of the first complete directory knowledge snapshot differs from the data acquisition period of the batch directory logs, further improving the fault tolerance of virtual file system recovery in the file directory computing service.

[0108] In an optional embodiment of the present invention, it also includes: detecting whether each file directory management service is online through the role switching service; solidifying all log knowledge snapshots and all directory logs corresponding to the virtual file system in the memory to the server corresponding to the file directory computing service through the role switching service when it is detected that each file directory management service is not online; creating a directory knowledge manager and a directory log manager corresponding to the file directory computing service in the memory through the file directory computing service; initializing the directory knowledge manager and the directory log manager corresponding to the file directory computing service using all log knowledge snapshots and all directory logs corresponding to the virtual file system through the file directory computing service; adjusting the target address captured by the file directory computing service to the directory log manager corresponding to the file directory computing service through the file directory computing service after the directory knowledge manager and the directory log manager corresponding to the file directory computing service are initialized, and upgrading the file directory computing service role to dual-role operation.

[0109] The role switching service can be used to implement role switching between the file directory management service and the file directory calculation service. Detecting whether each file directory management service is online can be understood as detecting whether each file directory management service is available. If a file directory management service is online, it can be understood as being available; if a file directory management service is offline, it can be understood as being unavailable. The full log knowledge snapshot can be the latest log knowledge snapshot in the file directory calculation service. Upgrading the file directory calculation service role to dual-role operation can be understood as the file directory calculation service providing both directory calculation services and directory management services. At this point, the file directory calculation service can become an alternative file directory management service for each file directory management service.

[0110] Specifically, the role switching service can be used to detect whether each file directory management service is online. When it is detected that each file directory management service is online, the file directory management service can implement its own functions. When it is detected that each file directory management service is not online, all log knowledge snapshots and all directory logs corresponding to the virtual file system in the memory can be solidified to the server corresponding to the file directory computing service. Through the file directory computing service, the directory knowledge manager and directory log manager corresponding to the file directory computing service are constructed in the memory. All log knowledge snapshots and all directory logs corresponding to the virtual file system can be stored in the directory knowledge manager and directory log manager corresponding to the file directory computing service. After the directory knowledge manager and directory log manager corresponding to the file directory computing service are initialized, the target address captured by the file directory computing service can be adjusted to the directory log manager corresponding to the file directory computing service, and the file directory computing service role can be upgraded to dual-role operation.

[0111] This solution introduces a role switching service to detect whether each file directory management service is online. When each file directory management service is not online, the file directory calculation service implements the functions of each file directory management service. The file directory calculation service is used as an alternative file directory management service for each file directory management service, realizing the dual-role operation of the file directory calculation service role. When each file directory management service is not online, the normal operation of the report file management system is guaranteed, further improving the fault tolerance of the report file management system.

[0112] Example 4

[0113] Figure 4 This is a schematic diagram of the structure of a distributed report file management system provided in Example 4 of the present invention. This embodiment of the present invention is applicable to situations where distributed management of report files is required. The system can execute a distributed report file management method. The system can be implemented in hardware and / or software and can be configured in an electronic device that carries the distributed report file management function.

[0114] See also Figure 4 The distributed report file management system shown in the figure includes: a client 410, a file directory management service 420 and the file storage node service 430; wherein the client 410 is used to obtain a report file directory creation request and send it to the file directory management service; the file directory management service 420 is used to construct a current report file directory and generate a current directory log based on the current report file directory information contained in the report file directory creation request; the file directory management service 420 is used to perform a disk flushing operation on the current directory log, solidify the current directory log to the corresponding file storage node service, and feedback the file storage node service to the client; the client 410 is used to upload the report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory.

[0115] The technical solution of the embodiment of the present invention is to obtain a report file directory creation request through a client and send it to a file directory management service. The file directory management service constructs a current report file directory based on the current report file directory information contained in the report file directory creation request, generates a current directory log, performs a disk flushing operation on the current directory log, solidifies the current directory log into a corresponding file storage node service, and feeds back the file storage node service to the client. The client uploads the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory. The report file directory is managed by the file directory management service, and the report file is stored accordingly by the file storage node service. This realizes distributed management of report files, avoids the problem of functional redundancy of the report file management system, realizes lightweight management of distributed report files, and facilitates the operation and maintenance of the report file management system during actual use. For example, when a problem occurs with the data, the file directory management service and the file storage node service cooperate with each other to quickly locate the cause of the data occurrence, thereby improving the availability, consistency and security of the report files stored in the report file management system.

[0116] In an optional embodiment of the present invention, the file directory management service 420 includes a directory knowledge manager and a directory log manager; accordingly, the file directory management service 420 includes: the directory log manager is used to obtain the current directory log transaction number when receiving the current directory log record request sent by the directory knowledge manager, and to increment the current directory log transaction number, while creating a local transaction variable of the current thread, storing the incremented current directory log transaction number in the local transaction variable, and writing the incremented current directory log transaction number and the current directory log into the dual-handle log buffer ; Wherein, the dual-handle log buffer is constructed by the directory log manager; the directory log manager is used to detect whether the memory buffer amount of the dual-handle log buffer reaches the write disk threshold after the dual-handle log buffer is written; the dual-handle log buffer is used to use the current thread to perform a disk flush operation on the current directory log transaction number and the current directory log after the self-increment corresponding to the local transaction variable when it detects that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, and solidify the current directory log transaction number and the current directory log after the self-increment to the corresponding file storage node service.

[0117] In an optional embodiment of the present invention, the dual-handle log buffer includes a working buffer and a synchronization buffer; accordingly, the dual-handle log buffer includes: the dual-handle log buffer, which is used to detect whether the used storage capacity of the working buffer exceeds the preset buffer storage capacity when receiving the current directory log; the working buffer, which is used to forward the current directory log in the working buffer to the synchronization buffer when the used storage capacity of the working buffer exceeds the preset buffer storage capacity; the synchronization buffer, which is used to use the current thread to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log, and solidify the self-incremented current directory log transaction number and the current directory log to the corresponding file storage node service.

[0118] In an optional embodiment of the present invention, the system further includes: a file directory calculation service, which is used to periodically obtain batch directory logs from the directory log managers of each of the file directory management services; the file directory calculation service is used to load and replay the batch directory logs in memory, update the first complete directory knowledge snapshot to update the virtual file system, and store the batch directory logs in the server corresponding to the file directory calculation service.

[0119] In an optional embodiment of the present invention, the system also includes: the file directory computing service, which is used to scan and read all directory logs in the server corresponding to the file directory computing service when the file directory computing service is started; the file directory computing service is used to load and replay all directory logs in the memory, generate a second complete directory knowledge snapshot, and restore the virtual file system.

[0120] In an optional embodiment of the present invention, the system also includes: the file directory computing service, which is used to periodically obtain the first complete directory knowledge snapshot from the virtual file system and store it in the server corresponding to the file directory computing service; accordingly, the file directory computing service includes: the file directory computing service, which is used to scan and read all directory logs and the first complete directory knowledge snapshot in the server corresponding to the file directory computing service when the file directory computing service is started; the file directory computing service is used to filter the target directory log obtained after the first complete directory knowledge snapshot from all directory logs, and load and replay the target directory log to generate a third complete directory knowledge snapshot to restore the virtual file system.

[0121] In an optional embodiment of the present invention, the system also includes: a role switching service, which is used to detect whether each of the file directory management services is online; the role switching service, which is used to solidify all log knowledge snapshots and all directory logs corresponding to the virtual file system in the memory to the server corresponding to the file directory computing service when it is detected that each of the file directory management services is not online; the file directory computing service, which is used to create a directory knowledge manager and a directory log manager corresponding to the file directory computing service in the memory; the file directory computing service, which is used to initialize the directory knowledge manager and the directory log manager corresponding to the file directory computing service using all log knowledge snapshots and all directory logs corresponding to the virtual file system; the file directory computing service, which is used to adjust the target address captured by the file directory computing service to the directory log manager corresponding to the file directory computing service after the directory knowledge manager and the directory log manager corresponding to the file directory computing service are initialized, and upgrade the file directory computing service role to dual-role operation.

[0122] The distributed report file management system provided by the embodiment of the present invention can execute the distributed report file management method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] In the technical solution of the embodiment of the present invention, the acquisition, storage and application of the report file directory creation request, current directory log transaction number, batch directory log, first complete directory knowledge snapshot and target directory log, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] Example 5

[0125] Figure 5 A schematic diagram of an electronic device 500 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile systems, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing systems. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0126] like Figure 5As shown, electronic device 500 includes at least one processor 501 and memory, such as read-only memory (ROM) 502 and random access memory (RAM) 503, communicatively connected to the at least one processor 501. The memory stores computer programs executable by the at least one processor. Processor 501 can perform various appropriate actions and processes based on the computer programs stored in ROM 502 or loaded from storage unit 508 into RAM 503. RAM 503 can also store various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0127] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0128] Processor 501 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of processor 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, or microcontroller. Processor 501 executes the various methods and processes described above, such as the distributed report file management method.

[0129] In some embodiments, the distributed report file management method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by processor 501, one or more steps of the distributed report file management method described above can be performed. Alternatively, in other embodiments, processor 501 can be configured to execute the distributed report file management method via any other suitable means (e.g., via firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input system, and at least one output system, and transmit data and instructions to the storage system, the at least one input system, and the at least one output system.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing system so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, system, or device. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display system (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing system (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of systems can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include clients and servers. The clients and servers are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS (Virtual Private Server) services.

[0136] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0137] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A distributed report file management method, characterized in that: The method comprises: Obtain the report file directory creation request through the client and send it to the file directory management service; By means of the file directory management service, a current report file directory is constructed according to the current report file directory information included in the report file directory creation request, and a current directory log is generated; Through the file directory management service, a disk flushing operation is performed on the current directory log, the current directory log is solidified to the corresponding file storage node service, and the file storage node service is fed back to the client; Uploading, by the client, the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory; Periodically obtaining batches of directory logs from the directory log managers of the file directory management services through the file directory calculation service; By means of the file directory computing service, the batch directory logs are loaded and replayed in memory, the first complete directory knowledge snapshot is updated to update the virtual file system, and the batch directory logs are stored in a server corresponding to the file directory computing service; Detecting whether each of the file directory management services is online through the role switching service; When it is detected that each of the file directory management services is offline, all log knowledge snapshots and all directory logs corresponding to the virtual file system in the memory are solidified to the server corresponding to the file directory calculation service through the role switching service; By using the file directory computing service, a directory knowledge manager and a directory log manager corresponding to the file directory computing service are created in memory; Initialize the directory knowledge manager and directory log manager corresponding to the file directory computing service by using all log knowledge snapshots and all directory logs corresponding to the virtual file system through the file directory computing service; Through the file directory computing service, after the directory knowledge manager and directory log manager corresponding to the file directory computing service are initialized, the target address captured by the file directory computing service is adjusted to the directory log manager corresponding to the file directory computing service, and the file directory computing service role is upgraded to dual-role operation.

2. The distributed report file management method according to claim 1, characterized in that: The file directory management service includes a directory knowledge manager and a directory log manager; accordingly, the file directory management service performs a disk flushing operation on the current directory log and solidifies the current directory log into the corresponding file storage node service, including: Upon receiving a current directory log record request sent by the directory knowledge manager, the directory log manager obtains a current directory log transaction number, increments the current directory log transaction number, creates a local transaction variable of the current thread, stores the incremented current directory log transaction number in the local transaction variable, and writes the incremented current directory log transaction number and the current directory log into a dual-handle log buffer; wherein the dual-handle log buffer is constructed by the directory log manager; After the dual-handle log buffer is written, detecting, by the directory log manager, whether the memory buffer amount of the dual-handle log buffer reaches a write-to-disk threshold; Through the dual-handle log buffer, when it is detected that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread is used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log corresponding to the local transaction variable, and the self-incremented current directory log transaction number and the current directory log are solidified to the corresponding file storage node service.

3. The distributed report file management method according to claim 2, characterized in that: The dual-handle log buffer includes a working buffer and a synchronization buffer; accordingly, when the dual-handle log buffer detects that the memory buffer amount of the dual-handle log buffer reaches the write disk threshold, the current thread is used to perform a disk flushing operation on the current directory log transaction number and the current directory log after the auto-increment corresponding to the local transaction variable, and the auto-incremented current directory log transaction number and the current directory log are solidified in the corresponding file storage node service, including: By means of the dual-handle log buffer, upon receiving the current directory log, detecting whether the used storage capacity of the working buffer exceeds a preset buffer storage capacity; forwarding the current directory log in the working buffer to the synchronization buffer via the working buffer when the used storage capacity of the working buffer exceeds a preset buffer storage capacity; Through the synchronization buffer, the current thread is used to perform a disk flushing operation on the self-incremented current directory log transaction number and the current directory log, and the self-incremented current directory log transaction number and the current directory log are solidified into the corresponding file storage node service.

4. The distributed report file management method according to claim 1, characterized in that: Also includes: By means of the file directory calculation service, when the file directory calculation service is started, all directory logs in the server corresponding to the file directory calculation service are scanned and read; Through the file directory calculation service, all the directory logs are loaded and replayed in the memory to generate a second complete directory knowledge snapshot and restore the virtual file system.

5. The distributed report file management method according to claim 1, characterized in that: Also includes: Periodically obtaining the first complete directory knowledge snapshot from the virtual file system through the file directory calculation service, and storing the snapshot in a server corresponding to the file directory calculation service; Accordingly, the file directory calculation service scans and reads all directory logs in the server corresponding to the file directory calculation service when the file directory calculation service is started, including: By means of the file directory calculation service, when the file directory calculation service is started, all directory logs and the first complete directory knowledge snapshot in the server corresponding to the file directory calculation service are scanned and read; The method of loading and replaying all directory logs in memory through the file directory computing service, constructing a second complete directory knowledge snapshot, and restoring the virtual file system includes: The file directory computing service is used to filter the target directory log obtained after the first complete directory knowledge snapshot from all directory logs, and the target directory log is loaded and replayed to generate a third complete directory knowledge snapshot and restore the virtual file system.

6. A distributed report file management system, characterized in that: The system includes: a client, a file directory management service, a file storage node service, a file directory calculation service and a role switching service; wherein, The client is used to obtain a report file directory creation request and send it to the file directory management service; The file directory management service is used to build the current report file directory according to the current report file directory information included in the report file directory creation request, and generate a current directory log; The file directory management service is used to perform a disk flushing operation on the current directory log, solidify the current directory log to the corresponding file storage node service, and feed back the file storage node service to the client; The client is configured to upload the current report file contained in the report file directory creation request to the current storage node directory of the file storage node service based on the current report file directory; The file directory calculation service is used to periodically obtain batch directory logs from the directory log managers of each file directory management service; The file directory computing service is configured to load and replay the batch directory logs in a memory, update the first complete directory knowledge snapshot to update the virtual file system, and store the batch directory logs in a server corresponding to the file directory computing service; The role switching service is used to detect whether each of the file directory management services is online; The role switching service is used to solidify all log knowledge snapshots and all directory logs corresponding to the virtual file system in the memory to the server corresponding to the file directory computing service when it is detected that each of the file directory management services is offline; The file directory calculation service is used to create a directory knowledge manager and a directory log manager corresponding to the file directory calculation service in memory; The file directory calculation service is used to initialize the directory knowledge manager and directory log manager corresponding to the file directory calculation service using all log knowledge snapshots and all directory logs corresponding to the virtual file system; The file directory computing service is used to adjust the target address captured by the file directory computing service to the directory log manager corresponding to the file directory computing service after the directory knowledge manager and directory log manager corresponding to the file directory computing service are initialized, and upgrade the file directory computing service role to dual-role operation.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed report file management method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed report file management method according to any one of claims 1 to 5 when executed.

Images