Document processing methods, devices, equipment, media and products

By introducing a dual-version data block management method with operation version identifier and upload version identifier in the cloud storage system, the client can perform multiple file operations under the operation version identifier returned by the server, reducing the interaction frequency, improving file operation efficiency, and supporting efficient management by the metadata management server.

CN123115074APending Publication Date: 2026-11-17TENCENT TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510622038.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In cloud storage systems, frequent interactions between clients and servers lead to inefficient file operations.

Method used

By adopting a dual-version data block management method with operation version identifier and upload version identifier, after receiving the operation version identifier returned by the server, the client can perform multiple file operations under that identifier and generate an upload version identifier to send to the server to update metadata.

Benefits of technology

It reduces the frequency of interaction between the client and the server, improves the efficiency of file operations, and supports the metadata management server for efficient file management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115074A_ABST
    Figure CN123115074A_ABST
Patent Text Reader

Abstract

This application provides a file processing method, apparatus, device, medium, and product. The method includes: sending a file operation request for a first file to a metadata management server; receiving a first operation version identifier of the first file; performing file operation processing on the first file in a cloud storage system based on the first operation version identifier; generating a first upload version identifier for a first data block in the first file during the execution of the file operation processing; and sending the first operation version identifier and the first upload version identifier as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information. This application achieves efficient file processing operations through a dual-version data block management method using operation version identifiers and upload version identifiers, thereby improving file operation efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document processing method, apparatus, device, medium and product, specifically to a document processing method, a document processing apparatus, a computer device, a computer-readable storage medium and a computer program product. Background Technology

[0002] A cloud-based big data storage system (or cloud storage system for short) is a system built on cloud storage that provides storage services for big data computing services. This cloud storage system stores data in the form of files, and the data in the files is usually organized in the form of data blocks.

[0003] Currently, every time a client writes or uploads a data block to a cloud storage system, it needs to interact with the server (such as a metadata management server). This frequent interaction leads to low efficiency in file operations. Summary of the Invention

[0004] This application provides a file processing method, apparatus, device, medium, and product that can achieve efficient file processing operations by using a dual-version data block management method that includes operation version identifiers and upload version identifiers, thereby improving file operation efficiency.

[0005] On one hand, embodiments of this application provide a file processing method, the method comprising:

[0006] Send a file operation request for the first file to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0007] Receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request;

[0008] Based on the first operation version identifier, file operation processing is performed on the first file in the cloud storage system; the file operation processing is used to indicate that at least one data processing operation is performed on the data blocks of the first file;

[0009] During the execution of file operation processing, a first upload version identifier is generated for the first data block in the first file. The first data block refers to the data block in the first file that changes with any data processing operation.

[0010] The first operation version identifier and the first upload version identifier are sent as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

[0011] On one hand, embodiments of this application provide a file processing method, the method comprising:

[0012] The system receives file operation requests for the first file from the client. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0013] In response to a file operation request, send the first operation version identifier of the first file to the client;

[0014] The system receives description information of the first data block sent by the client. The description information includes a first operation version identifier and a first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file.

[0015] The metadata of the first file is updated based on the description information of the first data block.

[0016] On one hand, embodiments of this application provide a document processing apparatus, which includes:

[0017] The sending unit is used to send a file operation request for the first file to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0018] The receiving unit is used to receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request;

[0019] The processing unit is configured to perform file operation processing on the first file in the cloud storage system based on the first operation version identifier; the file operation processing is used to instruct at least one data processing operation to be performed on the data blocks of the first file.

[0020] The processing unit is also used to generate a first upload version identifier for the first data block in the first file during the execution of file operation processing. The first data block refers to the data block in the first file that changes with any data processing operation.

[0021] The sending unit is also used to send the first operation version identifier and the first upload version identifier as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

[0022] In one possible implementation, the first file includes variable data blocks that can be written to; the data processing operation includes a data writing operation to the first file; during the execution of the file operation processing, the processing unit 1003 generates a first upload version identifier for the first data block in the first file, for use in performing the following operations:

[0023] The variable data block from the first file is loaded into the client's memory from the cloud storage system; the variable data block has an initial upload version identifier i, where i is an integer;

[0024] Obtain the first data corresponding to the data write operation, and write the first data into the variable data block in memory to generate the first data block;

[0025] Based on the initial upload version identifier i of the variable data block, generate the first upload version identifier i+1 for the first data block;

[0026] Specifically, when the cache interface of the cloud storage system is detected to be called, the first data block in memory is persisted to the cloud storage system.

[0027] In one possible implementation, the processing unit is also used to perform the following operations:

[0028] The first data block is loaded from the cloud storage system into the client's memory, and the second data to be written corresponding to the data write operation is obtained.

[0029] Write the second data into the first data block in memory to generate the second data block, and generate the second upload version identifier i+2 for the second data block based on the first upload version identifier i+1 of the first data block;

[0030] Until the length of the variable data block after the data write operation is performed reaches the preset length, a fixed data block of the preset length is submitted to the cloud storage system.

[0031] The variable data block after deleting the preset length is used as the new variable data block in the first file, and a new upload version identifier is set for the new variable data block.

[0032] In one possible implementation, the data processing operation further includes a data deletion operation on the first file; the processing unit performs file operation processing on the first file in the cloud storage system, and is also used to perform the following operations:

[0033] The second data block is loaded from the cloud storage system into the client's memory; the second data block has a second upload version identifier i+2;

[0034] Retrieve the data to be deleted corresponding to the data deletion operation, delete the data to be deleted from the second data block in memory, and generate the third data block;

[0035] Based on the second upload version identifier i+2 of the second data block, a third upload version identifier i+3 is generated for the third data block; wherein, the first data block, the second data block, and the third data block all have a first operation version identifier.

[0036] In one possible implementation, any data block in the first file is stored as an object in the cloud storage system, with each object corresponding to an object identifier; the file operation processing is also used to indicate data read operations on the first file; the processing unit is also used to perform the following operations:

[0037] Obtain the metadata of the first file sent by the metadata management server. The metadata of the first file records the description information of each data block in the first file. The description information of any data block includes at least one of the following: operation version identifier, upload version identifier, block sequence number, and length.

[0038] Parse the metadata of the first file to obtain the target description information of the data block to be read corresponding to the data read operation;

[0039] A target object identifier is generated based on the target description information, and the target data block indicated by the target object identifier is read from the first file in the cloud storage system based on the target object identifier.

[0040] In one possible implementation, the first operation version identifier of the first file is represented as j, where j is an integer; the processing unit is also used to perform the following operations:

[0041] In response to a call to the file closing interface of the cloud storage system, the first data block of the first file is persisted to the cloud storage system;

[0042] Send another file operation request for the first file to the metadata management server, and receive the second operation version identifier of the first file returned by the metadata management server; wherein, the second operation version identifier is j+1.

[0043] On one hand, embodiments of this application provide a document processing apparatus, which includes:

[0044] The receiving unit is used to receive file operation requests for the first file sent by the client. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0045] The sending unit is used to send the first operation version identifier of the first file to the client in response to a file operation request;

[0046] The receiving unit is also used to receive description information of the first data block sent by the client. The description information includes a first operation version identifier and a first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file.

[0047] The processing unit is used to update the metadata of the first file based on the description information of the first data block.

[0048] In one possible implementation, the processing unit updates the metadata of the first file based on the description information, for the following operations:

[0049] Obtain the metadata of the first file, and construct a first data storage structure for the first file based on the metadata of the first file;

[0050] The first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure after the first file is updated.

[0051] In one possible implementation, the first file includes N fixed data blocks and one variable data block. The data length of each fixed data block is equal to a preset length, and the data length of the variable data block is less than the preset length. N is a positive integer. The metadata of the first file records the description information of each data block in the first file. The processing unit constructs a first data storage structure for the first file based on the metadata of the first file, and performs the following operations:

[0052] Obtain the first description information of each fixed data block from the metadata of the first file. The first description information includes: block number and operation version identifier.

[0053] Based on the first description information of N fixed data blocks, K interval trees are constructed for the N fixed data blocks, and the K interval trees are stored in the first storage area in order; wherein, each interval tree is used to record at least one fixed data block with the same operation version identifier, and K is a positive integer;

[0054] The second description information of the variable data block is obtained from the metadata of the first file and stored in the second storage area; the second description information includes at least one of the following: initial operation version identifier, initial upload version identifier, and initial length;

[0055] The first storage area and the second storage area are combined to form the first data storage structure of the first file.

[0056] In one possible implementation, the sending unit sends a first operation version identifier of the first file to the client for performing the following operations:

[0057] If the file operation request is used to indicate that a first file is to be created, then the first operation version identifier is set to the initial value;

[0058] If the file operation request is used to indicate that the first file is to be processed by file operation, then the initial operation version identifier of the variable data block is obtained from the first data storage structure of the first file;

[0059] A first operation version identifier is generated based on the initial operation version identifier, and the first operation version identifier is returned to the client.

[0060] In one possible implementation, the processing unit is also used to perform the following operations:

[0061] Obtain the target client identifier for sending the file operation request, and verify the client based on the target client identifier; wherein the verification includes at least one of the following: permission verification, security verification, and legality verification;

[0062] If the client verification passes, the step of retrieving the initial operation version identifier of the variable data block from the first data storage structure of the first file is triggered.

[0063] In one possible implementation, the processing unit updates the first data storage structure based on the description information of the first data block to obtain the updated second data storage structure for the first file, which is then used to perform the following operations:

[0064] Parse the description information of the first data block to obtain the target length corresponding to the first data block;

[0065] If the target length is less than the preset length, the second storage area in the first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure of the first file.

[0066] In one possible implementation, the processing unit is also used to perform the following operations:

[0067] If the target length is equal to the preset length, then the first data block is treated as a fixed data block, and the first operation version identifier of the first data block is added to the first storage area of ​​the first data storage structure to obtain the second data storage structure of the first file; or,

[0068] If the target length is greater than the preset length, the first data block is divided into at least one fixed data block and a new variable data block; the first operation version identifier of each fixed data block is added to the first storage area of ​​the first data storage structure, and the second storage area in the first data storage structure is updated based on the description information of the new variable data block to obtain the second data storage structure of the first file.

[0069] On one hand, embodiments of this application provide a computer device, which includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it performs the above-described file processing method.

[0070] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the aforementioned file processing method.

[0071] On the one hand, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it performs the above-described file processing method.

[0072] In this embodiment, a file operation request for a first file is sent to a metadata management server. The first file is stored in a cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system. A first operation version identifier of the first file is received. The first operation version identifier of the first file is returned by the metadata management server in response to the file operation request. Based on the first operation version identifier, file operation processing is performed on the first file in the cloud storage system. This file operation processing is used to indicate that at least one data processing operation is performed on the data blocks of the first file. It can be seen that when the client requests to perform file operation processing on any file (such as the first file) in the cloud storage system, multiple file operation processing can be performed on the first file based on the first operation version identifier returned by the metadata management server, without having to interact with the metadata management server every time file operation processing is performed. During file operation processing, a first upload version identifier is generated for the first data block in the first file. The first data block refers to the data block in the first file that changes with any data processing operation. The first operation version identifier and the first upload version identifier are sent as description information of the first data block to the metadata management server, enabling the metadata management server to update the metadata in the first file based on the description information. Therefore, each time a data processing operation is performed on a file, the client can generate an upload version identifier corresponding to the changed data block, and then send the first operation version identifier and the first upload version identifier as description information of the changed first data block to the metadata management server. In summary, when a client requests to perform file operations, the server returns an operation version identifier to the client, which allows the client to perform multiple data processing operations on the file under that operation version identifier, reducing frequent interactions with the server and improving file operation efficiency. Furthermore, by designing a dual-version data block with operation version identifier and upload version identifier, the metadata management server can efficiently manage files. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a schematic diagram of the architecture of a file processing system provided in an embodiment of this application;

[0075] Figure 2 This is a schematic diagram of the module structure of a cloud storage system provided in an embodiment of this application;

[0076] Figure 3 This is a flowchart illustrating a document processing method provided in an embodiment of this application;

[0077] Figure 4 This is a schematic diagram illustrating a process of a client writing data to a cloud storage system, provided in an embodiment of this application.

[0078] Figure 5 This is a schematic diagram of the data block structure of a first file provided in an embodiment of this application;

[0079] Figure 6 This is a flowchart illustrating another file processing method provided in an embodiment of this application;

[0080] Figure 7 This is a schematic diagram of a first data storage structure for a first file provided in an embodiment of this application;

[0081] Figure 8a This is a schematic diagram of a process for updating a first data storage structure provided in an embodiment of this application;

[0082] Figure 8b This is a schematic diagram of another process for updating the first data storage structure provided in an embodiment of this application;

[0083] Figure 8c This is a schematic diagram of another process for updating the first data storage structure provided in an embodiment of this application;

[0084] Figure 9 This is a schematic diagram of the interactive flow of a file processing method provided in an embodiment of this application;

[0085] Figure 10 This is a schematic diagram of the structure of a document processing device provided in an embodiment of this application;

[0086] Figure 11 This is a schematic diagram of another document processing device provided in an embodiment of this application;

[0087] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0088] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0089] I. The following is an introduction to the key terms used in this application.

[0090] (1) Cloud storage system.

[0091] A cloud storage system, also known as a cloud-based big data storage system, is a system that provides big data storage services based on cloud storage technology. Big data storage services are generally organized in the form of a file system, which mainly contains a directory structure and files; that is, it is used to store large numbers of files in a cloud storage system. The cloud storage system involved in this application can be a Cloud Object Storage (COS) system. A COS system is a cloud computing-based storage solution specifically designed for storing massive amounts of unstructured data (such as images, videos, documents, logs, etc.). It achieves high scalability, high reliability, and low-cost data storage and management through a flat storage architecture and distributed technology. The core concepts involved in the COS system include:

[0092] 1. Object: The basic unit of data storage, containing the data itself and a globally unique identifier (such as an object identifier data block key). It supports data storage of any type of file (such as text, image, video) and has no format restrictions. This application supports storing data blocks from files into the COS system as objects.

[0093] 2. Metadata: User-defined or system-generated descriptive information (such as file type, creation time, and permission policies), supporting quick retrieval and management of objects; in this application, a metadata management server is used to manage the metadata of each file in the cloud storage system.

[0094] 3. Bucket: A container for storing objects, similar to a folder, but with a flat structure (no directory hierarchy); and each bucket must be set with a corresponding geographic attribute (such as city A, city B), which affects access latency and data compliance.

[0095] (2) First file, data block.

[0096] The first file refers to any file stored in the cloud storage system, including but not limited to: videos, images, documents, logs, and other data of any type. Specifically, files are often divided into multiple data blocks stored in the cloud storage system. The methods for dividing data blocks mainly involve: fixed-length division, random division, and variable-length division. This application mainly relates to using a fixed-length division method to divide a file into multiple fixed-length data blocks of equal length.

[0097] A data block (or block) refers to a basic data unit that constitutes the first file. In this application, the data blocks of the first file mainly involve two types: immutable blocks and mutable blocks (or unsealed blocks). An immutable block is a fixed data block with a preset data length (e.g., 4MB), which does not support appending or writing data. A mutable block is a data block with a data length less than the preset length (<4MB), which supports appending or writing data. If the data length of the mutable block after data writing reaches the preset length, then the mutable block after data writing can also become an immutable block. It should be understood that a file can include multiple immutable blocks and one mutable block. When writing data to a file, data is written to the mutable block in that file to generate an immutable block for that file.

[0098] (3) First operation version identifier, first upload version identifier.

[0099] The first open version identifier (open version) is a identifier returned by the metadata management server to a client when the client requests file operations on any file (such as the first file). This identifier indicates the number of times the first file has been opened. For example, if the first file is opened for the first time (i.e., when it is first created), the open version is 1. If the first file is opened a second time, the open version is 2, and so on. Each time the first file is reopened, the open version increments by 1. One or more clients can have file operation permissions to open the first file, but the first file cannot be opened by multiple clients simultaneously. For example, if client 1 opens the first file and it is not closed, it cannot be opened by other clients (such as client 2). Furthermore, after opening the first file, any client can perform file operations on it based on the open version returned by the metadata management server, such as data writing, data deletion, and data reading. Each data block in the first file has an open version.

[0100] The first upload version identifier (put version) refers to a data identifier generated by the client for a data block (a data block that changes with the current data processing operation, such as the first data block) in the first file opened by the client during the data processing operation (such as data writing operation or data deletion operation). In this application, the data block to which the data processing operation and data deletion operation are performed often refers to the aforementioned mutable block (or unsealed block, variable data block). That is, a mutable block often has a put version, and during the continuous writing of data into the mutable block, the put version of the new mutable block is incremented by 1. For example, the put version of mutable block1 in the current first file is 1. If a data writing operation is performed on mutable block1, the put version of mutable block2 is 2. Further, after another data writing operation is performed on mutable block2, the put version of mutable block3 is 3, and so on, until the mutable blocks in the file are filled (reaching the preset length of the immutable block), at which point the put version of the mutable block is reinitialized. In other words, put version is a data identifier designed for blocks that are not full (i.e., data blocks that have not reached the preset length).

[0101] (4) Metadata of the first file.

[0102] The metadata of the first file records descriptive information about each data block in the first file. Specifically, the data blocks in the first file involve two types: immutable blocks and mutable blocks. The descriptive information corresponding to different types of data blocks may be the same or different. For example, the descriptive information of an immutable block includes: block number, length (i.e., the preset length, such as 4MB), and open version; while the descriptive information of a mutable block includes: length (less than the preset length, such as 1MB or 2MB), open version, and put version.

[0103] II. Introduction to the proposed scheme.

[0104] This application provides a file processing solution applicable to file processing scenarios involving any type of cloud storage system. Specifically, this solution supports the allocation of an open version by the metadata management server to the client each time the client opens a file. Subsequently, the client can upload (or delete) any number of data blocks, and distinguish different data blocks through a put version maintained by the client itself, eliminating the need for frequent requests to the metadata management server and improving file processing efficiency. In addition, the metadata management server's management method based on this dual-version data block approach of open version + put version can improve file management efficiency. The specific process of this file processing solution is as follows:

[0105] 1. The client sends a file operation request for the first file to the metadata management server. This first file is stored in the cloud storage system in a block-based manner, and the metadata management server manages the metadata of the files stored in the cloud storage system. The cloud storage system contains at least one file, and the first file refers to any file within the cloud storage system.

[0106] 2. Receive the first operation version identifier (open version) of the first file. The first operation version identifier of the first file is returned by the metadata management server in response to the file operation request. For example, if the first file is opened for the first time, then open version = 1; if the first file is opened for the i-th time, then open version = i.

[0107] 3. Based on the first operation version identifier, perform file operation processing on the first file in the cloud storage system; file operation processing is used to indicate that at least one data processing operation is performed on the data blocks of the first file. For example, data processing operations include: data write operation and data deletion operation. The data blocks that change corresponding to each data processing operation performed on the first file all possess the aforementioned first operation version identifier.

[0108] 4. During the execution of file operation processing, a first upload version identifier is generated for the first data block in the first file. Here, the first data block refers to the data block in the first file that changes with any data processing operation. That is, each time the client performs a data processing operation (such as a data write operation) on a data block in the first file, the client will assign a put version to the updated data block. For example, put version = 1 is assigned to the first data block, and put version = 2 is assigned to the second data block.

[0109] 5. Send the first operation version identifier and the first upload version identifier as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information. Steps 3-5 above can be repeated, and each updated data block contains the aforementioned open version and corresponding put version, and is sent to the metadata management server.

[0110] In the aforementioned file processing scheme, this application provides a file data block management scheme in a cloud storage system. On the one hand, this scheme supports the metadata management server assigning an open version (operation version identifier) ​​to the client each time the client opens a file. Subsequently, the client can upload (or delete) any number of data blocks without the client frequently sending requests to the metadata management server, thus improving file processing efficiency. On the other hand, after the client performs a data processing operation (such as data write operation, data deletion operation, or data update operation) on a data block in the file, it can assign a put version (upload version identifier) ​​to the data block that changes with this data processing operation. The client can then distinguish different data blocks using different put versions. After the first file undergoes a data processing operation, the client sends the open version + put version dual-version data block to the metadata management server, so that the metadata management server can effectively manage the metadata of the first file based on the open version + put version information, thereby achieving efficient file management.

[0111] It should be noted that the document processing solution provided in this application also needs to address the following two points:

[0112] (1) The relevant data involved in the file processing process of this application (such as file metadata, first operation version identifier, and first upload version identifier, etc.). When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target must be obtained, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the region, conform to the principles of legality, legitimacy, and necessity, and not involve obtaining data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data involved in the embodiments of this application is obtained after separate authorization from the target. In addition, when obtaining separate authorization from the target, the purpose of the relevant data is indicated to the target.

[0113] (2) It is understood that the term "at least once" in this application means once or more times, and "more times" means twice or more; for example, at least one data processing operation means one, two or more data processing operations. The terms "first", "second", etc. are used to distinguish the same or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity or execution order.

[0114] The document processing system provided in this application will be described in detail below.

[0115] I. Introduction to the System Macro Architecture.

[0116] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a file processing system provided in an embodiment of this application. For example... Figure 1 As shown in the diagram, the architecture of this file processing system may include at least: a client cluster 100, a metadata management server 104, and a cloud storage system 105. The client cluster includes at least multiple clients such as a first client 101, a second client 102, and a third client 103. It should be understood that the number of clients in the client cluster is for illustrative purposes only, and this embodiment does not limit the number or type of clients. Specifically, any client in the client cluster can store files in the cloud storage system 105, and each file is stored in the cloud storage system in a block-based manner. The metadata management server 104 manages the metadata of the files stored in the cloud storage system 105. Furthermore, any client in the client cluster can connect directly or indirectly to the metadata management server 104 and the cloud storage system 105 via a network. The network may include, but is not limited to, wired networks and wireless networks. Wired networks include local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs), while wireless networks include Bluetooth, Wi-Fi (Wireless Fidelity, a standard wireless LAN), and other networks that enable wireless communication.

[0117] Specifically, the aforementioned clients include, but are not limited to: mobile phones, tablets, laptops, desktop computers, gaming devices, in-vehicle devices, aircraft, wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), virtual reality devices (such as VR (Virtual Reality) devices, AR (Augmented Reality) devices), in-vehicle devices, and any terminal device with data processing capabilities; wherein, the types of clients in the client cluster can be the same or different, for example: the first client 101 can be a laptop, the second client 102 can be a desktop computer, and the third client 103 can be a tablet. This application does not limit the number or type of clients in the client cluster. In addition, the aforementioned servers refer to: independent physical servers, server clusters or distributed systems composed of multiple physical servers, and cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0118] Next, with Figure 1 Taking any client (e.g., the first client 101) as an example, the data interaction process between the first client 101, the metadata management server 104, and the cloud storage system 105 will be described accordingly, specifically including:

[0119] ① The first client 101 sends a file operation request for the first file to the metadata management server 104. The first file is stored in the cloud storage system 105 in the form of data blocks, and the metadata management server is used to manage the metadata of the files stored in the cloud storage system 105.

[0120] ② In response to a file operation request, the metadata management server 104 returns the first operation version identifier (open version) of the first file to the first client 101. For example, if the first file is opened for the first time, then open version = 1; if the first file is opened for the second time, then open version = 2. Optionally, the metadata management server 104 may verify the first client 101 (such as at least one of permission verification, security verification, or legitimacy verification), and respond to the file operation request after the first client 101 passes the verification.

[0121] ③ The first client 101 performs file operation processing on the first file in the cloud storage system 105 based on the first operation version identifier; the file operation processing is used to indicate that at least one data processing operation is performed on the data blocks of the first file. For example, the data processing operation includes any one of the following: data write operation, data deletion operation, data update operation, etc. Among them, the data blocks that change corresponding to each data processing operation performed on the first file all have the aforementioned first operation version identifier (e.g., open version = 1).

[0122] ④ During the execution of file operation processing, the first client 101 generates a first upload version identifier for the first data block in the first file. Here, the first data block refers to the data block in the first file that changes with any data processing operation. For example, after the first client 101 writes data to the data block of the first file in the cloud storage system 105 and generates the first data block, the first client 101 generates a put version = 1 for the first data block; as another example, after the first client 101 writes data to the first data block of the first file in the cloud storage system 105 again and generates the second data block, the first client 101 generates a put version = 2 for the second data block; and so on. Each time the client performs a data processing operation (such as a data write operation) on a data block in the first file, the client will allocate a put version for the updated data block.

[0123] ⑤ The first client 101 sends the first operation version identifier "open version" and the first upload version identifier "put version" as description information of the first data block to the metadata management server 104.

[0124] ⑥ The metadata management server 104 can update the metadata of the first file based on the open version + put version of the first data block, such as updating the data storage structure of the first file.

[0125] In one possible implementation, the file processing system provided in this application embodiment can be deployed on a blockchain node. For example, the first client 101, the second client 102, the third client 103, and the server 104 in the file processing system can all be treated as blockchain node devices, jointly forming a blockchain network. Therefore, the file processing flow executed in this application embodiment can be executed on the blockchain, which can ensure the fairness and impartiality of the file processing flow, make the flow traceable, and ensure data security during the file processing, thereby improving the security and reliability of the entire file processing flow.

[0126] II. Introduction to the modules of the system's layered structure.

[0127] Please see Figure 2 , Figure 2 This is a schematic diagram of the module structure of a cloud storage system provided in an embodiment of this application. For example... Figure 2 As shown, this cloud storage system mainly includes the following three service modules: big data logic module, metadata management module, and data block storage module. The functions of each service module are explained below:

[0128] (1) Big Data Logic Module: This module provides the client side with various logical processing frameworks for big data (files), such as MR (Map Reduce, a batch processing computing framework designed for large-scale data), Spark (a general big data processing framework), Flink (a stream processing engine for big data), and Presto (a distributed SQL query engine).

[0129] (2) Metadata management module (Meta service): Provides metadata management services for managing the metadata (COS Meta) of files stored in the cloud storage system, specifically managing the data blocks in the files; this metadata management module can be deployed separately in the server (the server with the metadata management module deployed is the metadata management server), or it can be integrated into the cloud storage system.

[0130] (3) Block Storage Module (COS): This module provides block storage services for various files in the cloud storage system and can be deployed in the cloud storage system.

[0131] Based on this, by introducing the layered modules of the cloud storage system, the complex processing logic of big data in the cloud can be clearly decomposed into various service modules, ensuring that the functions of each service module are relatively independent. Furthermore, with the joint collaboration of these service modules, efficient management of file data blocks in big data in the cloud can be achieved, providing high-quality data storage services for cloud computing and cloud storage services on the client side, thereby improving the scalability and data processing efficiency of the entire file processing system.

[0132] The file processing system provided in this application is used to implement a file data block management scheme in a cloud storage system. On the one hand, the scheme supports the allocation of an open version (operation version identifier) ​​to the client by the metadata management server each time the client opens a file. Subsequently, the client can upload (or delete) any number of data blocks without the client frequently sending requests to the metadata management server, which can improve file processing efficiency. On the other hand, after the client performs a data processing operation (such as data write operation, data deletion operation, data update operation) on a data block in the file, it can allocate a put version (upload version identifier) ​​to the data block that changes with this data processing operation. Then, the client can distinguish different data blocks by maintaining different put versions. After each data processing operation is performed on the first file, the client sends the data block with both open version and put version to the metadata management server, so that the metadata management server can effectively manage the metadata of the first file based on the open version and put version information, thus achieving efficient file management.

[0133] It is understood that the file processing system described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0134] The specific embodiments of the document processing scheme involved in this application will be described in detail below.

[0135] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a file processing method provided in an embodiment of this application. This file processing method can be performed by a computer device (such as...) Figure 1 Execute on any of the clients shown; as Figure 3 As shown, the file processing method includes, but is not limited to, the following steps S301-S305:

[0136] S301. Send a file operation request for the first file to the metadata management server.

[0137] Specifically, the first file is stored in the cloud storage system in a data block format. The cloud storage system stores at least one file, and the first file here refers to any file or specified file that the client needs to access from the cloud storage system. In this application, storing any file (such as the first file) in the cloud storage system in a data block format specifically means dividing the first file into multiple data blocks of a preset length (e.g., 4MB). These data blocks include two types: immutable blocks of a preset length and mutable blocks smaller than the preset length. Based on this, each data block included in the first file is stored in the cloud storage system. Furthermore, the metadata management server is used to manage the metadata of the files stored in the cloud storage system. Since the files are stored in the cloud storage system in a data block format, the metadata management server's management of file metadata essentially means managing the data blocks included in the files.

[0138] In one possible implementation, the client can request access to a first file in the cloud storage system and, after requesting the invocation of the API interface for that first file, trigger the generation of a file operation request for the first file, which is then sent to the metadata management server. Specifically, the aforementioned API interface includes, but is not limited to, any one of: create, append, and truncate. Here, create refers to creating a new file or overwriting an existing file. If the file already exists, it will be truncated (cleared); if the file does not exist, a new file will be created. append refers to adding data to the end of a file, such as writing data to the first file. truncate refers to truncating / deleting data in the first file. That is, the file operation request here is used to request file processing operations (e.g., data writing operation append, data deletion operation truncate, or data update operation) on the first file; optionally, the file operation request may include the file identifier (File ID) of the first file.

[0139] S302. Receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request.

[0140] The first operation version identifier (open version) is associated with the first file and is used to indicate the number of times the first file has been opened. For example, if the first file is created for the first time, then open version = 0; if the first file is opened for the first time, then open version = 1; and if the first file is opened for the third time, then open version = 3.

[0141] S303. Based on the first operation version identifier, perform file operation processing on the first file in the cloud storage system; the file operation processing is used to instruct at least one data processing operation to be performed on the data blocks of the first file.

[0142] The file operation processing is used to instruct at least one data processing operation to be performed on a data block of the first file. This data processing operation includes at least one of the following: data write operation, data deletion operation, and data update operation. In this embodiment, the main focus is on the data write operation of the first file. For example, based on a first operation version identifier, multiple data write operations can be performed on the first file. Each time a data processing operation is performed on the first file, one or more data blocks in the first file will change accordingly, and each data processing operation can carry the aforementioned first operation version identifier for the corresponding changed data blocks.

[0143] Based on this, the client can perform multiple data processing operations (such as data writing operations) on the data blocks in the first file, based on the first operation version identifier returned by the metadata management server. In other words, the client can upload or write new data blocks to the cloud storage system multiple times without having to interact with the metadata management server every time data is written, thus reducing the interaction frequency and improving the client's data processing efficiency.

[0144] S304. During the execution of file operation processing, a first upload version identifier is generated for the first data block in the first file. The first data block refers to the data block in the first file that changes with any data processing operation.

[0145] In the process of any data processing operation (such as a data write operation) performed by the client on a data block in the first file, the client can assign an upload version identifier (put version) to the data block corresponding to this data processing operation. Since the processing procedure is similar for each data processing operation, this application uses any data processing operation (such as a data write operation, a data deletion operation, or a data update operation) performed on the first file as an example for illustration.

[0146] I. Data writing process (append):

[0147] In one possible implementation, the first file includes an unsealed block that can be written to. The data processing operation includes writing data to the first file; the process by which the client generates a first upload version identifier for the first data block in the first file during the execution of file operation processing includes the following steps (1)-(4):

[0148] (1) Load the variable data block from the first file into the client's memory from the cloud storage system. The variable data block refers to a data block with a data length less than a preset length, and the variable data block has an initial upload version identifier i (which can be obtained from the metadata management server), where i is an integer.

[0149] (2) Obtain the first data corresponding to the data write operation, and write the first data into the variable data block in memory to generate the first data block. Since the cloud storage system itself does not support data write operations (or data deletion operations), the data write operation needs to be performed in the client's memory (a temporary cache area in the client).

[0150] (3) Based on the initial upload version identifier i of the variable data block, generate the first upload version identifier i+1 for the first data block. After performing a data write operation on the variable data block (unsealed block), a put version can be assigned to the new variable data block (i.e., the first data block unsealed block-1). For example, if the put version of unsealed block is 1, then the put version of unsealed block-1 is 2.

[0151] (4) When the cloud storage system's cache interface (flush interface) is detected to be called, the first data block in memory is persisted to the cloud storage system. Here, persistence means saving it permanently or indefinitely; that is, when the client calls the COS system's flush interface, the unsealed block-1 (the first data block) after the data is written in memory will be written to the COS system for permanent storage, thereby realizing the process of writing data to the first file in the cloud storage system.

[0152] Furthermore, embodiments of this application support performing multiple data write operations on unsealed block-1. In one possible implementation, the multiple data write operation process includes the following steps:

[0153] ① Load the first data block (unsealed block-1) from the cloud storage system into the client's memory, and obtain the second data to be written corresponding to the data write operation; similarly, the first data block has the first upload version identifier i+1 (which has been generated by the aforementioned steps (1)-(3)).

[0154] ② Write the second data into the first data block in memory to generate the second data block, and generate the second upload version identifier i+2 for the second data block based on the first upload version identifier i+1 of the first data block. Based on this, the second data block unsealed block-2 can be obtained after performing a data write operation on the first data block unsealed block-1; optionally, the above steps ①-② can be executed repeatedly, thereby enabling multiple data write operations to be performed on the variable data block unsealed block in the first file, and each new unsealed block (such as the first data block and the second data block mentioned above) generated by each data write operation is assigned an upload version identifier put version.

[0155] ③ When the length of the variable data block after the data write operation reaches the preset length (e.g., 4MB), submit a fixed data block (immutable block) of the preset length to the cloud storage system. For example, call the flush interface of the COS system to submit the immutable block to the COS system.

[0156] ④ The variable data block after deleting the preset length is used as the new variable data block in the first file, and a new upload version identifier is set for the new variable data block. The data length of the new variable data block can be 0 (that is, the data in the variable data block in the first file is empty at this time). At this time, a new putversion can be reinitialized for the new variable data block (e.g., 0 or 1).

[0157] For example, please see Figure 4 , Figure 4 This is a schematic diagram illustrating a process for a client to write data to a cloud storage system, as provided in an embodiment of this application. Figure 4As shown, assuming the first file in the cloud storage system includes variable data blocks, the unsealed block has a data length of 1MB and a put version of 1, then after performing a data write operation (e.g., a put operation to write 1MB of data) on the variable data block, the resulting first data block (unsealed block-1) has a data length of 2MB, and its put version is 2. Further, performing another data write operation (e.g., a put operation to write 1MB of data) on the first data block (unsealed block-1) results in a second data block (unsealed block-2) with a data length of 3MB, and its put version is 3. This continues until the unsealed block reaches a preset length (e.g., 4MB), at which point the unsealed block is submitted as an immutable block to the cloud storage system COS. Based on this, the process of writing data to the first file mainly involves writing data multiple times to the unsealed block of the first file (that is, performing multiple data write operations on the unsealed block). After each data write operation, a put version can be assigned to the newly generated unsealed block to distinguish different unsealed blocks. Compared with the method of frequently interacting with the metadata management server for each data write, this application can support multiple data write operations on the file after the client opens the file, thereby improving data processing efficiency.

[0158] II. Data deletion (truncate) process:

[0159] In one possible implementation, the data processing operation also includes a data deletion operation on the first file; then, the data deletion process performed on the first file in the cloud storage system is as follows: (1)-(3):

[0160] (1) Load the second data block from the cloud storage system into the client's memory; the second data block has a second upload version identifier i+2 (such as put version=3 corresponding to unsealed block-2 above).

[0161] (2) Obtain the data to be deleted corresponding to the data deletion operation, and delete the data to be deleted from the second data block in memory to generate the third data block; wherein, the data deletion operation can be used to indicate the deletion of data of a specified length in the second data block (such as 1MB of data, which refers to the data to be deleted), or to delete specified data in the second data block (the specified data here is the data to be deleted), and then delete the data to be deleted from the second data block to obtain the third data block unsealed block-3.

[0162] (3) Based on the second upload version identifier i+2 of the second data block, generate the third upload version identifier i+3 for the third data block; for example, the put version corresponding to the third data block unsealed block-3 is 4.

[0163] The first, second, and third data blocks each have a first operation version identifier. That is, under the first operation version identifier, each time a data processing operation (such as a data write operation or a data delete operation) is performed on a pair of data blocks in the first file, a putversion will be assigned to the data block that changes with this data processing operation. However, each data block generated by the above data processing operation corresponds to the same open version.

[0164] III. Data Reading Process:

[0165] In one possible implementation, any data block in the first file (e.g., an immutable block or a mutable block) is stored as an object in the cloud storage system, with each object corresponding to an object identifier (block key). File operation processing is also used to indicate data read operations on the first file. Therefore, the process of reading data in the cloud storage system is as follows: (1)-(3):

[0166] (1) Obtain the metadata of the first file sent by the metadata management server. The metadata of the first file records the description information of each data block in the first file; the description information of any data block includes: the operation version identifier (openversion) and the upload version identifier (put version). Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the data block structure of a first file provided in an embodiment of this application, such as... Figure 5As shown, the first file may include N (N is a positive integer) fixed data blocks (immutable blocks, all with a preset data length of 4MB) and one variable data block (mutable block, with a data length less than 4MB), and the N fixed data blocks are arranged in order; in the metadata of the first file, the description information of the immutable block may include: operation version identifier (open version) and block number (block id); the description information of the mutable block may include: operation version identifier (open version), upload version identifier (put version) and data length (less than the preset length).

[0167] (2) Parse the metadata of the first file to obtain the target description information of the data block to be read corresponding to the data reading operation; if the data block to be read refers to the kth data block in the first file (such as the 10th data block), then the description information of the kth data block can be obtained from the metadata of the first file as the target description information. For example, if the kth data block is an immutable block, the obtained target description information includes: open version and block number (such as block id=10).

[0168] (3) Generate a target object identifier based on the target description information, and read the target data block indicated by the target object identifier from the first file in the cloud storage system based on the target object identifier. For example, obtain the file identifier (File ID) of the first file, and combine the File ID, open version, and blockid into a target object identifier block key. Further, based on this target object identifier block key, the target data block indicated by the block key (i.e., the kth data block of the first file) can be read from the COS system.

[0169] Based on this, in the process of reading any data block in the first file, this application supports obtaining the metadata of the first file from the metadata management server, and then determining the target object identifier of the data block to be read based on the metadata of the first file, and then conveniently reading the corresponding target data block from the cloud storage system based on the target object identifier, making the data reading process efficient and convenient.

[0170] S305. The first operation version identifier and the first upload version identifier are sent as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

[0171] Since the data in the first file will change (e.g., data increase or decrease) every time a file processing operation (such as data write operation or data delete operation) is performed, the relevant information of the changed data blocks in the first file (including open version and putversion) needs to be sent to the metadata management server after each file processing operation. The metadata management server then updates the metadata of the first file based on the received relevant information.

[0172] In one possible implementation, the client needs to repeat steps S304-S305 above each time it performs a data processing operation on the first file. For example, suppose that during the file operation processing of the first file in the cloud storage system, three data processing operations (such as data write operations) are specifically performed on the data blocks of the first file. If the first data write operation refers to writing data into the variable data block of the first file and creating the first data block, then the description information of the first data block includes: the first operation version identifier and the first upload version identifier. Therefore, the client needs to send the description information of the first data block to the metadata management server. Similarly, if the second data write operation involves writing data into the first data block and creating the second data block, the description information of the second data block includes: a first operation version identifier and a second upload version identifier. Therefore, the client needs to send the description information of the second data block to the metadata management server. And, if the third data write operation involves writing data into the second data block and creating the third data block, the description information of the third data block includes: a first operation version identifier and a third upload version identifier. The client needs to send the description information of the third data block to the metadata management server. The first, second, and third data blocks all correspond to the same open version, and each data block has a different put version.

[0173] In one possible implementation, if the first file is closed and then reopened, the `openversion` is incremented by 1. Specifically, assume the first operation version identifier of the first file is represented as `j`, where `j` is an integer. In response to a call to the file closing interface (`close` interface) of the cloud storage system, the first data block of the first file is persisted to the cloud storage system; a file operation request for the first file is sent again to the metadata management server, and the second operation version identifier of the first file returned by the metadata management server is received. This second operation version identifier is `j+1`, that is, if the client closes the first file and then reopens it, the `openversion` assigned to the client by the metadata management server is incremented by 1. Furthermore, the client can perform one or more data processing operations (e.g., data write operations, data deletion operations, data update operations, etc.) on the first file based on the second operation version identifier (`j+1`). This process is detailed in steps S303-S305 above and will not be repeated here.

[0174] This application provides a scheme for a client to perform file processing in a cloud storage system. On the one hand, this scheme allows the metadata management server to assign an openversion (operation version identifier) ​​to the client each time the client opens a file. Subsequently, the client can upload (or delete) any number of data blocks without frequently sending requests to the metadata management server, thus improving file processing efficiency. On the other hand, after the client performs a data processing operation (such as data write operation, data deletion operation, or data update operation) on a data block in the file, it can assign a put version (upload version identifier) ​​to the data block that changes with this data processing operation. The client can then distinguish different data blocks using different put versions. After the first file has undergone a data processing operation, the client sends the data block with both open version and put version to the metadata management server. This allows the metadata management server to effectively manage the metadata of the first file based on the open version and put version information, achieving efficient file management.

[0175] Please see Figure 6 , Figure 6 This is a schematic flowchart of another file processing method provided in an embodiment of this application. This file processing method can be performed by a computer device (such as...) Figure 1 The metadata management server shown executes this; as... Figure 6 As shown, the file processing method includes, but is not limited to, the following steps S601-S604:

[0176] S601, Receive a file operation request for the first file sent by the client.

[0177] Specifically, the first file is stored in the cloud storage system in a block-based manner. The metadata management server manages the metadata of the files stored in the cloud storage system. For example, the file's metadata includes descriptive information about each data block in the file. Therefore, the metadata management server's management of the file refers to managing the file's data storage structure based on the descriptive information of each data block. A file operation request is generated after the client calls an API interface (such as create, append, or truncate) in the cloud storage system for the first file. In other words, the file operation request is used to request file processing operations (e.g., append, truncate, or update) to be performed on the first file.

[0178] In one possible implementation, the metadata management server can construct a first data storage structure for the first file based on its metadata, and manage the first file according to this structure. The specific process of constructing the first data storage structure for the first file is explained below:

[0179] Specifically, assume the first file contains N immutable data blocks and one unsealed data block. Each immutable data block has a length equal to a preset length (e.g., 4MB), and the unsealed data block has a length less than the preset length. N is a positive integer, and the metadata of the first file records descriptive information for each data block. Then, the process by which the metadata management server constructs the first data storage structure for the first file based on its metadata is as follows:

[0180] (1) Obtain the first description information of each fixed data block from the metadata of the first file. The first description information includes: block number (block id) and operation version identifier (open version).

[0181] (2) Based on the first description information of N fixed data blocks, construct K interval trees for the N fixed data blocks, and store the K interval trees in the first storage area in order; wherein, each interval tree is used to record at least one fixed data block with the same operation version identifier, and K is a positive integer. For example, the first file includes 5 immutable blocks (stored in order). The open version of the first two blocks of these 5 immutable blocks is 1, and the open version of the last three blocks is 2. Therefore, the following two interval trees are constructed for these 5 immutable blocks: (1->[1,2],2->[3,5]). Each interval tree stores data in a k(key)-v(value) key-value pair manner. In each interval tree, the key refers to the open version, and the corresponding value refers to the sequence number of at least one block with the same open version. For example, the data block with open version = 1 includes the first immutable block and the second immutable block; and the data block with open version = 2 includes the third immutable block, the fourth immutable block, and the fifth immutable block.

[0182] (3) Obtain the second description information of the variable data block from the metadata of the first file and store the second description information in the second storage area; the second description information here includes at least one of the following: initial operation version identifier (open version), initial upload version identifier (put version), initial length (less than the preset length, for example, it can be 0MB, 1MB, etc.).

[0183] (4) Combine the first storage area and the second storage area to form the first data storage structure of the first file. For example, please refer to [link to example]. Figure 7 , Figure 7 This is a schematic diagram of a first data storage structure for a first file provided in an embodiment of this application. For example... Figure 7As shown, the first data storage structure includes a first storage area and a second storage area. The first storage area stores the first description information of each fixed data block in the first file, and the second storage area stores the second description information of the variable data blocks in the first file. For example, the first storage area records interval tree 1 and interval tree 2. Interval tree 1 records that the first and second immutable blocks have the same open version (1). Similarly, interval tree 2 records that the third, fourth, and fifth immutable blocks have the same open version (2). In addition, the second storage area is used to record the second description information of unsealed blocks. The second description information here includes: the data length of the unsealed block (e.g., 1MB), the put version (1), and the open version (2).

[0184] Based on the steps (1)-(4) above, the metadata management server can construct the corresponding data storage structure for the first file according to the description information of each data block (immutable block or unsealed block) in the first file (e.g., Figure 7 The interval set structure shown in this application can record multiple data blocks (immutable blocks) with the same open version. Compared with the method of recording the description information of each data block, the data storage structure designed in this application can use less memory to manage a large number of data blocks in the first file, thus achieving efficient management of data blocks.

[0185] S602. In response to a file operation request, send the first operation version identifier of the first file to the client.

[0186] In one possible implementation, client verification of file operation requests is supported. The specific process is as follows: The metadata management server can obtain the target client identifier of the file operation request and verify the client based on this identifier. This verification includes at least one of the following: permission verification, security verification, and legality verification. For example, the metadata management server stores an access permission policy for a first file. This access permission policy includes at least one client identifier that has access or operation permissions for the first file. If the access permission policy includes the aforementioned target client identifier, the permission verification for that client passes; otherwise, it fails. Further, if the client verification passes, the file operation request is responded to, and subsequent steps are triggered; otherwise, if the client verification fails, the file operation request is deleted. This implementation, which supports client verification of file operation requests, ensures data security during file processing.

[0187] In one possible implementation, the metadata management server responds to a file operation request by sending a first operation version identifier of the first file to the client, including the following two cases:

[0188] Case 1: If the file operation request is used to indicate that the first file is to be created, then the first operation version identifier is directly set to the initial value (such as 0); that is, when the first file is created for the first time, the first operation version identifier open version = 0 can be generated directly.

[0189] Scenario 2: If the file operation request indicates that a file operation (such as append or truncate) is to be performed on the first file, then the initial operation version identifier of the variable data block is obtained from the first data storage structure of the first file. A first operation version identifier is generated based on the initial operation version identifier and returned to the client. For example, if the initial operation version identifier open version = 1, then the first operation version identifier open version = 2. That is, if a file operation (append or truncate) is performed on an existing first file, the open version returned to the client is incremented (e.g., +1) based on the open version of the existing variable data block in the first file.

[0190] S603. Receive the description information of the first data block sent by the client. The description information includes the first operation version identifier and the first upload version identifier of the first data block in the first file.

[0191] The first upload version identifier is generated by the client for the first data block in the first file during file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. File operation processing indicates that at least one data processing operation is performed on the data block of the first file, such as a data append operation, a data truncate operation, or a data update operation. It should be noted that detailed information on how to generate the first upload version identifier and how to perform file operation processing on the first file can be found in [reference needed]. Figure 3 The steps performed by the client in the embodiments are not repeated here.

[0192] S604. Update the metadata of the first file based on the description information of the first data block.

[0193] Specifically, because the data blocks in the first file change during data processing operations (e.g., data is written to or deleted from a data block), and descriptive information (such as open version, put version, etc.) is generated during these operations, the metadata management server needs to update the metadata of the first file based on the descriptive information of the changed data block. The specific process for updating the metadata of the first file is explained below:

[0194] In one possible implementation, the specific process of updating the metadata of the first file based on the description information of the first data block includes the following steps (1)-(2): (1) Obtaining the metadata of the first file and constructing a first data storage structure for the first file based on the metadata of the first file; wherein, the specific process of how to construct the first data storage structure can refer to the relevant process in step S601 above, and will not be repeated here. That is, the metadata of the first file is stored in the form of data blocks, and the metadata management server can be configured as designed in this application. Figure 7The interval set structure shown uses less memory to manage a large number of data blocks in the first file, achieving efficient management of data blocks. (2) The first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure after the first file is updated. Specifically, the first data storage structure includes a first storage area (used to store the first description information of fixed data blocks) and a second storage area (used to store the second description information of variable data blocks); then, the update process of the data storage structure is as follows: parse the description information of the first data block to obtain the target length corresponding to the first data block; and determine the first storage area and / or the second storage area in the first data storage structure to be updated based on the target length, specifically including the following situations:

[0195] ① Update the data in the second storage area. The specific process includes: if the target length is less than a preset length, updating the second storage area in the first data storage structure based on the description information of the first data block, thus obtaining the second data storage structure of the first file. Please refer to [link / reference]. Figure 8a , Figure 8a This is a schematic diagram illustrating a process for updating a first data storage structure according to an embodiment of this application. Figure 8a As shown, assuming the preset length is 4MB and the target length is 2MB, that is, the first data block is still a variable data block (unsealed block). Then, the second description information of the variable data block originally recorded in the second storage area (initial operation version identifier open version=2, initial upload version identifier put version=1, initial length 1MB) is directly replaced with the description information of the first data block (including: first operation version identifier open version=2, first upload version identifier put version=2, target length 2MB).

[0196] ② Update the data in the first storage area. The specific process includes: if the target length equals the preset length, then treat the first data block as a fixed data block and add the first operation version identifier of the first data block to the first storage area of ​​the first data storage structure, thus obtaining the second data storage structure of the first file. Please refer to [link to relevant documentation]. Figure 8b , Figure 8b This is a schematic diagram illustrating another process for updating the first data storage structure provided in an embodiment of this application. For example... Figure 8bAs shown, assuming the preset length is 4MB and the target length is 4MB, that is, the first data block is a fixed data block (immutable block). Then the open version of the immutable block can be added to the existing interval tree corresponding to the first storage area. For example, if the description information of the first data block includes the first operation version identifier open version=2, then the first data block can be added to the interval tree 2 (because the interval tree 2 maintains open version=2), and a block number block id=6 can be added to the first data block (the block number is sequentially incremented because the first file currently contains 5 immutable blocks). Then the interval tree 2 in the updated second storage area records: 2->[3,6].

[0197] ③ Update data in both the first and second storage areas. The specific process includes: if the target length is greater than a preset length, divide the first data block into at least one fixed data block and a new variable data block; add the first operation version identifier of each fixed data block to the first storage area of ​​the first data storage structure, and update the data in the second storage area of ​​the first data storage structure based on the description information of the new variable data block, thus obtaining the second data storage structure of the first file. Please refer to [link / reference]. Figure 8c , Figure 8c This is a schematic diagram illustrating another process for updating the first data storage structure provided in an embodiment of this application. For example... Figure 8c As shown, assuming a preset length of 4MB and a target length of 6MB, the first data block can be divided into an immutable block and a new variable data block. Further, the open version of the immutable block can be added to the existing interval tree corresponding to the first storage area. For example, if the description information of the first data block includes the first operation version identifier open version = 2, then the first data block can be added to interval tree 2, and a block number block id = 6 can be added to the first data block. Thus, the updated interval tree 2 in the second storage area records: 2->[3,6]. Furthermore, the second description information of the variable data block originally recorded in the second storage area (initial operation version identifier open version = 2, initial upload version identifier put version = 1, initial length 1MB) is replaced with the description information of the new variable data block (e.g., open version = 2, put version = 1, target length 2MB).

[0198] In summary, during the update of the first data storage structure of the first file, data update processing is supported for at least one of the first storage area or the second storage area. This enables file operation processing performed on the first file on the client side. The metadata management server can update the data storage structure corresponding to the metadata of the first file in a timely manner to achieve timely updates of the file data structure and ensure efficient management of file data blocks.

[0199] In this embodiment of the application, the metadata management server mainly involves building an interval set data storage structure for the file based on the file's metadata during the process of managing the file's metadata. This data storage structure is built through dual-version data blocks, which can use less memory to manage a large number of data blocks in the file, thereby improving the management efficiency of data blocks in the file and reducing memory usage.

[0200] Please see Figure 9 , Figure 9 This is a schematic diagram of the interactive flow of a file processing method provided in an embodiment of this application. This file processing method can be... Figure 1 The client, the metadata management server, and the cloud storage system (COS) all work together to execute the command; for example... Figure 9 As shown, the interactive flow of this file processing method includes the following steps S901-S908:

[0201] S901, The client sends a file operation request to the metadata management server.

[0202] The file operation request is used to request file operation processing on the first file. The first file is stored in the cloud storage system in the form of data blocks, and the metadata management server is used to manage the metadata of each file (such as the first file) stored in the cloud storage system.

[0203] In one possible implementation, the client can request access to the first file in the cloud storage system and request to call the API interface of the first file (such as any one of create, append, truncate, etc.) to trigger the generation of a file operation request for the first file, and then send the file operation request for the first file to the metadata management server.

[0204] S902. The metadata management server returns an acknowledgment character to the client, along with an operation version identifier.

[0205] In one possible implementation, the metadata management server can obtain the target client identifier that initiated the file operation request and verify the current client based on the target client identifier. This verification includes at least one of permission verification, security verification, and legality verification. For example, the metadata management server stores an access permission policy for the first file. This access permission policy includes at least one client identifier that has access or operation permissions for the first file. If the access permission policy includes the aforementioned target client identifier, the permission verification for that client passes; otherwise, it fails. Further, if the client verification passes, in response to the file operation request, a first operation version identifier for the first file is generated, and an acknowledgment character (ack) is returned to the client, carrying the first operation version identifier of the first file. It should be noted that the specific process of how the metadata management server generates the first operation version identifier (openversion) for the first file can be found in the above description. Figure 6 The relevant steps of step S602 in the embodiments will not be repeated here.

[0206] S903: The client uploads the first data block to the cloud storage system, carrying the upload version identifier (put version).

[0207] In one possible implementation, the first file includes an unsealed block that supports data writing. Specifically, the process of the client writing the first data block to the cloud storage system is as follows: (1) Load the unsealed block from the cloud storage system into the client's memory. The unsealed block has an initial upload version identifier put version = i, where i is an integer. (2) Obtain the first data corresponding to the data writing operation and write the first data into the unsealed block in memory to generate the first data block. (3) Based on the initial upload version identifier i of the unsealed block, generate a first upload version identifier put version = i + 1 for the first data block. (4) Call the cloud storage system's cache interface (flush interface) to persist the first data block in the client's memory to the cloud storage system, thereby completing the process of the client uploading the first data block to the cloud storage system. Optionally, if the first file is created for the first time, that is, the data in the first file is empty, then the data in the variable data block is also empty, and the first data block is the first data block written in the first file. In this case, the first upload version identifier (put version) generated for the first data block can be 0.

[0208] Specifically, data blocks written by the client to the cloud storage system (COS) are stored as objects, and when a block is written to COS, it corresponds to an object identifier, a data block key. The data block key mainly consists of two parts: a common part and a part unique to unsealed blocks.

[0209] a) The common section mainly includes the following parameters:

[0210] Key Type (identifier type) is used to identify the purpose of the data block key, such as identifying a data block.

[0211] FileID (File Identifier) ​​refers to the identifier of the file (such as the first file) where the current data block is stored.

[0212] ClusterID (cluster identifier) ​​is used in scenarios where multiple clusters share a COS bucket (optional).

[0213] The Block ID corresponds to the sequence number of the current data block in the file (such as 1, 2, 3, etc.). Since the blocks in this application have a fixed length (such as 4MB, and variable data blocks have their lengths recorded separately), the block number can be obtained by converting the file logical offset.

[0214] Open Version (Operation Version Identifier) ​​is a monotonically increasing value returned to the client by the metadata management server whenever the client opens a file (such as with create, append, or truncate operations).

[0215] b) Unique part of Unsealed Block: Put Version (upload version identifier). Whenever the client opens a file (for create, append, truncate), the client will initialize an initial value (e.g., 0). Whenever the client uploads an immutable block of the file, the Put Version value is initialized by the client and monotonically incremented.

[0216] In summary, each data block uploaded / written to the cloud storage system is stored as an object, and each data block has a corresponding data block key (object identifier). This object identifier includes, but is not limited to, at least one of the following parameters: Key Type, File ID, Cluster ID, Block ID, Open Version, and Put Version. These parameters can be used by the metadata management server to manage and update the file's metadata.

[0217] S904, The cloud storage system returns a message to the client that the first data block has been successfully uploaded.

[0218] Specifically, if the client successfully uploads the first data block to the cloud storage system, the cloud storage system can return a confirmation message to the client confirming the successful upload of the first data block.

[0219] S905. The client submits the description information of the first data block to the metadata management server.

[0220] Specifically, the client generates a description of the first data block based on the first operation version identifier returned by the metadata management server and the first upload version identifier generated by the client for the first data block; and submits the description of the first data block to the metadata management server. Optionally, the description of the first data block may also include: the data length, block sequence number, identifier type, and other data of the first data block.

[0221] S906, Metadata Management Server updates file metadata.

[0222] In one possible implementation, the metadata management server can update the metadata of the first file based on the description information of the first data block. The specific process is as follows: Obtain the metadata of the first file and construct a first data storage structure for the first file based on this metadata; update the first data storage structure based on the description information of the first data block to obtain the updated second data storage structure for the first file. Subsequently, the metadata management server can manage the metadata of the first file according to the updated second data storage structure. It should be noted that for details on how to construct the first data storage structure and the process of updating it, please refer to [reference needed]. Figure 6 The relevant processes of step S604 in the embodiments will not be repeated here.

[0223] S907, The metadata management server returns a message to the client indicating that the first data block has been successfully submitted.

[0224] S908: The client uploads the second data block to the cloud storage system, carrying the upload version identifier (put version+1).

[0225] Specifically, the client can continue to write a second data block to the cloud storage system, and the put version of this second data block is increased by 1 compared to the first data block; however, both the first and second data blocks have the same open version. Furthermore, by repeating steps S903-S907, the client can write any number of data blocks to the cloud storage system, and the put version identifier of each data block written is incremented compared to the previously written data block.

[0226] This application provides a file processing solution based on a cloud storage system. On one hand, this solution allows the metadata management server to assign an open version (operation version identifier) ​​to the client each time it opens a file. Subsequently, the client can upload (or delete) any number of data blocks without frequently requesting the metadata management server, thus improving file processing efficiency. On the other hand, during the management of file metadata, the metadata management server constructs an interval set data storage structure for the file using dual-version data blocks. This data storage structure can manage a large number of data blocks in a file with relatively little memory, thereby improving the efficiency of file data block management.

[0227] The following describes the relevant apparatus corresponding to the document processing method provided in this application.

[0228] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0229] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a file processing device provided in an embodiment of this application. The file processing device 1000 is applied to... Figure 1 Any of the clients shown, and used to execute this application. Figure 3 The corresponding steps in the file processing method provided in the embodiment. In specific implementation, the data processing device 1000 may specifically include:

[0230] Sending unit 1001 is used to send a file operation request for the first file to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0231] The receiving unit 1002 is used to receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request;

[0232] Processing unit 1003 is used to perform file operation processing on a first file in a cloud storage system based on a first operation version identifier; the file operation processing is used to instruct at least one data processing operation to be performed on the data blocks of the first file;

[0233] The processing unit 1003 is also used to generate a first upload version identifier for a first data block in the first file during the execution of file operation processing. The first data block refers to the data block in the first file that changes with any data processing operation.

[0234] The sending unit 1001 is also used to send the first operation version identifier and the first upload version identifier as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

[0235] In one possible implementation, the first file includes variable data blocks that can be written to; the data processing operation includes a data writing operation to the first file; during the execution of the file operation processing, the processing unit 1003 generates a first upload version identifier for the first data block in the first file, for use in performing the following operations:

[0236] The variable data block from the first file is loaded into the client's memory from the cloud storage system; the variable data block has an initial upload version identifier i, where i is an integer;

[0237] Obtain the first data corresponding to the data write operation, and write the first data into the variable data block in memory to generate the first data block;

[0238] Based on the initial upload version identifier i of the variable data block, generate the first upload version identifier i+1 for the first data block;

[0239] Specifically, when the cache interface of the cloud storage system is detected to be called, the first data block in memory is persisted to the cloud storage system.

[0240] In one possible implementation, the processing unit 1003 is further configured to perform the following operations:

[0241] The first data block is loaded from the cloud storage system into the client's memory, and the second data to be written corresponding to the data write operation is obtained.

[0242] Write the second data into the first data block in memory to generate the second data block, and generate the second upload version identifier i+2 for the second data block based on the first upload version identifier i+1 of the first data block;

[0243] Until the length of the variable data block after the data write operation is performed reaches the preset length, a fixed data block of the preset length is submitted to the cloud storage system.

[0244] The variable data block after deleting the preset length is used as the new variable data block in the first file, and a new upload version identifier is set for the new variable data block.

[0245] In one possible implementation, the data processing operation further includes a data deletion operation on the first file; the processing unit 1003 performs file operation processing on the first file in the cloud storage system and is also used to perform the following operations:

[0246] The second data block is loaded from the cloud storage system into the client's memory; the second data block has a second upload version identifier i+2;

[0247] Retrieve the data to be deleted corresponding to the data deletion operation, delete the data to be deleted from the second data block in memory, and generate the third data block;

[0248] Based on the second upload version identifier i+2 of the second data block, a third upload version identifier i+3 is generated for the third data block; wherein, the first data block, the second data block, and the third data block all have a first operation version identifier.

[0249] In one possible implementation, any data block in the first file is stored as an object in the cloud storage system, and each object corresponds to an object identifier; the file operation processing is also used to indicate data read operations on the first file; the processing unit 1003 is also used to perform the following operations:

[0250] Obtain the metadata of the first file sent by the metadata management server. The metadata of the first file records the description information of each data block in the first file. The description information of any data block includes at least one of the following: operation version identifier, upload version identifier, block sequence number, and length.

[0251] Parse the metadata of the first file to obtain the target description information of the data block to be read corresponding to the data read operation;

[0252] A target object identifier is generated based on the target description information, and the target data block indicated by the target object identifier is read from the first file in the cloud storage system based on the target object identifier.

[0253] In one possible implementation, the first operation version identifier of the first file is represented as j, where j is an integer; the processing unit 1003 is also used to perform the following operations:

[0254] In response to a call to the file closing interface of the cloud storage system, the first data block of the first file is persisted to the cloud storage system;

[0255] Send another file operation request for the first file to the metadata management server, and receive the second operation version identifier of the first file returned by the metadata management server; wherein, the second operation version identifier is j+1.

[0256] This application provides a scheme for a client to perform file processing in a cloud storage system. On the one hand, this scheme allows the metadata management server to assign an openversion (operation version identifier) ​​to the client each time the client opens a file. Subsequently, the client can upload (or delete) any number of data blocks without frequently sending requests to the metadata management server, thus improving file processing efficiency. On the other hand, after the client performs a data processing operation (such as data write operation, data deletion operation, or data update operation) on a data block in the file, it can assign a put version (upload version identifier) ​​to the data block that changes with this data processing operation. The client can then distinguish different data blocks using different put versions. After the first file has undergone a data processing operation, the client sends the data block with both open version and put version to the metadata management server. This allows the metadata management server to effectively manage the metadata of the first file based on the open version and put version information, achieving efficient file management.

[0257] Please see Figure 11 , Figure 11 This is a schematic diagram of a file processing apparatus provided in an embodiment of this application. The file processing apparatus 1100 is applied to... Figure 1 The metadata management server shown is used to execute this application. Figure 6 The corresponding steps in the file processing method provided in the embodiment. In specific implementation, the data processing device 1100 may specifically include:

[0258] The receiving unit 1101 is used to receive file operation requests for the first file sent by the client. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0259] The sending unit 1102 is used to send a first operation version identifier of the first file to the client in response to a file operation request;

[0260] The receiving unit 1101 is further configured to receive description information of the first data block sent by the client. The description information includes a first operation version identifier and a first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file.

[0261] The processing unit 1103 is used to update the metadata of the first file based on the description information of the first data block.

[0262] In one possible implementation, the processing unit 1103 updates the metadata of the first file based on the description information, for the following operations:

[0263] Obtain the metadata of the first file, and construct a first data storage structure for the first file based on the metadata of the first file;

[0264] The first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure after the first file is updated.

[0265] In one possible implementation, the first file includes N fixed data blocks and one variable data block. The data length of each fixed data block is equal to a preset length, and the data length of the variable data block is less than the preset length. N is a positive integer. The metadata of the first file records the description information of each data block in the first file. The processing unit 1103 constructs a first data storage structure for the first file based on the metadata of the first file, and performs the following operations:

[0266] Obtain the first description information of each fixed data block from the metadata of the first file. The first description information includes: block number and operation version identifier.

[0267] Based on the first description information of N fixed data blocks, K interval trees are constructed for the N fixed data blocks, and the K interval trees are stored in the first storage area in order; wherein, each interval tree is used to record at least one fixed data block with the same operation version identifier, and K is a positive integer;

[0268] The second description information of the variable data block is obtained from the metadata of the first file and stored in the second storage area; the second description information includes at least one of the following: initial operation version identifier, initial upload version identifier, and initial length;

[0269] The first storage area and the second storage area are combined to form the first data storage structure of the first file.

[0270] In one possible implementation, the sending unit 1102 sends a first operation version identifier of the first file to the client for performing the following operations:

[0271] If the file operation request is used to indicate that a first file is to be created, then the first operation version identifier is set to the initial value;

[0272] If the file operation request is used to indicate that the first file is to be processed by file operation, then the initial operation version identifier of the variable data block is obtained from the first data storage structure of the first file;

[0273] A first operation version identifier is generated based on the initial operation version identifier, and the first operation version identifier is returned to the client.

[0274] In one possible implementation, the processing unit 1103 is further configured to perform the following operations:

[0275] Obtain the target client identifier for sending the file operation request, and verify the client based on the target client identifier; wherein the verification includes at least one of the following: permission verification, security verification, and legality verification;

[0276] If the client verification passes, the step of retrieving the initial operation version identifier of the variable data block from the first data storage structure of the first file is triggered.

[0277] In one possible implementation, the processing unit 1103 updates the first data storage structure based on the description information of the first data block to obtain the updated second data storage structure of the first file, which is used to perform the following operations:

[0278] Parse the description information of the first data block to obtain the target length corresponding to the first data block;

[0279] If the target length is less than the preset length, the second storage area in the first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure of the first file.

[0280] In one possible implementation, the processing unit 1103 is further configured to perform the following operations:

[0281] If the target length is equal to the preset length, then the first data block is treated as a fixed data block, and the first operation version identifier of the first data block is added to the first storage area of ​​the first data storage structure to obtain the second data storage structure of the first file; or,

[0282] If the target length is greater than the preset length, the first data block is divided into at least one fixed data block and a new variable data block; the first operation version identifier of each fixed data block is added to the first storage area of ​​the first data storage structure, and the second storage area in the first data storage structure is updated based on the description information of the new variable data block to obtain the second data storage structure of the first file.

[0283] In this embodiment of the application, the metadata management server mainly involves building an interval set data storage structure for the file based on the file's metadata during the process of managing the file's metadata. This data storage structure is built through dual-version data blocks, which can use less memory to manage a large number of data blocks in the file, thereby improving the management efficiency of data blocks in the file and reducing memory usage.

[0284] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 1200 is used to execute the steps performed by the computer device (such as a client or metadata management server) mentioned in the foregoing method embodiments. The computer device 1200 may include one or more devices (such as servers, nodes, terminal devices, etc.) and internal components (such as chips, software modules, or hardware modules). The computer device may include at least one processor 1201 and a communication interface 1202. Further optionally, the computer device may also include at least one memory 1203 and a bus 1204. Additionally, the processor 1201, communication interface 1202, and memory 1203 are connected via the bus 1204.

[0285] (1) The processor 1201 is a module that performs arithmetic and / or logical operations. Specifically, it may be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a micro controller unit (MCU).

[0286] (2) The communication interface 1202 can be used to provide information input or output to at least one processor 1201. And / or, the communication interface 1202 can be used to receive data sent externally and / or send data externally, and can be a wired link interface including such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicle short-range communication technology, and other short-range wireless communication technologies, etc.). The communication interface 1202 can serve as a network interface.

[0287] (3) The memory 1203 is used to provide storage space, in which data such as the operating system and computer programs (including program instructions) can be stored. The memory 1203 can be one or a combination of random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), etc.

[0288] On the one hand, when the computer device is a client, the processor 1201 executes the following steps by running the computer program stored in the memory 1203:

[0289] Send a file operation request for the first file to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0290] Receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request;

[0291] Based on the first operation version identifier, file operation processing is performed on the first file in the cloud storage system; the file operation processing is used to indicate that at least one data processing operation is performed on the data blocks of the first file;

[0292] During the execution of file operation processing, a first upload version identifier is generated for the first data block in the first file. The first data block refers to the data block in the first file that changes with any data processing operation.

[0293] The first operation version identifier and the first upload version identifier are sent as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

[0294] In one possible implementation, the first file includes variable data blocks that can be written to; the data processing operation includes writing data to the first file; during the execution of the file operation processing, the processor 1201 generates a first upload version identifier for the first data block in the first file, for performing the following operations:

[0295] The variable data block from the first file is loaded into the client's memory from the cloud storage system; the variable data block has an initial upload version identifier i, where i is an integer;

[0296] Obtain the first data corresponding to the data write operation, and write the first data into the variable data block in memory to generate the first data block;

[0297] Based on the initial upload version identifier i of the variable data block, generate the first upload version identifier i+1 for the first data block;

[0298] Specifically, when the cache interface of the cloud storage system is detected to be called, the first data block in memory is persisted to the cloud storage system.

[0299] In one possible implementation, processor 1201 is also used to perform the following operations:

[0300] The first data block is loaded from the cloud storage system into the client's memory, and the second data to be written corresponding to the data write operation is obtained.

[0301] Write the second data into the first data block in memory to generate the second data block, and generate the second upload version identifier i+2 for the second data block based on the first upload version identifier i+1 of the first data block;

[0302] Until the length of the variable data block after the data write operation is performed reaches the preset length, a fixed data block of the preset length is submitted to the cloud storage system.

[0303] The variable data block after deleting the preset length is used as the new variable data block in the first file, and a new upload version identifier is set for the new variable data block.

[0304] In one possible implementation, the data processing operation further includes a data deletion operation on the first file; the processor 1201 performs file operation processing on the first file in the cloud storage system, and is also used to perform the following operations:

[0305] The second data block is loaded from the cloud storage system into the client's memory; the second data block has a second upload version identifier i+2;

[0306] Retrieve the data to be deleted corresponding to the data deletion operation, delete the data to be deleted from the second data block in memory, and generate the third data block;

[0307] Based on the second upload version identifier i+2 of the second data block, a third upload version identifier i+3 is generated for the third data block; wherein, the first data block, the second data block, and the third data block all have a first operation version identifier.

[0308] In one possible implementation, any data block in the first file is stored as an object in the cloud storage system, with each object corresponding to an object identifier; the file operation processing is also used to indicate data read operations on the first file; the processor 1201 is also used to perform the following operations:

[0309] Obtain the metadata of the first file sent by the metadata management server. The metadata of the first file records the description information of each data block in the first file. The description information of any data block includes at least one of the following: operation version identifier, upload version identifier, block sequence number, and length.

[0310] Parse the metadata of the first file to obtain the target description information of the data block to be read corresponding to the data read operation;

[0311] A target object identifier is generated based on the target description information, and the target data block indicated by the target object identifier is read from the first file in the cloud storage system based on the target object identifier.

[0312] In one possible implementation, the first operation version identifier of the first file is represented as j, where j is an integer; the processor 1201 is also used to perform the following operations:

[0313] In response to a call to the file closing interface of the cloud storage system, the first data block of the first file is persisted to the cloud storage system;

[0314] Send another file operation request for the first file to the metadata management server, and receive the second operation version identifier of the first file returned by the metadata management server; wherein, the second operation version identifier is j+1.

[0315] On the other hand, when the computer device is a metadata management server, the processor 1201 executes the following steps by running a computer program stored in the memory 1203:

[0316] The system receives file operation requests for the first file from the client. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system.

[0317] In response to a file operation request, send the first operation version identifier of the first file to the client;

[0318] The system receives description information of the first data block sent by the client. The description information includes a first operation version identifier and a first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file.

[0319] The metadata of the first file is updated based on the description information of the first data block.

[0320] In one possible implementation, the processor 1201 updates the metadata of the first file based on the description information to perform the following operations:

[0321] Obtain the metadata of the first file, and construct a first data storage structure for the first file based on the metadata of the first file;

[0322] The first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure after the first file is updated.

[0323] In one possible implementation, the first file includes N fixed data blocks and one variable data block. The data length of each fixed data block is equal to a preset length, and the data length of the variable data block is less than the preset length. N is a positive integer. The metadata of the first file records the description information of each data block in the first file. The processor 1201 constructs a first data storage structure for the first file based on the metadata of the first file, for performing the following operations:

[0324] Obtain the first description information of each fixed data block from the metadata of the first file. The first description information includes: block number and operation version identifier.

[0325] Based on the first description information of N fixed data blocks, K interval trees are constructed for the N fixed data blocks, and the K interval trees are stored in the first storage area in order; wherein, each interval tree is used to record at least one fixed data block with the same operation version identifier, and K is a positive integer;

[0326] The second description information of the variable data block is obtained from the metadata of the first file and stored in the second storage area; the second description information includes at least one of the following: initial operation version identifier, initial upload version identifier, and initial length;

[0327] The first storage area and the second storage area are combined to form the first data storage structure of the first file.

[0328] In one possible implementation, the processor 1201 sends a first operation version identifier of the first file to the client for performing the following operations:

[0329] If the file operation request is used to indicate that a first file is to be created, then the first operation version identifier is set to the initial value;

[0330] If the file operation request is used to indicate that the first file is to be processed by file operation, then the initial operation version identifier of the variable data block is obtained from the first data storage structure of the first file;

[0331] A first operation version identifier is generated based on the initial operation version identifier, and the first operation version identifier is returned to the client.

[0332] In one possible implementation, processor 1201 is also used to perform the following operations:

[0333] Obtain the target client identifier for sending the file operation request, and verify the client based on the target client identifier; wherein the verification includes at least one of the following: permission verification, security verification, and legality verification;

[0334] If the client verification passes, the step of retrieving the initial operation version identifier of the variable data block from the first data storage structure of the first file is triggered.

[0335] In one possible implementation, the processor 1201 updates the first data storage structure based on the description information of the first data block to obtain a second data storage structure with the first file updated, which is used to perform the following operations:

[0336] Parse the description information of the first data block to obtain the target length corresponding to the first data block;

[0337] If the target length is less than the preset length, the second storage area in the first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure of the first file.

[0338] In one possible implementation, processor 1201 is also used to perform the following operations:

[0339] If the target length is equal to the preset length, then the first data block is treated as a fixed data block, and the first operation version identifier of the first data block is added to the first storage area of ​​the first data storage structure to obtain the second data storage structure of the first file; or,

[0340] If the target length is greater than the preset length, the first data block is divided into at least one fixed data block and a new variable data block; the first operation version identifier of each fixed data block is added to the first storage area of ​​the first data storage structure, and the second storage area in the first data storage structure is updated based on the description information of the new variable data block to obtain the second data storage structure of the first file.

[0341] This application provides a file processing solution based on a cloud storage system. On one hand, this solution allows the metadata management server to assign an open version (operation version identifier) ​​to the client each time it opens a file. Subsequently, the client can upload (or delete) any number of data blocks without frequently requesting the metadata management server, thus improving file processing efficiency. On the other hand, during the management of file metadata, the metadata management server constructs an interval set data storage structure for the file using dual-version data blocks. This data storage structure can manage a large number of data blocks in a file with relatively little memory, thereby improving the efficiency of file data block management.

[0342] According to one aspect of this application, embodiments of this application also provide a computer-readable storage medium storing a computer program. When a processor executes the computer program, it can perform the methods described in the preceding embodiments; therefore, further details will not be repeated here. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the computer program can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0343] According to one aspect of this application, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, enabling the computer device to perform the methods described in the foregoing embodiments; therefore, further details will not be repeated here. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0344] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product, which includes one or more computer programs. When the computer program is loaded and executed on a computer device, it generates, in whole or in part, the processes or functions described in the embodiments of this application; the computer device can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium; the computer program can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible to the computer device or a data processing device such as a server or data center that integrates one or more available media; wherein, the available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0345] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A file processing method, characterized in that, The method includes: A file operation request for a first file is sent to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system. Receive the first operation version identifier of the first file, which is returned by the metadata management server in response to the file operation request; Based on the first operation version identifier, file operation processing is performed on the first file in the cloud storage system; the file operation processing is used to instruct at least one data processing operation to be performed on the data blocks of the first file. During the execution of the file operation processing, a first upload version identifier is generated for the first data block in the first file. The first data block refers to the data block in the first file that changes with any data processing operation. The first operation version identifier and the first upload version identifier are sent as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

2. The method as described in claim 1, characterized in that, The first file includes variable data blocks, which support the writing of data to the first file; the data processing operation includes a data writing operation to the first file. During the execution of the file operation processing, generating a first upload version identifier for the first data block in the first file includes: The variable data block from the first file is loaded into the client's memory from the cloud storage system; the variable data block has an initial upload version identifier i, where i is an integer; Obtain the first data corresponding to the data write operation, and write the first data into the variable data block in the memory to generate the first data block; Based on the initial upload version identifier i of the variable data block, a first upload version identifier i+1 is generated for the first data block; Specifically, when the cache interface of the cloud storage system is detected to be invoked, the first data block in memory is persisted to the cloud storage system.

3. The method as described in claim 2, characterized in that, The method further includes: The first data block is loaded from the cloud storage system into the client's memory, and the second data to be written corresponding to the data write operation is obtained; Write the second data into the first data block in the memory to generate the second data block, and generate the second upload version identifier i+2 for the second data block based on the first upload version identifier i+1 of the first data block; Until the data length of the variable data block after the data write operation is performed reaches the preset length, a fixed data block of the preset length is submitted to the cloud storage system; The variable data block after deleting the preset length is used as the new variable data block in the first file, and a new upload version identifier is set for the new variable data block.

4. The method as described in claim 3, characterized in that, The data processing operation further includes a data deletion operation on the first file; the file operation processing on the first file in the cloud storage system further includes: The second data block is loaded from the cloud storage system into the client's memory; the second data block has a second upload version identifier i+2; Obtain the data to be deleted corresponding to the data deletion operation, and delete the data to be deleted from the second data block in the memory to generate a third data block; Based on the second upload version identifier i+2 of the second data block, a third upload version identifier i+3 is generated for the third data block; The first data block, the second data block, and the third data block all have the first operation version identifier.

5. The method according to any one of claims 1-4, characterized in that, Any data block in the first file is stored as an object in the cloud storage system, and each object corresponds to an object identifier; the file operation processing is further used to indicate a data read operation on the first file; the method further includes: Obtain the metadata of the first file sent by the metadata management server. The metadata of the first file records the description information of each data block in the first file. The description information of any one of the data blocks includes at least one of the following: operation version identifier, upload version identifier, block sequence number, and length. Parse the metadata of the first file to obtain the target description information of the data block to be read corresponding to the data read operation; A target object identifier is generated based on the target description information, and the target data block indicated by the target object identifier is read from the first file in the cloud storage system based on the target object identifier.

6. The method as described in claim 1, characterized in that, The first operation version identifier of the first file is represented by j, where j is an integer; the method further includes: In response to a call to the file closing interface of the cloud storage system, the first data block of the first file is persisted to the cloud storage system; Send a file operation request for the first file to the metadata management server again, and receive the second operation version identifier of the first file returned by the metadata management server; wherein the second operation version identifier is j+1.

7. A file processing method, characterized in that, The method includes: The system receives a file operation request for a first file sent by a client. The first file is stored in a cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system. In response to the file operation request, the first operation version identifier of the first file is sent to the client; The system receives description information of a first data block sent by the client. The description information includes a first operation version identifier and a first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to a data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file. The metadata of the first file is updated based on the description information of the first data block.

8. The method as described in claim 7, characterized in that, The step of updating the metadata of the first file based on the description information includes: Obtain the metadata of the first file, and construct a first data storage structure for the first file based on the metadata of the first file; The first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure after the first file is updated.

9. The method as described in claim 8, characterized in that, The first file includes N fixed data blocks and one variable data block. The data length of each fixed data block is equal to a preset length, and the data length of the variable data block is less than the preset length. N is a positive integer. The metadata of the first file records the description information of each data block in the first file. The step of constructing a first data storage structure for the first file based on the metadata of the first file includes: Obtain first description information for each fixed data block from the metadata of the first file. The first description information includes: block number and operation version identifier. According to the first description information of the N fixed data blocks, K interval trees are constructed for the N fixed data blocks, and the K interval trees are stored in the first storage area in order; wherein, each interval tree is used to record at least one fixed data block with the same operation version identifier, and K is a positive integer; The second description information of the variable data block is obtained from the metadata of the first file and stored in the second storage area; the second description information includes at least one of the following: initial operation version identifier, initial upload version identifier, and initial length; The first storage area and the second storage area are combined to form the first data storage structure of the first file.

10. The method as described in claim 9, characterized in that, Sending the first operation version identifier of the first file to the client includes: If the file operation request is used to indicate that the first file is to be created, then the first operation version identifier is set to the initial value; If the file operation request is used to instruct the first file to perform file operation processing, then the initial operation version identifier of the variable data block is obtained from the first data storage structure of the first file; A first operation version identifier is generated based on the initial operation version identifier, and the first operation version identifier is returned to the client.

11. The method as described in claim 10, characterized in that, The method further includes: Obtain the target client identifier that sent the file operation request, and verify the client based on the target client identifier; wherein, the verification includes at least one of: permission verification, security verification, and legality verification; If the client verification passes, the step of obtaining the initial operation version identifier of the variable data block from the first data storage structure of the first file is triggered.

12. The method according to any one of claims 8-11, characterized in that, The step of updating the first data storage structure based on the description information of the first data block to obtain the updated second data storage structure for the first file includes: Parse the description information of the first data block to obtain the target length corresponding to the first data block; If the target length is less than the preset length, then the second storage area in the first data storage structure is updated based on the description information of the first data block to obtain the second data storage structure of the first file.

13. The method as described in claim 12, characterized in that, The method further includes: If the target length is equal to the preset length, then the first data block is treated as a fixed data block, and the first operation version identifier of the first data block is added to the first storage area of ​​the first data storage structure to obtain the second data storage structure of the first file; or, If the target length is greater than the preset length, the first data block is divided into at least one fixed data block and a new variable data block; the first operation version identifier of each fixed data block is added to the first storage area of ​​the first data storage structure, and the second storage area in the first data storage structure is updated based on the description information of the new variable data block to obtain the second data storage structure of the first file.

14. A document processing apparatus, characterized in that, The device includes: The sending unit is used to send a file operation request for a first file to the metadata management server. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system. A receiving unit is configured to receive a first operation version identifier of the first file, wherein the first operation version identifier of the first file is returned by the metadata management server in response to the file operation request; The processing unit is configured to perform file operation processing on the first file in the cloud storage system based on the first operation version identifier; the file operation processing is configured to instruct at least one data processing operation to be performed on the data blocks of the first file; The processing unit is further configured to generate a first upload version identifier for a first data block in the first file during the execution of the file operation processing, wherein the first data block refers to a data block in the first file that changes with any data processing operation; The sending unit is configured to send the first operation version identifier and the first upload version identifier as description information of the first data block to the metadata management server, so that the metadata management server updates the metadata in the first file based on the description information.

15. A document processing device, characterized in that, The device includes: The receiving unit is used to receive a file operation request for a first file sent by a client. The first file is stored in the cloud storage system in the form of data blocks. The metadata management server is used to manage the metadata of the files stored in the cloud storage system. The sending unit is configured to send a first operation version identifier of the first file to the client in response to the file operation request; The receiving unit is further configured to receive description information of the first data block sent by the client. The description information includes the first operation version identifier and the first upload version identifier of the first data block in the first file. The first upload version identifier is generated by the client for the first data block in the first file during the file operation processing performed on the first file in the cloud storage system. The first data block refers to the data block in the first file that changes with any data processing operation. The file operation processing is used to indicate that at least one data processing operation is performed on the data block of the first file. The processing unit is used to update the metadata of the first file based on the description information.

16. A computer device, characterized in that, include: Memory and processor; The memory stores one or more computer programs; A processor for loading one or more computer programs to implement the method as described in any one of claims 1-6 or 7-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1-6 or 7-13.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the method as described in any one of claims 1-6 or 7-13.