A data processing method, device, apparatus, and computer-readable storage medium

Abstract

The application provides a data processing method and device, equipment and a computer readable storage medium, relates to cloud technology and blockchain technology, and the method comprises the following steps: receiving a processing operation for a target file, and encoding a target directory where the target file is located to obtain a directory index parameter; finding out directory metadata describing the attributes of the target directory from the directory mapping data through the directory index parameter; finding out target metadata describing the attributes of the target file from the directory mapping data according to file index parameters constructed according to the directory metadata and the file name of the target file; and implementing the processing operation on the target file based on the operation on the file index parameters and the target metadata. Through the application, the overall efficiency of data processing can be improved.

Description

Technical Field

[0001] This application relates to distributed file system technology in cloud technology, and more particularly to a data processing method, apparatus, device and computer-readable storage medium. Background Technology

[0002] Namespaces are responsible for managing the directory tree structure and file metadata of a distributed file system. When a namespace processes a directory tree, it essentially involves processing data within the directory tree or its files. However, in related technologies, due to the long time consumption of data queries and redundancy in index parameters, processing data on files within the directory tree takes a considerable amount of time, resulting in overall low efficiency in data processing. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and computer-readable storage medium that can improve the overall efficiency of data processing.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, including:

[0006] Receive processing operations for the target file and encode the target directory where the target file is located to obtain directory index parameters;

[0007] The directory index parameters are used to retrieve directory metadata describing the attributes of the target directory from the directory mapping data; the directory mapping data stores the mapping relationship between the target index parameters and the directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file.

[0008] Based on the file index parameters constructed from the directory metadata and the file name of the target file, the target metadata describing the attributes of the target file is retrieved from the directory mapping data;

[0009] Based on the operations on the file index parameters and the target metadata, the processing operations on the target file are implemented, and the processing operations include at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation.

[0010] This application provides a data processing apparatus, including:

[0011] The operation receiving module is used to receive processing operations on the target file;

[0012] The information encoding module is used to encode the target directory where the target file is located to obtain directory index parameters;

[0013] The data lookup module is used to retrieve directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameters; the directory mapping data stores the mapping relationship between the directory index parameters and the directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file; the file index parameters, constructed based on the directory metadata and the file name of the target file, are used to retrieve the target metadata describing the attributes of the target file from the directory mapping data;

[0014] The file processing module is used to implement the processing operation on the target file based on the operation on the file index parameters and the target metadata. The processing operation includes at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation.

[0015] In some embodiments of this application, the data lookup module is further configured to parse the directory identifier corresponding to the target directory from the directory metadata; generate the file index parameter using the directory identifier and the file name of the target file; and look up the target metadata describing the attributes of the target file from the directory mapping data based on the file index parameter.

[0016] In some embodiments of this application, the processing operation includes: the move operation; the file processing module is further configured to encode the move directory corresponding to the move operation to obtain a move index parameter; the move directory represents the directory to which the target file is moved; based on the move index parameter, filter the move metadata corresponding to the move directory from the directory mapping data; parse the move identifier of the move directory from the move metadata, and use the move identifier and the file name to concatenate the latest index parameter; establish a mapping relationship between the latest index parameter and the target metadata, and write the latest index parameter into the directory mapping data; delete the file index parameter from the directory mapping data to realize the move operation on the target file.

[0017] In some embodiments of this application, the processing operation includes: the renaming operation; the file processing module is further configured to obtain the renaming name corresponding to the renaming operation; concatenate the renaming index parameter according to the directory identifier and the renaming name; establish a mapping relationship between the renaming index parameter and the target metadata, and write the renaming index parameter into the directory mapping data; delete the file index parameter from the directory mapping data, thereby realizing the renaming operation for the target file.

[0018] In some embodiments of this application, the directory mapping data further includes: a mapping relationship between metadata index parameters and data block metadata; the processing operation includes: the metadata reading operation; the file processing module is further configured to parse the file identifier of the target file from the target metadata; use the file identifier and the block identifier of the data block corresponding to the metadata reading operation to concatenate the metadata index parameters; wherein, the data block is obtained by splitting the target file, and the block identifier is used to uniquely identify the data block; based on the metadata index parameters, the data block metadata corresponding to the data block is found from the directory mapping data; the data block metadata is parsed to obtain the file metadata of the target file, thereby realizing the metadata reading operation for the target file.

[0019] In some embodiments of this application, the file processing module is further configured to update the data block metadata to obtain updated data block metadata; establish a mapping relationship between the metadata lookup parameters and the updated data block metadata, and write the updated data block metadata into the directory mapping data; and delete the data block metadata from the directory mapping data.

[0020] In some embodiments of this application, the processing operation includes: the deletion operation; the file processing module is further configured to delete the target metadata and the file index parameters from the directory mapping data, thereby implementing the deletion operation for the target file.

[0021] In some embodiments of this application, the data processing apparatus further includes: a file creation module;

[0022] The file creation module is configured to: create the target file in the target directory; generate file index parameters corresponding to the target file using a unique directory identifier assigned to the target directory and the file name of the target file; determine the attribute information of the target file as the target metadata corresponding to the target file; wherein the attribute information of the target file includes at least one or more of the following: a unique file identifier assigned to the target file and the number of data blocks in the target file; establish a mapping relationship between the file index parameters and the target metadata; and write the file index parameters and the target metadata into the directory mapping data.

[0023] In some embodiments of this application, the data processing apparatus further includes: a directory processing module;

[0024] The directory processing module is used to read the attribute information corresponding to the target directory from the directory metadata. The attribute information corresponding to the target directory includes at least one or more of the following: directory identifier and directory creation time. The directory identifier is used to uniquely identify the target directory.

[0025] In some embodiments of this application, the directory processing module is further configured to construct a file query prefix based on the directory identifier in the attribute information corresponding to the target directory; find candidate index parameters with the file query prefix and candidate file metadata corresponding to the candidate search parameters from the directory mapping data; parse the candidate file metadata to obtain the names of each file under the target directory and the number of data blocks of each file.

[0026] In some embodiments of this application, the directory processing module is further configured to create the target directory and generate the directory identifier for the target directory; encode the target directory to obtain the directory index parameter; determine the directory identifier and the creation time of the target directory as the directory metadata; establish a mapping relationship between the directory index parameter and the directory metadata; and write the directory index parameter and the directory metadata into the directory mapping data.

[0027] In some embodiments of this application, the directory processing module is further configured to obtain a replacement directory and encode the replacement directory to obtain a directory index prefix; wherein, the replacement directory refers to the directory to which the directory under the target directory is to be moved; from the directory mapping data, find the matching directory index parameter with the directory index parameter as a prefix, and the matching directory metadata corresponding to the matching directory index parameter; replace the prefix of the matching directory index parameter with the directory index prefix to obtain the replaced index parameter, and establish a mapping relationship between the replaced index parameter and the matching directory metadata; write the replaced index parameter into the directory mapping data, and delete the matching directory index parameter from the directory mapping data.

[0028] This application provides a data processing device, including:

[0029] Memory, used to store executable data processing instructions;

[0030] The processor, when executing executable data processing instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing executable data processing instructions, which, when executed by a processor, implement the data processing method provided in this application.

[0032] The embodiments of this application have the following beneficial effects: The data processing device receives a processing operation for a target file, encodes the target directory corresponding to the target file to obtain directory index parameters, and then uses the directory index parameters to perform a first search in the directory mapping data to obtain directory metadata describing the attributes of the target directory. Then, based on the file index parameters obtained from the directory metadata and file name, a second search is performed in the directory mapping data. Thus, only two searches are needed to obtain the target metadata describing the attributes of the target file, resulting in a shorter query time. Furthermore, since the file index parameters are based on the directory metadata and file name, they are not affected by the name of the target directory. Even if the target file is moved, there is no need to modify the index parameters of the target file, reducing the number of parameters that need to be modified when processing the file. Therefore, when the data processing device processes files in the directory tree, the search efficiency is high, the number of parameters modified is less, thereby reducing the data processing time and ultimately improving the overall efficiency of data processing. Attached Figure Description

[0033] Figure 1 This is a diagram of the directory tree;

[0034] Figure 2AThis is an optional architecture diagram of the data processing system provided in the embodiments of this application;

[0035] Figure 2B This is another optional architecture diagram of the data processing system provided in the embodiments of this application;

[0036] Figure 3 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application;

[0037] Figure 4 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ;

[0038] Figure 5 This is a schematic diagram illustrating the parameters constituting the catalog index provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram illustrating the search for target metadata provided in an embodiment of this application;

[0040] Figure 7 This is a second optional flowchart illustrating the data processing method provided in the embodiments of this application;

[0041] Figure 8 This is a schematic diagram illustrating the process of renaming a target file according to an embodiment of this application;

[0042] Figure 9 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3 ;

[0043] Figure 10 This is a schematic diagram illustrating the process of querying candidate file metadata provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of a system framework for processing directories and files in a directory tree, provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0050] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0051] 1) Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computing, storage, processing, and sharing of data.

[0052] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0053] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0054] 2) A distributed file system refers to a file system whose physical storage resources are not necessarily directly connected to local nodes, but are connected to nodes through a computer network, or formed by combining several different logical disk partitions or volume labels to create a complete hierarchical file system. A distributed file system provides a logical tree-like file system result for resources at any location on the network, thus making it easier for users to access shared files distributed across the network.

[0055] 3) Namespaces refer to the way file directories are organized in a distributed file system. They are an important part of the file system and provide users with a visual and understandable view of the file system, thereby solving or reducing the semantic gap between humans and computers in data storage.

[0056] 4) Metadata, primarily used to describe data attributes, indicates storage location, historical data, resource lookup, file records, and other functions. Metadata is a type of electronic catalog; users can first view its metadata when using data to obtain the information they need. Metadata is also data and can be stored and retrieved using similar methods.

[0057] 5) Index parameters are parameters used to look up metadata. In this application, the mapping relationship between index parameters and metadata can be implemented using "key-value pairs," that is, using index parameters as keys and metadata as values ​​to establish a mapping relationship.

[0058] 6) Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. It consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks record the data submitted by the nodes in the blockchain system.

[0059] Namespaces are responsible for managing the directory tree structure and file metadata of a distributed file system. When a namespace processes a directory tree, it essentially performs data processing on the files within that tree. Currently, there are two main ways to construct directory trees: one is based on building the directory tree in memory, and the other is based on the Log-Structured Merge-Tree (LSM-Tree) storage engine.

[0060] However, building a directory tree in memory consumes a lot of memory, the number of directory nodes depends on the memory capacity, and once the process restarts, it needs to read data from the disk and rebuild the directory tree in memory. Rebuilding the directory tree takes a long time and requires a long time to restore the service.

[0061] Building a directory tree based on LSM-Tree allows for persistent data storage with unlimited capacity. Even if the process restarts, there is no overhead associated with rebuilding directories, thus resolving issues related to data processing of directories and files within an in-memory directory tree. However, the applicant discovered shortcomings in file data processing within this LSM-Tree-based directory tree.

[0062] Specifically, there are two different implementation methods when building a directory tree based on LSM-Tree, which are referred to here as the first solution based on LSM-Tree and the second solution based on LSM-Tree.

[0063] Below, in conjunction with Figure 1 The first and second schemes based on LSM-Tree are described separately. Figure 1 This is a diagram of the directory tree. Under / (root node) 1-1, there are user nodes home1-2 and system nodes usr1-3. Under the user node home1-2, there are files a.dat1-4 and b.dat1-5 stored by the user.

[0064] In the first scheme based on LSM-Tree, the directory tree is stored using file paths as keys and file metadata as values. Table 1 shows an example of the key-value pair structure of the first scheme based on LSM-Tree.

[0065] Table 1

[0066]

[0067] Directory metadata is data used to describe the attributes of a directory, while file metadata is data used to describe the attributes of a file.

[0068] As shown in Table 1, in the first scheme based on LSM-Tree, if a user wants to obtain the metadata of the file a.dat, the key of the file needs to be constructed first, that is, / home / a.dat needs to be constructed first, so as to traverse all key-value pairs; when the user renames the / home directory, all keys with " / home" as the prefix need to be updated.

[0069] Therefore, although the first approach based on LSM-Tree has a higher search efficiency, the fact that each node's key contains path information results in a large amount of duplicate and redundant data. This means that when moving a file (which can be seen as renaming the directory where the file is located), the file's key needs to be modified. When there are many files, this process will undoubtedly increase the time consumption, and thus the overall data processing efficiency is still low.

[0070] In the second scheme based on LSM-Tree, metadata is created for each directory node, and a unique directory identifier (entry_id) is assigned. Then, the metadata of the directory or file is entered using the parent directory's entry_id + directory / filename as the key. Table 2 shows an example of the key-value pair structure of the second scheme based on LSM-Tree.

[0071] Table 2

[0072]

[0073] As shown in Table 2, in the second scheme based on LSM-Tree, searching for files or directories requires searching according to the directory hierarchy. That is, first, the entry_id of the parent directory is searched; then, using the parent directory's entry_id + directory name as the key, the corresponding entry_id is searched, and so on, layer by layer, until the search is complete. For example, if you want to search for... Figure 1 When renaming a.dat, the process first searches for the entry_id in the root directory, which is 1. Then, using 1 / home as the key, it searches for the entry_id in the / home directory, which is 2. Finally, using 2 / a.dat as the key, it finds the file metadata of a.dat. When renaming a directory, only the corresponding key needs to be modified; the entry_id remains unchanged. For example, to change / home to / test, simply change the key 1 / home to 1 / test.

[0074] Therefore, although the second scheme based on LSM-Tree has less redundant data and is more efficient in file movement and directory renaming operations, its query efficiency is reduced, and the number of searches equals the directory depth, resulting in a relatively low overall data processing efficiency.

[0075] In summary, due to the long data query time and redundancy in index parameters, data processing of files in the directory tree takes a long time, resulting in low overall data processing efficiency.

[0076] This application provides a data processing method, apparatus, device, and computer-readable storage medium, which can improve the overall efficiency of data processing. The following describes exemplary applications of the data processing device provided in this application. This data processing device can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server, or as a cluster of devices composed of servers and terminals. The following describes exemplary applications when the data processing device is implemented as a server.

[0077] See Figure 2A , Figure 2A This is an optional architecture diagram of the data processing system provided in this application embodiment. To support a data processing application, in the data processing system 100, the terminal 400 connects to the server 200 through a network 300, which can be a wide area network, a local area network, or a combination of both.

[0078] Terminal 400 is used to obtain user processing operations on target files through graphical interface 410 and send the processing operations to server 200 through network 300.

[0079] Server 200 is used to receive processing operations on a target file, encode the target directory where the target file is located to obtain directory index parameters, and use the directory index parameters to find directory metadata describing the attributes of the target directory from the directory mapping data. The directory mapping data stores the mapping relationship between the directory index parameters and directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file. Based on the file metadata and the filename of the target file, the file index parameters are constructed to find the target metadata describing the attributes of the target file from the directory mapping data. Based on the operations on the file index parameters and target metadata, the server 200 performs processing operations on the target file, including at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation. After data processing, server 200 is also used to return a processing completion prompt to terminal 400 via network 300.

[0080] Terminal 400 is also used to display a notification that processing is complete on the graphical interface 410.

[0081] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, in-vehicle terminal, smart TV, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0082] This application's embodiments can also be implemented using blockchain technology; see [link to relevant documentation]. Figure 2B , Figure 2B This is another optional architecture diagram of the data processing system provided in the embodiments of this application. Figure 2BIn the data processing system 100, the server uploads the operation type (i.e., move, rename, etc.), file index parameters, and target metadata to the blockchain to ensure that all processing operations on the target file can be recorded and that the records cannot be tampered with.

[0083] In some embodiments, server 200 can join blockchain network 700 and become a node within it. The type of blockchain network 700 is flexible and diverse; for example, it can be any of a public blockchain, private blockchain, or consortium blockchain. Taking a public blockchain as an example, any electronic device of any business entity can access blockchain network 700 without authorization to act as a consensus node in blockchain network 700. For example, terminal 600 can be mapped to consensus node 700-1 in blockchain network 700 to achieve consensus when uploading metadata to the blockchain.

[0084] Taking blockchain network 700 as a consortium blockchain as an example, terminal 600 can become a node after obtaining authorization and accessing blockchain network 700. Server 200 receives processing operations for the target file and encodes the target directory where the target file is located to obtain directory index parameters; using the directory index parameters, it retrieves directory metadata describing the attributes of the target directory from the directory mapping data; based on the directory metadata and the file name of the target file, it constructs file index parameters and retrieves target metadata describing the attributes of the target file from the directory mapping data; based on the operations on the file index parameters and target metadata, it implements the processing operation on the target file, and simultaneously sends the operation type, file index parameters, and target metadata of the processing operation to blockchain network 700 for consensus. When consensus is reached, blockchain network 700 allocates a new block containing the operation type, file index parameters, and target metadata of the processing operation, writes the operation type, file index parameters, and target metadata of the processing operation into the new block, and adds the new block to the end of the blockchain.

[0085] See Figure 3 , Figure 3 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application. Figure 3 The data processing device 500 shown includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the data processing device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 540.

[0086] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0087] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0088] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0089] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0090] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0091] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0092] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0093] Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530.

[0094] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0095] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 3 A data processing device 555 stored in memory 550 is shown. It may be software in the form of programs and plug-ins, including the following software modules: operation receiving module 5551, information encoding module 5552, data searching module 5553, file processing module 5554, file creation module 5555, and directory processing module 5556. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0096] The functions of each module will be explained below.

[0097] In other embodiments, the data processing apparatus provided in this application can be implemented in hardware. As an example, the data processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0098] For example, an embodiment of this application provides a data processing device, including:

[0099] Memory, used to store executable data processing instructions;

[0100] The processor, when executing executable data processing instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0101] The data processing method provided in this application will be described below with reference to exemplary applications and implementations of the data processing device provided in the embodiments of this application. It should be noted that when the data processing device is implemented as a device cluster composed of servers and terminals, the data processing method provided in the embodiments of this application can be implemented with the aid of cloud technology.

[0102] See Figure 4 , Figure 4 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 4 The steps shown are explained.

[0103] S101: Receive the processing operation for the target file, and encode the target directory where the target file is located to obtain the directory index parameter.

[0104] This application embodiment is implemented in a scenario of data processing of files in a directory tree, such as moving files in the directory tree or reading metadata. The data processing device receives the processing operation for the target file in real time, extracts the path of the directory where the target file is located, i.e., obtains the target directory, and then encodes the target directory to obtain the directory index parameters for the metadata of the directory where the target file is located.

[0105] It should be noted that the target file is specified by the path information. The path information of the target file consists of the path of the directory where the target file is located, i.e. the target directory, and the file name of the target file. The data processing device can extract the complete target directory from the path information of the target file for encoding.

[0106] In some embodiments, the target file may be specified by path information entered by the user. For example, the user enters the path information of the file to be processed in the command line, and then the data processing device determines the target file to be processed based on the received path information, thereby realizing data processing according to the user's instructions. In other embodiments, the target file may be specified by the data processing device. For example, after a set time is reached, the data processing device automatically determines the target file to be processed (by random selection or by selecting files that the user operates on frequently, etc.) to automatically process the target file, reducing the user's operational burden during data processing.

[0107] Data processing equipment can encode the target directory and obtain directory index parameters by adding a fixed prefix to the target directory. For example, Figure 5 This is a schematic diagram illustrating the parameters constituting the catalog index provided in an embodiment of this application. See also... Figure 5When the target directory is / data / dir1 / sub_dir 5-1, the data processing device can add a fixed prefix 5-2 corresponding to the root directory " / " to the target directory, for example, 0, so "0 / data / dir1 / sub_dir" is the directory index parameter 5-3.

[0108] The data processing device can also directly convert each directory node in the target directory into numbers or characters, and then concatenate the numbers or characters corresponding to each directory node to complete the encoding of the target directory and obtain the directory index parameter. For example, when the target directory is " / data / a.file", the data processing device encodes " / data" as 1 and " / a.file" as 2, and then concatenates them. The resulting encoding result 12 is the directory index parameter.

[0109] The data processing device can also generate a unique string for the target directory and use that string as a directory index parameter. For example, the data processing device can convert " / data / a.file" into the string "daf" and use that string as a directory index parameter.

[0110] It is understood that the processing operations on the target file can be moving, deleting, renaming, reading metadata, etc., and this application does not limit them here.

[0111] Data processing devices can perform processing operations through user input via command line or user interface. For example, a user can enter the command string corresponding to a move operation in the command line or select the target file on the user interface and click the delete button to trigger a deletion operation.

[0112] The data processing device can also predict the processing operations required for the target file by analyzing the user's past operations on the target file. For example, if the user frequently reads the metadata of the target file, the data processing device can identify the current processing operation for the target file as a metadata reading operation, and automatically read the metadata of the target file and provide it to the user.

[0113] It should be noted that the target file refers to the file to be processed, which can be any file in the directory tree, and does not specifically refer to a particular file.

[0114] S102. Using the directory index parameters, retrieve the directory metadata describing the attributes of the target directory from the directory mapping data.

[0115] The directory tree illustrates the hierarchical relationship between directories and files. Directory mapping data stores the directory tree through a mapping relationship between index parameters and metadata. Therefore, directory mapping data stores the mapping relationship between directory index parameters and directory metadata, as well as the mapping relationship between file index data and the target metadata corresponding to the target file. After obtaining the directory index parameters, the data processing device can use this mapping relationship to find the metadata corresponding to the directory index parameters from the directory mapping data. The found metadata is the directory metadata, which describes the attributes of the target directory.

[0116] It should be noted that, in some embodiments, the mapping relationship between directory index parameters and directory metadata, as well as the mapping relationship between file index parameters and target metadata, can be represented using key-value pairs. That is, the directory mapping data is stored with directory index parameters and file index parameters as keys, and directory metadata and target metadata as values. In other embodiments, the mapping relationship between directory index parameters and directory metadata, as well as the mapping relationship between file index parameters and target metadata, can also be organized using a relational model. For example, the directory index parameters and their corresponding directory metadata can be stored in the same row of a data table. This application does not impose any limitations on this.

[0117] It is understandable that the attributes of the target directory described by the directory metadata may include the directory's creation time, the directory identifier that the data processing device assigned to the directory to uniquely represent the target directory when it was created, and so on.

[0118] S103. Based on the file index parameters constructed from the directory metadata and the file name of the target file, find the target metadata describing the attributes of the target file from the directory mapping data.

[0119] The data processing device initiates a second search of the directory mapping data based on the directory metadata and file name to obtain the target metadata corresponding to the target file. In some embodiments, the data processing device can parse the directory metadata and construct file index parameters based on the parsed directory identifier and file name to find the target metadata. In other embodiments, the data processing device can also construct file index parameters using the creation time and file name in the directory metadata, and then find the target metadata based on the file index parameters.

[0120] Understandably, the filename usually ends with the complete path information of the target file and has a certain format (e.g., using "." to distinguish the name from the format suffix). Therefore, the data processing device can extract the filename of the target file from its path information. For example, when the complete path information of the target file is / data / dir1 / sub_dir1 / a.dat, the filename of the target file is a.dat, where the suffix ".dat" indicates the format of the target file.

[0121] It should be noted that, in this embodiment of the application, on the one hand, the data processing device first uses the directory index parameters obtained by encoding to find the directory metadata corresponding to the target directory. At this time, a search has been performed. Then, the file index parameters are obtained by combining the directory metadata with the file name. After another search, the target metadata corresponding to the target file can be obtained. The search efficiency is high and stable (stable at two times). This will undoubtedly avoid the problem of dragging down the overall efficiency of data processing due to excessive search depth.

[0122] On the other hand, since the file index parameters are based on directory metadata and the target file, when constructing index parameters for the target metadata, it is not necessary to include the complete directory path of the target file. That is, it does not involve the specific target directory and does not contain redundant data. Therefore, even if the data processing device needs to move the target file or rename the target directory where the target file is located, only the directory index parameters corresponding to the target directory need to be modified. It will not involve modifying the file index parameters of the target file under the target directory. Thus, the number of parameters that need to be modified when processing the target file will be reduced, and the problem of needing to modify the index parameters when moving the file can be avoided.

[0123] It is understood that the attributes of the target file described by the target metadata may include the target file's creation time, file identifier, size, number of data blocks, etc., which are not limited in this application.

[0124] S104. Based on the operations on file index parameters and target metadata, implement the processing operations on the target file.

[0125] Since the directory tree is stored as a mapping between index parameters and metadata, processing a file in the directory tree essentially involves processing the file's corresponding index parameters and metadata. This requires manipulating both the index parameters and metadata. After obtaining the target metadata through two searches, the data processing device determines the appropriate operation and then manipulates the file's index parameters and target metadata to perform the processing operation on the target file. The processing operation includes at least one or more of the following: move, delete, rename, and metadata read.

[0126] In some embodiments, the data processing device can perform processing operations on the target file by changing the mapping relationship between file index parameters and target metadata. In other embodiments, the data processing device can also perform processing operations on the target file by updating or deleting file index parameters and / or target metadata.

[0127] In this embodiment, the data processing device receives a processing operation for a target file, encodes the target directory corresponding to the target file to obtain directory index parameters, and then performs a first lookup in the directory mapping data using the directory index parameters to obtain directory metadata describing the attributes of the target directory. Then, based on the file index parameters obtained from the directory metadata and file name, a second lookup is performed in the directory mapping data. Thus, only two lookups are needed to obtain the target metadata describing the attributes of the target file, resulting in a shorter query time. Furthermore, since the file index parameters are based on directory metadata and file name, they are not affected by the name of the target directory. Even if the target file is moved, there is no need to modify the index parameters of the target file, reducing the number of parameters that need to be modified during file processing. Therefore, when processing files in the directory tree, the data processing device has high search efficiency and requires fewer parameter modifications, thereby reducing data processing time and ultimately improving the overall efficiency of data processing.

[0128] In some embodiments of this application, the specific implementation process of retrieving target metadata describing the attributes of the target file from the directory mapping data based on the file index parameters constructed according to the directory metadata and the file name of the target file, i.e., S103, may include: S1031-S1033, as follows:

[0129] S1031. Parse the directory identifier corresponding to the target directory from the directory metadata.

[0130] The directory metadata stores various attribute information of the target directory. The data processing device parses the directory metadata and extracts the directory identifier corresponding to the target directory.

[0131] It should be noted that the directory identifier corresponding to the target directory is a globally unique identifier assigned to the target directory by the data processing device when the target directory is created. The directory identifier can be a number or a string, and this application does not impose any restrictions on it.

[0132] S1032. Generate file index parameters using the directory identifier and the file name of the target file.

[0133] S1033. Based on the file index parameters, retrieve the target metadata describing the attributes of the target file from the directory mapping data.

[0134] The data processing device uses the directory identifier and the file name of the target file to generate a file index parameter for finding the metadata corresponding to the target file. Then, it uses the file index metadata to find the target metadata from the directory mapping data.

[0135] Data processing devices can obtain file index parameters by concatenating directory identifiers and filenames. For example, when the directory identifier of / data / dir1 / sub_dir1 is 1, the data processing device can use 1+a.dat as the file index parameter. Alternatively, the data processing device can encode the directory identifier and filename, using the encoded result as the file index parameter. For example, encoding 1 and a.dat using an encoding algorithm yields the encoded result 1a0021, which is the file index parameter.

[0136] For example, Figure 6 This is a schematic diagram illustrating the process of finding target metadata provided in an embodiment of this application. See also... Figure 6 The data processing device searches for directory metadata 6-3 during the first search 6-2, based on the target directory 6-1. Then, based on the directory identifier 6-31 and file name 6-4 in the directory metadata 6-3, it performs a second search 6-5 to obtain the target metadata 6-6. Thus, it can be seen that in this embodiment, the number of searches is 2 for any file.

[0137] In this embodiment, the data processing device parses the directory identifier from the directory metadata, and then generates file index parameters based on the directory identifier and file name to locate the target metadata. Thus, the data processing device only needs two steps to obtain the metadata of any file, and the file index parameters do not contain redundant data such as the target directory. Even if the target directory containing the target file is modified, the file index parameters do not need to be modified.

[0138] See Figure 7 , Figure 7This is a second optional flowchart illustrating the data processing method provided in this application. In some embodiments of this application, the processing operation includes a move operation. In this case, based on the operations on the file index parameters and target metadata, the processing operation on the target file is implemented. That is, the specific implementation process of S104 may include: S1041-S1045, as follows:

[0139] S1041. Encode the moving directory corresponding to the moving operation to obtain the moving index parameter.

[0140] A move operation requires a corresponding move directory to function. The move directory represents the directory to which the target file will be moved; that is, during a move operation, the target directory is the original directory of the target file, and the move directory is the new directory. The data processing device encodes the move directory in the same way as the target directory, and the resulting encoding is the move index parameter.

[0141] In some embodiments, the move directory may be defined when the data processing device receives the move operation. Thus, the data processing device first defines the move directory and then encodes the target directory, thereby obtaining the target metadata through two lookups, and then encodes the move directory to obtain the move index parameters.

[0142] In other embodiments, the moving directory may be determined after the data processing device obtains the target metadata, that is, the data processing device determines the moving directory before it needs to operate on the target metadata and file index parameters, and then begins to encode the moving directory.

[0143] It is understandable that the move directory can be a directory specified by the user, or a directory created and specified by the data processing device itself. When the move directory is created, the data processing device assigns it a unique identifier, which is the move identifier.

[0144] S1042. Based on the moving index parameters, filter out the latest metadata corresponding to the moving directory from the directory mapping data.

[0145] S1043. Parse the mobile identifier of the mobile directory from the mobile metadata, and use the mobile identifier and file name to construct the latest index parameters.

[0146] After obtaining the move index parameters, the data processing device searches for the corresponding metadata in the directory mapping data. The obtained metadata is the metadata of the moved directory, i.e., the move metadata. The data processing device then parses the move metadata to obtain a move identifier that can uniquely represent the moved directory. Then, it concatenates the move identifier with the filename of the target file to obtain the latest index parameters of the target metadata.

[0147] It is understood that the data processing device can use preset characters to concatenate the movement identifier and the file name, or it can directly concatenate the movement identifier and the file name; this application does not impose any limitation on this. When concatenating, the data processing device can concatenate in the order of movement identifier first and file name last, or it can concatenate in the order of file name first and movement identifier last; this application does not impose any limitation on this.

[0148] S1044. Establish the mapping relationship between the latest index parameters and target metadata, and write the latest search parameters into the directory mapping data.

[0149] S1045. Delete the file index parameter from the directory mapping data to perform the move operation on the target file.

[0150] The data processing device uses the latest search parameters as the key and the target metadata as the value to establish a mapping relationship between the latest index parameters and the target metadata, and adds the latest index parameters to the directory mapping data. Simultaneously, to ensure that the target metadata can only be found using the latest index parameters, the data processing device deletes the original file index parameters from the target mapping data, replacing the mapping relationship between the latest index parameters and the target metadata. This establishes a connection between the target file and the moving directory, enabling the movement of the target file.

[0151] In this embodiment, the data processing device can first encode the moving directory corresponding to the moving operation to obtain the moving index parameters, then index the moving metadata, and then parse the moving identifier from the moving metadata to reconstruct the latest index parameters in combination with the file name. The new mapping relationship between the latest index parameters and the target metadata is used to replace the old mapping relationship between the file index parameters and the target metadata, thereby realizing the moving operation of the target file.

[0152] In some embodiments of this application, the processing operation includes a renaming operation, thereby implementing the processing operation on the target file based on the operation on the file index parameters and target metadata. That is, the specific implementation of S104 may include: S1046-S1049, as follows:

[0153] S1046. Obtain the rename name corresponding to the rename operation.

[0154] The renaming operation requires changing the filename of the target file. Therefore, in this embodiment, the data processing device first obtains the rename name corresponding to the renaming operation, that is, obtains the new name of the target file. It is understood that the rename name can be specified by the user or automatically generated by the data processing device, for example, using a randomly generated string as the rename name.

[0155] S1047. Construct the renaming index parameters based on the directory identifier and the rename name.

[0156] The data processing device concatenates the original directory identifier of the target file with the renamed name; the resulting concatenation is the renaming index parameter. It can be understood that the data processing device can use preset characters to concatenate the directory identifier and the renamed name, or it can directly concatenate them to achieve the concatenation. The order of the directory identifier and the renamed name can be set according to the actual situation.

[0157] S1048. Establish the mapping relationship between the rename index parameters and the target metadata, and write the rename index parameters into the directory mapping data.

[0158] S1049. Delete the file index parameter from the directory mapping data to perform a renaming operation on the target file.

[0159] The data processing device uses the rename index parameter as the key and the target metadata as the value, establishing a key-value pair as the mapping relationship between the rename index parameter and the target metadata. It then writes the rename index parameter into the directory mapping data and deletes the file index parameter, thereby completing the renaming operation.

[0160] For example, Figure 8 This is a schematic diagram illustrating the process of renaming a target file according to an embodiment of this application. The target file is named a.dat8-1, the directory identifier is X8-2, and the renamed name is b.dat8-3. In the directory mapping data, the original file index parameter X+a.dat8-4 and the target metadata 8-5 are originally stored (dashed line). The data processing device concatenates the directory identifier X8-1 and b.dat8-3 to obtain the renaming index parameter X+b.dat8-6, establishes the mapping between X+b.dat8-6 and the target metadata 8-5, deletes the original file index parameter X+a.dat8-4, thereby deleting the mapping between X+a.dat8-4 and the target metadata 8-5, thus completing the renaming of the target file.

[0161] In this embodiment, the data processing device generates renaming index parameters using the directory identifier and the rename name, and replaces the mapping relationship between the file index parameters and the target metadata by using the established mapping relationship between the renaming index parameters and the target metadata, thereby realizing the renaming operation of the target file.

[0162] In some embodiments of this application, the directory mapping data further includes: a mapping relationship between metadata index parameters and data block metadata; the processing operation includes: a metadata reading operation, thereby, based on the operation on the file index parameters and the target metadata, the processing operation on the target file is realized, that is, the specific implementation process of S104 may include: S10410-S10413, as follows:

[0163] S10410. Parse the file identifier of the target file from the target metadata.

[0164] S10411. Using the file identifier and the block identifier of the data block corresponding to the metadata read operation, concatenate them into metadata index parameters.

[0165] The target metadata stores the attribute information of the target file. The data processing device parses the file identifier, which can uniquely identify the target file, from the target metadata. Then, the data processing device concatenates the file identifier and the block identifier of the data block corresponding to the metadata read operation to obtain the metadata index parameters.

[0166] Data blocks are obtained by splitting the target file; this splitting is done when the target file is created. The reason the data processing device splits the target file is that when the target file is large, the operational cost of viewing or modifying it is also high. Furthermore, when users view the metadata of the target file, they may only be viewing a portion of the data. Therefore, the data processing device splits the target file into data blocks after creation to reduce the operational cost of viewing or modifying it.

[0167] As is understandable, block identifiers are used to uniquely identify data blocks; that is, each data block has its own unique block identifier. These block identifiers are assigned by the data processing device when the data block is generated.

[0168] S10412. Based on the metadata index parameters, retrieve the data block metadata corresponding to the data block from the directory mapping data.

[0169] S10413. Parse the metadata of the data block to obtain the file metadata of the target file, and realize the metadata reading operation for the target object.

[0170] After obtaining the metadata index parameters, the data processing device searches the directory mapping data for metadata that has a mapping relationship with the metadata index parameters. The resulting metadata is the data block metadata corresponding to the data block. Next, the data processing device parses the data block metadata to extract the file metadata of the target file, such as the size and format of the target file's data blocks. In this way, the data processing device performs the metadata reading operation on the target file.

[0171] In this embodiment, the data processing device parses the file identifier from the found target metadata, and then combines the file identifier and the block identifier of the data block to obtain the metadata index parameter in order to find the data block metadata corresponding to the data block. Then, the data block metadata is parsed to obtain the file metadata of the target file, thereby realizing the metadata reading operation of the target file.

[0172] In some embodiments of this application, after retrieving the data block metadata corresponding to the data block from the directory mapping data based on the metadata index parameters (i.e., S10412), the method may further include: S10414-S10416, as follows:

[0173] S10414. Update the data block metadata to obtain the updated data block metadata.

[0174] The data processing device determines the information that needs to be modified from the attribute information in the data block, then modifies the information that needs to be modified, and after the modification is completed, the updated data block metadata is obtained.

[0175] It is understandable that the information that needs to be modified in the attribute information, as well as the method of modifying the information that needs to be modified, can be determined based on the user's operation or automatically determined by the data processing device. This application does not impose any restrictions on this.

[0176] S10415. Establish the mapping relationship between metadata lookup parameters and updated data block metadata, and write the updated data block metadata into the directory mapping data.

[0177] S10416. Delete data block metadata from directory mapping data.

[0178] After updating the data block metadata, the data processing device establishes a mapping relationship between the metadata lookup parameters and the updated data block metadata, replacing the data block metadata of the directory mapping data with the updated data block metadata. It should be noted that updating the data block metadata in this embodiment essentially updates the file metadata of the target file; that is, this embodiment implements a file metadata update operation.

[0179] In this embodiment, the data processing device updates the read data block metadata, and then uses the mapping relationship between the metadata lookup parameters and the updated data block metadata in the directory mapping data to replace the original mapping relationship between the metadata lookup parameters and the data block metadata, thereby realizing the update operation of file metadata.

[0180] In some embodiments of this application, the processing operation includes a deletion operation. In this case, based on operations on the file index parameters and target metadata, the processing operation on the target file is implemented. Specifically, the implementation of S104 may include S10417, as follows:

[0181] S10417. Delete the target metadata and file index parameters from the directory mapping data to perform a deletion operation on the target file.

[0182] When the data processing device determines that the processing operation is a deletion operation, it will remove the target metadata and file index parameters from the directory mapping data. This removes the file index parameters and target metadata from the directory mapping data, so that the relevant data of the target file is no longer in the directory tree, thus realizing the deletion operation on the target file.

[0183] In this embodiment of the application, the data processing device can also delete the target metadata and file index parameters from the directory mapping data when the processing operation is a deletion operation, thereby realizing the deletion operation for the target file.

[0184] based on Figure 4 See Figure 9 , Figure 9 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3 In some embodiments of this application, before receiving the processing operation on the target file, i.e. before S101, the method may further include: S105-S108, as follows:

[0185] S105. Create the target file in the target directory.

[0186] Before receiving processing operations on a target file, the data processing device first needs to create the target file and add the mapping relationship between the file index parameters and target metadata of the target file to the directory mapping data. This way, the target metadata can be retrieved from the directory mapping data based on the file index parameters in the future.

[0187] It is understood that the data processing device may create the target file in the target directory in response to the user's operation, or it may create the target file in the target directory automatically (e.g., automatically generating operation logs, etc.). This application embodiment does not limit this.

[0188] S106. Using the unique directory identifier assigned to the target directory and the file name of the target file, generate the file index parameters corresponding to the target file.

[0189] After creating the target file, the data processing device will concatenate the directory identifier assigned to the target directory and the file name of the target file to obtain the file index parameter corresponding to the target file, or encode the directory identifier and the file name to obtain the file index parameter corresponding to the target file. This application does not limit this.

[0190] S107. Determine the attribute information of the target file as the target metadata corresponding to the target file.

[0191] It should be noted that the attribute information of the target file includes at least one of the following: a unique file identifier assigned to the target file and the number of data blocks in the target file.

[0192] The unique file identifier is assigned to the target file by the data processing device when it is created. The file identifier can be a number or a string, and this application does not impose any limitation on it. After creating the target file, the data processing device will also divide the target file into data blocks. After the division is completed, the data processing device will obtain the number of data blocks.

[0193] It is understood that the execution order of S106 and S107 will not affect the establishment of the mapping relationship between the file index parameters and the target metadata. Therefore, in other embodiments of this application, the data processing device may execute S107 first and then S106, or execute S106 and S107 simultaneously to obtain the file index parameters and the target metadata.

[0194] S108. Establish the mapping relationship between file index parameters and target metadata, and write the file index parameters and target metadata into the directory mapping data.

[0195] After obtaining the file index parameters and target metadata, the data processing device can establish a mapping relationship between them by using the file index parameters as the key and the target metadata as the value, or by writing the file index parameters and target metadata into different columns of the same row of a data table to establish an index relationship between them. Finally, the data processing device will write the file index parameters and target metadata into the directory mapping data so that the target metadata can be found in the directory mapping data later using the file index parameters.

[0196] In this embodiment, after receiving a processing operation for a target file, the data processing device can first create the target file, construct file index parameters and target metadata for the target file, establish a mapping relationship between the file index parameters and target metadata, and write the file index parameters and target metadata into directory mapping data. Thus, after obtaining the file index parameters through encoding, the data processing device can subsequently locate the target metadata in the directory mapping data.

[0197] In this application, after obtaining the directory metadata, the data processing device, in addition to processing the target file using information such as the directory identifier in the directory metadata, can also process the directory based on operations on the directory index parameters and directory metadata. The directory processing is described below.

[0198] In some embodiments of this application, after retrieving the directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameter, i.e., after S102, the method may further include: S109, as follows:

[0199] S109. Read the attribute information corresponding to the target directory from the directory metadata.

[0200] In this embodiment, after obtaining the directory metadata, the data processing device can further parse the directory metadata to obtain the attribute information corresponding to the target directory, so that the user or the data processing device can understand the situation of the target directory. Thus, this embodiment implements the operation of reading directory attributes.

[0201] It should be noted that the attribute information corresponding to the target directory includes at least one or more of the following: directory identifier and directory creation time. The directory identifier is used to uniquely identify the target directory and is assigned by the data processing device when the target directory is created.

[0202] In this embodiment of the application, after obtaining the directory metadata, the data processing device can parse the directory metadata to obtain the attribute information corresponding to the target directory, so as to realize the reading operation of the directory attributes.

[0203] In some embodiments of this application, after reading the attribute information corresponding to the target directory from the directory metadata, i.e., after S109, the method may further include: S110-S112, as follows:

[0204] S110. Construct a file query prefix based on the directory identifier in the attribute information corresponding to the target directory.

[0205] The data processing device extracts the directory identifier from the attribute information corresponding to the target directory. Then, the data processing device can directly use the directory identifier as a file query prefix, or it can encode the target identifier to obtain the file query prefix.

[0206] S111. From the directory mapping data, find the candidate index parameters with file query prefixes, and the candidate file metadata corresponding to the candidate search parameters.

[0207] After obtaining the file query prefix, the data processing device will search for index parameters with the file query prefix from the various index parameters of the directory mapping data, and record these index parameters as candidate index parameters. Then, the data processing device will use the candidate index parameters to continue searching for the corresponding file metadata in the directory mapping data, thereby obtaining candidate file metadata.

[0208] For example, Figure 10 This is a schematic diagram illustrating the process of querying candidate file metadata provided in an embodiment of this application. See also... Figure 10 The directory identifier in the attribute information corresponding to the target directory is N10-1. The data processing device first queries the directory mapping data 10-2 and retrieves all index parameters prefixed with N, namely index parameter N / 1.dat10-3 and index parameter N / s.txt10-4. These retrieved index parameters are the candidate index parameters. Next, the data processing device performs a second query on the directory mapping data, obtaining the file metadata 10-5 corresponding to N / 1.dat10-3 and the file metadata 10-6 corresponding to N / s.txt10-4. These file metadata are the candidate file metadata.

[0209] S112. Parse the candidate file metadata to obtain the name of each file in the target directory and the number of data blocks in each file.

[0210] The data processing device parses the metadata of the candidate files, extracts the file names and the number of data blocks, thus obtaining the names and data block counts of different files. Since these files are all located in the target directory, the obtained file names and data block counts can be used to construct a list of files in the target directory, thereby clarifying the file situation within the target directory.

[0211] In this embodiment of the application, after obtaining the attribute information corresponding to the target directory, the data processing device will use the directory identifier in the attribute information corresponding to the target directory to construct a file query prefix, and query the candidate index parameter with the file query prefix in the target mapping data, and then query the candidate file metadata again. Finally, by parsing the candidate file metadata, the query of the file status under the target directory can be realized.

[0212] In some embodiments of this application, after searching for the directory metadata corresponding to the target directory from the directory mapping data using the directory index parameter, i.e., after S102, the method may further include: S113, as follows:

[0213] S113. Remove the directory index parameters and directory metadata from the directory mapping data.

[0214] Directory processing naturally includes directory deletion. After obtaining the directory index parameters and directory metadata, the data processing device can remove the directory index parameters and directory metadata from the directory mapping data to clear the mapping relationship between them, ensuring that the target directory no longer exists in the directory tree.

[0215] In this embodiment of the application, the data processing device can perform a deletion operation on the target directory by deleting the directory index parameter and directory metadata from the directory mapping data, so that the data processing device will no longer obtain the directory metadata in the directory mapping data.

[0216] In some embodiments of this application, before receiving the processing operation on the target file, i.e. before S101, the method may further include: S114-S117, as follows:

[0217] S114. Create the target directory and generate a directory identifier for the target directory.

[0218] Before receiving processing operations for the target file, the data processing device will first create a target directory and then assign a unique target identifier to the target directory after the creation is completed.

[0219] S115. Encode the target directory to obtain the directory index parameters.

[0220] It is understandable that the encoding process of S115 is similar to that of S101, and will not be described in detail here.

[0221] S116. Determine the directory identifier and the creation time of the target directory as directory metadata.

[0222] The data processing device packages the directory identifier and the timestamp of the target directory when it was created, i.e. the creation time of the target directory, into directory metadata so that a mapping relationship can be established later.

[0223] It is understood that the order in which the target index parameters are generated and the directory metadata is determined will not affect the mapping relationship between the directory index parameters and the directory metadata. Therefore, in some embodiments, the data processing device may execute S116 first and then S115, or it may execute S115 and S116 simultaneously. This application embodiment does not limit this.

[0224] S117. Establish the mapping relationship between directory index parameters and directory metadata, and write the directory index parameters and directory metadata into the directory mapping data.

[0225] After obtaining the directory index parameters and directory metadata, the data processing device will establish a mapping relationship between the directory index parameters and directory metadata, and write the directory index parameters and directory metadata into the directory mapping data so that the directory metadata can be found from the directory mapping data using the directory index parameters later.

[0226] In this embodiment of the application, before receiving the processing operation for the target file, the data processing device can first create the target directory, construct the directory index parameters using the target directory, construct the directory metadata using the directory identifier and creation time, and establish a mapping relationship between the directory index parameters and the directory metadata. This enables the target index parameters to be used to find the directory metadata from the directory mapping data, so as to determine the target metadata corresponding to the target file.

[0227] In some embodiments of this application, after retrieving the directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameter, i.e., after S102, the method may further include: S118-S121, as follows:

[0228] S118. Obtain the replacement directory and encode it to obtain the directory index prefix.

[0229] In this embodiment of the application, the data processing device needs to move a directory under the target directory to another directory. At this time, the data processing device will obtain the replacement directory, which refers to the directory to which the directory under the target directory is to be moved. Then, the replacement directory is encoded, and the encoding result is the directory index prefix.

[0230] S119. From the directory mapping data, find the matching directory index parameters that are prefixed with the directory index parameters, and the matching directory metadata corresponding to the matching directory index parameters.

[0231] The data processing device uses the directory index parameter as a prefix to search for index parameters with the same prefix in the directory mapping data. The index parameter found is the matching directory parameter.

[0232] It is understandable that, in the directory mapping data, only the index parameter corresponding to the directory is encoded as a directory, while the index parameter corresponding to the file is generated from the directory identifier and the file name and does not have an encoding result for the directory. Therefore, the data processing device uses the directory index parameter as a prefix and will definitely find the index parameter of the directory.

[0233] Next, the data processing device will use the matching directory index parameter as a search condition to continue searching for the matching directory metadata corresponding to the matching directory index parameter from the directory mapping data, thereby obtaining the metadata of the directory under the target directory.

[0234] S120. Replace the prefix of the matching index parameter with the directory index prefix to obtain the replaced index parameter, and establish a mapping relationship between the replaced index parameter and the matching directory metadata.

[0235] After obtaining the matching directory index parameters, the data processing device replaces the prefix (directory index parameter) in the matching directory index parameters with the move index parameters to obtain the replaced index parameters. Then, the data processing device establishes a mapping relationship between the replaced index parameters and the matching directory metadata, that is, it establishes a connection between the target directory and the various directories originally located under the target directory.

[0236] S121. Write the replaced index parameter into the directory mapping data, and delete the matching directory index parameter from the directory mapping data.

[0237] Finally, the data processing device writes the replaced index parameters into the directory mapping data and deletes the matching directory index parameters. In this way, only by using the replaced directory index parameters with the move index parameter prefix can the matching directory metadata be found, thereby moving the directory under the target directory to the target directory to realize the directory move operation.

[0238] In this embodiment, the data processing device can encode the acquired target directory to obtain the move index parameter, and find the matching directory index parameter with the directory index parameter as a prefix, thereby obtaining the matching directory metadata corresponding to the target directory. The move directory index parameter is used to replace the directory index parameter in the matching directory index parameter, and a mapping relationship is established. The original mapping relationship is deleted, thereby realizing the movement of the directory under the target directory to the target directory, which realizes the directory movement operation.

[0239] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0240] This application embodiment is implemented in the scenario of processing directories and files in the directory tree of the namespace of a distributed file system, including operations such as adding, deleting, modifying and querying directories and files.

[0241] Figure 11 This is a schematic diagram of a system framework for processing directories and files in a directory tree, as provided in an embodiment of this application. See also... Figure 11 The processing system framework deployed on the server (data processing device) includes three modules: directory tree operation module 11-1, directory tree encoding module 11-2, and LSM-Tree storage engine 11-3.

[0242] The directory tree operation module 11-1 is responsible for encapsulating the application interfaces provided to the outside world, including interfaces for creating, deleting, updating, moving, and reading lists of directories, and interfaces for creating, deleting, moving, renaming, reading metadata, and updating metadata of files.

[0243] The directory tree operation module 11-2 is responsible for encoding the directory name, file name, and metadata of the directory and file into key / value (mapping relationship) format according to a specific format, and calling the interface of the LSM-Tree storage engine 11-3 to write the key / value data (write it into the directory mapping data).

[0244] The LSM-Tree storage engine 11-3 is responsible for persistently storing key / value data and providing read functionality. In this application, the LSM-Tree storage engine 11-3 uses a high-performance storage engine (rocksdb), which supports data persistence to reliable storage media, such as SSDs / HDDs.

[0245] The following is a detailed introduction to each module.

[0246] In a distributed file system, the directory tree managed by the namespace includes two types of data: directories and files. The directory tree encoding module 11-2 writes these two types of data in key / value format, where the key is a string encoded according to specific rules, and the value is serialized directory metadata or file metadata.

[0247] The paths in the directory tree are strings composed of visible characters, starting with " / ". For example, " / data / a.file" represents the file "a.file" under the directory "data". The directory key (directory index parameter) consists of a fixed prefix and an encoded value. The fixed prefix is ​​1 byte long, the fixed value is set to 0, and the encoded value is the full path of the directory (the complete target directory). Further details are provided in Table 3, which illustrates the structure of directory keys.

[0248] Table 3

[0249]

[0250] The directory's metadata is serialized in the Protocol Data Exchange Format (protobuf) format and written as values ​​(directory metadata). The directory metadata includes several metadata-related fields, among which the directory ID (directory identifier) ​​is the most critical. The directory ID is assigned to each directory, ensuring a global assignment of 1 (the directory identifier is used to uniquely identify the target directory), starting from 1. Further details can be found in Table 4, which provides an illustration of the composition of the directory's metadata, i.e., the directory's values.

[0251] Table 4

[0252]

[0253] The description section explains the composition of each field in the directory's values.

[0254] Files use directory ID + filename as keys (using the directory identifier and the target filename to generate file index parameters). The file key (file index parameter) is prefixed with the directory ID (directory identifier), is 8 bytes long, and is used to distinguish files in different directories. The filename (filename) is used to distinguish different files in the same directory. Furthermore, Table 5 provides an illustration of the file key structure.

[0255] Table 5

[0256]

[0257] The file's value (target metadata) does not directly store the file's block metadata; instead, it stores information such as the number of blocks and the file ID. This information is stored after being serialized using protobuf format. Further details are provided in Table 6, illustrating the structure of the file's value.

[0258] Table 6

[0259]

[0260] As can be seen, the block count is used to record how many blocks are in the file.

[0261] File metadata uses "file id + block number" as the key (concatenated using the file identifier and the block identifier of the data block to form the metadata index parameter). The file metadata key is prefixed with the file id and is 8 bytes long. Different blocks under the same file are distinguished by their block numbers (block identifiers). Under the same file id, the allocation of block numbers is guaranteed to be incremental and unique. Furthermore, Table 7 provides a schematic diagram of the structure of file metadata keys.

[0262] Table 7

[0263]

[0264] File metadata (data block metadata) is stored after being serialized in protobuf format.

[0265] The directory tree operation module 11-1 needs to construct keys according to the above format and query or modify data within the LSM-Tree storage engine. Figure 1 Taking the directory tree shown as an example, the key-value storage structure is shown in Table 8.

[0266] Table 8

[0267]

[0268] It is evident that, in response to Figure 1 The directory tree shown uses the path of the directory containing the file as the key to store directory metadata when a file is created, assigning a unique directory ID. File metadata is then recorded using "directory ID + filename" as the key. When searching for a file, the directory ID of the directory containing the file is first obtained, and then the file's metadata is retrieved based on the directory ID and filename. This ensures the search depth remains consistently 2. When renaming a directory, only the directory name is modified; the directory ID remains unchanged, thus avoiding any alteration to the keys of the file's metadata.

[0269] The following sections describe the interface flow for operating the directory tree.

[0270] Directory creation / deletion: If it is creation, a globally unique directory ID needs to be assigned to the created directory. Then, the corresponding directory key is constructed according to Table 3, and the directory value is constructed according to Table 4 (establishing the mapping relationship between directory index parameters and directory metadata). Then, LSM-Tree is called to write / delete key-value pairs (writing directory index parameters and directory metadata into the directory mapping data).

[0271] Directory movement: Based on the directory path, construct the key prefix of the directory according to Table 3; traverse the LSM-Tree engine to read key-value pairs with the same key prefix (matching directory index parameters with the directory index parameter as the prefix); for each key-value pair, parse the complete directory path from the key, temporarily store the value in memory, write it to the new directory (replace the directory) according to the directory creation process, and delete the old directory according to the directory deletion process.

[0272] Read the list of files in the directory: Construct the keys of the directories according to Table 3, and read the values ​​of the directories from the LSM-Tree engine; parse the values ​​of the directories to obtain the directory ID; construct the prefix of the file keys (file query prefix) according to Table 5, and traverse the list of file values ​​(candidate file metadata) corresponding to the keys (candidate file metadata) of files with the same directory ID as the prefix from the LSM-Tree engine; parse the key-value pairs of the files to obtain the corresponding file list, including the file name and the number of blocks (parse the candidate file metadata to obtain the name of each file in the target directory and the number of data blocks of each file).

[0273] Read directory metadata: Construct the directory keys according to Table 3, and read the corresponding directory values ​​(directory metadata) from the LSM-Tree engine; parse the directory values ​​to obtain information such as directory ID and directory creation (read the attribute information corresponding to the target directory from the directory metadata).

[0274] File creation: Based on the file's path (path information), obtain the directory path information (target directory) where the file is located, and construct the directory key according to Table 3; read the directory value from the LSM-Tree engine (if it does not exist, create the directory according to the directory creation process); parse the directory value to obtain information such as directory ID and directory creation time; construct the file key according to Table 5 based on the directory ID, file name, etc., and construct the file value according to Table 6 (determine the target file's attribute information as the target metadata corresponding to the target file); call the LSM-Tree engine to write the file's key / value data.

[0275] File deletion: Based on the file's path, obtain the directory path information where the file is located; construct the directory key according to Table 3; read the directory value from the LSM-Tree engine; parse the directory value to obtain information such as directory ID and directory creation time; construct the file key (file index parameter) according to Table 5 based on the directory ID and file name; call the LSM-Tree engine to obtain the file value and delete the file's key / value data (remove the target metadata and file index parameter from the directory mapping data).

[0276] File movement: Obtain the directory path information where the file is located based on the file path; construct the directory key according to Table 3; read the directory value from the LSM-Tree engine and temporarily store it in memory; create the file according to the file process, using the value stored in memory in the previous step as the value corresponding to the file key; delete the original file according to the file deletion process.

[0277] File renaming: Based on the path of the old file, obtain the directory path information where the old file is located; construct the directory key according to Table 3; read the directory value from the LSM-Tree engine; parse the directory value to obtain information such as directory ID and directory creation time; construct the file key according to Table 5 based on the directory ID and file name; read the file value from the LSM-Tree engine and temporarily store it in memory; construct the new file key (rename index parameter) according to Table 5 based on the directory ID and new file name (rename name), and use the data temporarily stored in the previous step as the file value; call the LSM-Tree engine to write the new file key and file value (write the rename index parameter to the directory mapping data), and delete the original file key (delete the file index parameter from the directory mapping data).

[0278] File metadata reading: Based on the path of the old file, obtain the directory path information where the old file is located; construct the directory key according to Table 3; read the directory value from the LSM-Tree engine; parse the directory value to obtain information such as directory ID and directory creation time; construct the file key according to Table 5 based on the directory ID and file name; read the file value from the LSM-Tree engine; parse the file value to obtain information such as file ID and number of blocks; construct the metadata key (metadata index parameters) according to Table 7 based on the file ID and block sequence number; read the metadata value from the LSM-Tree engine; parse the metadata value to obtain the metadata (data block metadata).

[0279] File metadata writing / updating: Based on the path of the old file, obtain the directory path information where the old file is located; construct the directory key according to Table 3; read the directory value from the LSM-Tree engine; parse the directory value to obtain information such as directory ID and directory creation time; construct the file key according to Table 5 based on the directory ID and file name; read the file value from the LSM-Tree engine; parse the file value to obtain information such as file ID and number of blocks; construct the metadata key according to Table 7 based on the file ID and block sequence number; invoke the LSM-Tree engine to write the metadata key / value data, or update the metadata key / value data (update the data block metadata to obtain the updated data block metadata).

[0280] Using the above method, the server can find any file with just two queries, resulting in efficient searching. Furthermore, since the file index parameters do not include directory names, fewer index parameters need to be modified when renaming a directory, leading to high modification efficiency. In summary, the overall efficiency of data processing is improved. Additionally, building a directory tree on the hard disk reduces memory consumption, increases the number of storable directory tree nodes, and reduces the time required to restore the directory tree during device restarts.

[0281] The following description continues to illustrate the exemplary structure of the data processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 3 As shown, the software modules stored in the data processing device 555 of the memory 550 may include:

[0282] The operation receiving module 5551 is used to receive processing operations on the target file;

[0283] Information encoding module 5552 is used to encode the target directory where the target file is located to obtain directory index parameters;

[0284] The data lookup module 5553 is used to look up directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameters; the directory mapping data stores the mapping relationship between the directory index parameters and the directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file; the file index parameters, constructed based on the directory metadata and the file name of the target file, are used to look up the target metadata describing the attributes of the target file from the directory mapping data;

[0285] The file processing module 5554 is used to implement the processing operation on the target file based on the operation on the file index parameters and the target metadata. The processing operation includes at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation.

[0286] In some embodiments of this application, the data lookup module 5553 is further configured to parse the directory identifier corresponding to the target directory from the directory metadata; generate the file index parameter using the directory identifier and the file name of the target file; and look up the target metadata describing the attributes of the target file from the directory mapping data based on the file index parameter.

[0287] In some embodiments of this application, the processing operation includes: the move operation; the file processing module 5554 is further configured to encode the move directory corresponding to the move operation to obtain a move index parameter; the move directory represents the directory to which the target file is moved; based on the move index parameter, the move metadata corresponding to the move directory is filtered from the directory mapping data; the move identifier of the move directory is parsed from the move metadata, and the latest index parameter is concatenated using the move identifier and the file name; a mapping relationship is established between the latest index parameter and the target metadata, and the latest index parameter is written into the directory mapping data; the file index parameter is deleted from the directory mapping data to realize the move operation on the target file.

[0288] In some embodiments of this application, the processing operation includes: the renaming operation; the file processing module 5554 is further configured to obtain the renaming name corresponding to the renaming operation; concatenate the renaming index parameter according to the directory identifier and the renaming name; establish a mapping relationship between the renaming index parameter and the target metadata, and write the renaming index parameter into the directory mapping data; delete the file index parameter from the directory mapping data, thereby realizing the renaming operation for the target file.

[0289] In some embodiments of this application, the directory mapping data further includes: a mapping relationship between metadata index parameters and data block metadata; the processing operation includes: the metadata reading operation; the file processing module 5554 is further configured to parse the file identifier of the target file from the target metadata; use the file identifier and the block identifier of the data block corresponding to the metadata reading operation to concatenate the metadata index parameters; wherein, the data block is obtained by splitting the target file, and the block identifier is used to uniquely identify the data block; based on the metadata index parameters, the data block metadata corresponding to the data block is found from the directory mapping data; the data block metadata is parsed to obtain the file metadata of the target file, thereby realizing the metadata reading operation for the target file.

[0290] In some embodiments of this application, the file processing module 5554 is further configured to update the data block metadata to obtain updated data block metadata; establish a mapping relationship between the metadata lookup parameters and the updated data block metadata, and write the updated data block metadata into the directory mapping data; and delete the data block metadata from the directory mapping data.

[0291] In some embodiments of this application, the processing operation includes: the deletion operation; the file processing module 5554 is further configured to delete the target metadata and the file index parameters from the directory mapping data, thereby implementing the deletion operation for the target file.

[0292] In some embodiments of this application, the data processing apparatus 555 further includes: a file creation module 5555;

[0293] The file creation module 5555 is configured to: create the target file in the target directory; generate file index parameters corresponding to the target file using a unique directory identifier assigned to the target directory and the file name of the target file; determine the attribute information of the target file as the target metadata corresponding to the target file; wherein the attribute information of the target file includes at least one or more of the following: a unique file identifier assigned to the target file and the number of data blocks in the target file; establish a mapping relationship between the file index parameters and the target metadata; and write the file index parameters and the target metadata into the directory mapping data.

[0294] In some embodiments of this application, the data processing device 555 further includes: a directory processing module 5556;

[0295] The directory processing module 5556 is used to read the attribute information corresponding to the target directory from the directory metadata. The attribute information corresponding to the target directory includes at least one or more of the following: directory identifier and directory creation time. The directory identifier is used to uniquely identify the target directory.

[0296] In some embodiments of this application, the directory processing module 5556 is further configured to construct a file query prefix based on the directory identifier in the attribute information corresponding to the target directory; find candidate index parameters with the file query prefix and candidate file metadata corresponding to the candidate search parameters from the directory mapping data; parse the candidate file metadata to obtain the names of each file under the target directory and the number of data blocks of each file.

[0297] In some embodiments of this application, the directory processing module 5556 is further configured to create the target directory and generate the directory identifier for the target directory; encode the target directory to obtain the directory index parameter; determine the directory identifier and the creation time of the target directory as the directory metadata; establish a mapping relationship between the directory index parameter and the directory metadata; and write the directory index parameter and the directory metadata into the directory mapping data.

[0298] In some embodiments of this application, the directory processing module 5556 is further configured to obtain a replacement directory and encode the replacement directory to obtain a directory index prefix; wherein, the replacement directory refers to the directory to which the directory under the target directory is to be moved; from the directory mapping data, find the matching directory index parameter with the directory index parameter as a prefix, and the matching directory metadata corresponding to the matching directory index parameter; replace the prefix of the matching directory index parameter with the directory index prefix to obtain the replaced index parameter, and establish a mapping relationship between the replaced index parameter and the matching directory metadata; write the replaced index parameter into the directory mapping data, and delete the matching directory index parameter from the directory mapping data.

[0299] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in this application.

[0300] This application provides a computer-readable storage medium storing executable instructions, including executable data processing instructions. When these executable instructions are executed by a processor, they will cause the processor to perform the data processing method provided in this application, for example... Figure 4 The method shown.

[0301] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0302] In some embodiments, executable data processing instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0303] As an example, executable data processing instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0304] As an example, executable data processing instructions can be deployed to execute on a single computing device, or on multiple data processing devices located in one location, or on multiple data processing devices distributed across multiple locations and interconnected via a communication network.

[0305] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Receive processing operations for the target file and encode the target directory where the target file is located to obtain directory index parameters; Using the directory index parameters, directory metadata describing the attributes of the target directory is retrieved from the directory mapping data; The directory mapping data stores the mapping relationship between the directory index parameters and directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file, in the form of key-value pairs. Parse the directory identifier corresponding to the target directory from the directory metadata; The directory identifier is used as a prefix, and the prefix is ​​concatenated with the file name of the target file to obtain the file index parameter; Based on the file index parameters, the target metadata describing the attributes of the target file is retrieved from the directory mapping data. The target metadata is obtained through two searches: the first search is completed based on the directory index parameters, and the second search is completed based on the file index parameters. The target metadata includes the file identifier of the target file. The directory mapping data also stores a mapping relationship with the file identifier and data block number of the target file as keys and the data block metadata as values. Based on operations on the file index parameters and the target metadata, the processing operations on the target file are implemented. These processing operations include at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation. The move operation on the target file is implemented as follows: the move directory is encoded to obtain a move index parameter; based on the move index parameter, the move metadata corresponding to the move directory is filtered from the directory mapping data; based on the file name and the move identifier of the move directory in the move metadata, a latest index parameter is constructed; a mapping relationship is established between the latest index parameter and the target metadata, and the latest index parameter is written into the directory mapping data; the file index parameter is deleted from the directory mapping data.

2. The method according to claim 1, characterized in that, The processing operations include: the renaming operation; the processing operations on the target file based on the operations on the file index parameters and the target metadata include: Obtain the rename name corresponding to the rename operation; The renaming index parameter is generated by concatenating the directory identifier and the rename name; Establish a mapping relationship between the rename index parameters and the target metadata, and write the rename index parameters into the directory mapping data; The file index parameter is deleted from the directory mapping data to perform the renaming operation on the target file.

3. The method according to claim 1, characterized in that, The directory mapping data also includes: the mapping relationship between metadata index parameters and data block metadata; the processing operation includes: the metadata reading operation; the processing operation on the target file based on the operations on the file index parameters and the target metadata includes: The file identifier of the target file is parsed from the target metadata; The file identifier and the block identifier of the data block corresponding to the metadata read operation are concatenated to form a metadata index parameter; wherein, the data block is obtained by splitting the target file, and the block identifier is used to uniquely identify the data block; Based on the metadata index parameters, the metadata of the data block corresponding to the data block is retrieved from the directory mapping data; The metadata of the data block is parsed to obtain the file metadata of the target file, thereby enabling the metadata reading operation for the target file.

4. The method according to claim 3, characterized in that, After retrieving the data block metadata corresponding to the data block from the directory mapping data based on the metadata index parameters, the method further includes: The data block metadata is updated to obtain the updated data block metadata; Establish a mapping relationship between the metadata lookup parameters and the updated data block metadata, and write the updated data block metadata into the directory mapping data; Delete the data block metadata from the directory mapping data.

5. The method according to claim 1, characterized in that, The processing operation includes: the deletion operation; the processing operation on the target file based on the operations on the file index parameters and the target metadata includes: The target metadata and the file index parameters are removed from the directory mapping data to perform a deletion operation on the target file.

6. The method according to any one of claims 1 to 5, characterized in that, Before receiving the processing operation for the target file, the method further includes: Create the target file in the target directory; Using the unique directory identifier assigned to the target directory and the file name of the target file, generate the file index parameter corresponding to the target file; The attribute information of the target file is determined as the target metadata corresponding to the target file; wherein, the attribute information of the target file includes at least one or more of the following: a unique file identifier assigned to the target file and the number of data blocks in the target file; Establish a mapping relationship between the file index parameters and the target metadata, and write the file index parameters and the target metadata into the directory mapping data.

7. The method according to any one of claims 1 to 5, characterized in that, After retrieving the directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameters, the method further includes: The directory creation time of the target directory is read from the directory metadata, and the directory identifier is used to uniquely identify the target directory.

8. The method according to claim 7, characterized in that, After reading the attribute information corresponding to the target directory from the directory metadata, the method further includes: Based on the directory identifier in the attribute information corresponding to the target directory, a file query prefix is ​​constructed; From the directory mapping data, find the candidate index parameters that have the file query prefix, and the candidate file metadata corresponding to the candidate search parameters; The candidate file metadata is parsed to obtain the names of each file in the target directory and the number of data blocks in each file.

9. The method according to claim 7, characterized in that, Before receiving the processing operation for the target file, the method further includes: Create the target directory and generate the directory identifier for the target directory; The target directory is encoded to obtain the directory index parameters; The directory identifier and the creation time of the target directory are determined as the directory metadata; Establish a mapping relationship between the directory index parameters and the directory metadata, and write the directory index parameters and the directory metadata into the directory mapping data.

10. The method according to any one of claims 1 to 5, 8 or 9, characterized in that, After retrieving the directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameters, the method further includes: Obtain the replacement directory and encode the replacement directory to obtain the directory index prefix; wherein, the replacement directory refers to the directory to which the directory under the target directory is to be moved; From the directory mapping data, find the matching directory index parameter that is prefixed with the directory index parameter, and the matching directory metadata corresponding to the matching directory index parameter; Replace the prefix of the matching directory index parameter with the directory index prefix to obtain the replaced index parameter, and establish a mapping relationship between the replaced index parameter and the matching directory metadata; The replaced index parameter is written into the directory mapping data, and the matching directory index parameter is deleted from the directory mapping data.

11. A data processing apparatus, characterized in that, include: The operation receiving module is used to receive processing operations on the target file; The information encoding module is used to encode the target directory where the target file is located to obtain directory index parameters; The data lookup module is used to retrieve directory metadata describing the attributes of the target directory from the directory mapping data using the directory index parameters. The directory mapping data stores the mapping relationship between the directory index parameters and directory metadata, as well as the mapping relationship between the file index parameters and the target metadata corresponding to the target file, in the form of key-value pairs. Parse the directory identifier corresponding to the target directory from the directory metadata; The directory identifier is used as a prefix, and the prefix is ​​concatenated with the file name of the target file to obtain the file index parameter; Based on the file index parameters, the target metadata describing the attributes of the target file is retrieved from the directory mapping data. The target metadata is obtained through two searches: the first search is completed based on the directory index parameters, and the second search is completed based on the file index parameters. The target metadata includes the file identifier of the target file. The directory mapping data also stores a mapping relationship with the file identifier and data block number of the target file as keys and the data block metadata as values. The file processing module is used to implement the processing operations on the target file based on the operations on the file index parameters and the target metadata. The processing operations include at least one or more of the following: move operation, delete operation, rename operation, and metadata read operation. The move operation on the target file is implemented as follows: encoding the move directory to obtain a move index parameter; filtering the move metadata corresponding to the move directory from the directory mapping data based on the move index parameter; concatenating the latest index parameter based on the file name and the move identifier of the move directory in the move metadata; establishing a mapping relationship between the latest index parameter and the target metadata, and writing the latest index parameter into the directory mapping data; and deleting the file index parameter from the directory mapping data.

12. The apparatus as claimed in claim 11, characterized in that, The processing operations include: the renaming operation; the file processing module is further configured to: Obtain the rename name corresponding to the rename operation; construct the rename index parameter based on the directory identifier and the rename name; establish a mapping relationship between the rename index parameter and the target metadata, and write the rename index parameter into the directory mapping data; delete the file index parameter from the directory mapping data to realize the rename operation for the target file.

13. The apparatus as claimed in claim 11, characterized in that, The directory mapping data also includes: the mapping relationship between metadata index parameters and data block metadata; the processing operation includes: the metadata reading operation; the file processing module is further used for: The file identifier of the target file is parsed from the target metadata; the file identifier and the block identifier of the data block corresponding to the metadata read operation are concatenated to form a metadata index parameter; wherein, the data block is obtained by splitting the target file, and the block identifier is used to uniquely identify the data block; based on the metadata index parameter, the data block metadata corresponding to the data block is found from the directory mapping data; the data block metadata is parsed to obtain the file metadata of the target file, thereby realizing the metadata read operation for the target file.

14. The apparatus as claimed in claim 13, characterized in that, The file processing module is also used for: The data block metadata is updated to obtain the updated data block metadata; a mapping relationship is established between the metadata lookup parameters and the updated data block metadata, and the updated data block metadata is written into the directory mapping data; Delete the data block metadata from the directory mapping data.

15. The apparatus as claimed in claim 11, characterized in that, The processing operation includes: the deletion operation; the file processing module is further used for: The target metadata and the file index parameters are removed from the directory mapping data to perform a deletion operation on the target file.

16. The apparatus as claimed in claim 11 or 15, characterized in that, It also includes a file creation module for: In the target directory, the target file is created; using the unique directory identifier assigned to the target directory and the file name of the target file, the file index parameter corresponding to the target file is generated; The attribute information of the target file is determined as the target metadata corresponding to the target file; wherein, the attribute information of the target file includes at least one or more of the following: a unique file identifier assigned to the target file and the number of data blocks in the target file; a mapping relationship is established between the file index parameters and the target metadata, and the file index parameters and the target metadata are written into the directory mapping data.

17. The apparatus as claimed in claim 11 or 15, characterized in that, It also includes a directory processing module, used for: The directory creation time of the target directory is read from the directory metadata, and the directory identifier is used to uniquely identify the target directory.

18. A data processing device, characterized in that, include: Memory, used to store executable data processing instructions; A processor, when executing executable data processing instructions stored in the memory, implements the method according to any one of claims 1 to 10.

19. A computer-readable storage medium, characterized in that, It stores executable data processing instructions for implementing the method according to any one of claims 1 to 10 when executed by a processor.

20. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1 to 10.

Images