Data processing method and device
By merging and sharing computation methods, the problem of insufficient metadata throughput in distributed file systems is solved, and the system's metadata processing performance is improved, especially in the ability to handle massive amounts of small files during the training and inference of large language models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2026-03-03
AI Technical Summary
Existing distributed file systems are insufficient in terms of metadata throughput, especially during the training and inference of large language models, where they cannot effectively handle metadata requests for massive numbers of small files, leading to system performance bottlenecks.
By merging multiple metadata requests of the same type into a target request, and performing path resolution and shared computation in a single execution, the amount of access to shared resources is reduced, and an adaptive aggregation strategy is used to balance latency and throughput under different load conditions.
Significantly improved metadata processing performance, with a maximum IOPS increase of approximately 650% for Create and approximately 770% for Stat, effectively solving the problem of insufficient metadata throughput.
Smart Images
Figure CN121597635A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, specifically to a data processing method and apparatus. Background Technology
[0002] A distributed file system is a logically unified database composed of multiple physically dispersed database units connected by a computer network. Each connected database unit is called a site or node.
[0003] In a distributed file system, files can be shared across multiple hosts over a network, allowing multiple users on multiple machines to share files and storage space. Metadata and data have different characteristics. Metadata requires frequent access, but regardless of file size, its metadata size is generally similar and relatively small. Data, on the other hand, is the actual data stored in the file, occupying approximately the same space as the file size and potentially requiring a large amount of storage. Therefore, distributed file systems typically manage metadata and data separately. Data storage is largely similar, generally involving distributing data across different physical nodes according to an algorithm, thereby utilizing the storage space of different physical nodes.
[0004] Current distributed file systems primarily address the issue of insufficient storage space on a single machine, but their metadata throughput is relatively low. With the development of large language models, massive amounts of files are used during model training and inference (e.g., in multimodal models, a single modality typically uses a file as part of the input; furthermore, the training of large language models relies on massive amounts of high-quality data). These files may not be very large in size, but they are numerous. Therefore, improving the throughput for metadata is a pressing issue that needs to be addressed. Summary of the Invention
[0005] In a first aspect, this application provides a data processing method, characterized in that the method includes: obtaining a target request based on a plurality of first requests, wherein the first requests are operation requests for metadata, the first requests carry paths of the metadata to be operated on, and the plurality of first requests are requests of the same operation type, and the operation path included in the target request is an aggregation of the paths carried by the plurality of first requests; executing the target request during a request execution process to obtain the execution result of the plurality of first requests.
[0006] After obtaining multiple first requests, the operation parameters (paths) for metadata contained in the multiple first requests can be collected to obtain a merged request (target request). During execution, the target request can be executed as a single request, thereby improving the system's throughput for metadata requests and thus enhancing the processing performance for metadata.
[0007] In one possible implementation, the aggregation of the paths carried by the plurality of first requests includes a first character, a second character, and a separator; the first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
[0008] In one possible implementation, obtaining the target request based on multiple first requests includes: placing the multiple first requests in a waiting list when there is no free queue; aggregating the paths carried by the multiple first requests to obtain the target request when there is a free queue; and executing the target request during a single request execution includes: executing the target request during a single request execution through the free queue.
[0009] When the distributed file system is under low pressure, idle proxy threads can be assumed to always exist, and new requests will be processed immediately without introducing additional latency. However, when the distributed file system is under high pressure, the waiting time for idle proxy threads increases with the pressure, resulting in more requests being aggregated in the second step. This allows for better utilization of subsequent shared computation to increase system throughput. In summary, this aggregation strategy balances latency and throughput. Under low pressure, throughput requirements can always be met, and latency becomes more critical; this aggregation strategy can minimize latency as much as possible. Under high pressure, if throughput is low, latency cannot be improved due to request backlog; in this case, throughput becomes more important, and this aggregation strategy can be combined with shared computation to improve throughput.
[0010] In one possible implementation, before executing the target request during a request execution process, the method further includes: preparing the target request as a request for execution; wherein the preparation process for request execution includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
[0011] The preparation phase is an unavoidable overhead for a single execution. For example, if a distributed file system is implemented as a distributed database, the preparation phase includes network transmission, SQL parsing, transaction management, and resource management. However, although these overheads are unavoidable for a single execution, if an execution contains multiple requests, these overheads can be amortized across the average time of each request, which is one of the benefits of shared computing.
[0012] In one possible implementation, before executing the target request during a request execution process, the method further includes: performing target request path resolution as a request.
[0013] In one possible implementation, the paths carried by the plurality of first requests include a first path and a second path, the first path and the second path including a common directory node, and the directory node is parsed only once during the execution path resolution.
[0014] After the preparation phase is complete, the shared path resolution phase begins. The distributed file system needs to resolve the directory tree level by level, starting from the root node. If an execution contains only one request, each node on the path needs to access shared resources (such as the underlying file system, database tables, etc., depending on the distributed file system implementation). However, if an execution contains multiple requests, shared path resolution can reduce the number of accesses to shared resources, thereby reducing lock contention and improving system throughput.
[0015] In one possible implementation, the paths carried by the plurality of first requests are sorted according to a preset rule during the execution path resolution, and the directory nodes included on the paths carried by the plurality of first requests are locked according to the sorting.
[0016] In one possible implementation, the M first requests, including the target request, are executed in the same batch. The method further includes: when the first N first requests succeed and the (N+1)th first request fails, re-execute the N first requests and skip the (N+1)th first request; and execute the M second requests following the M first requests as requests to be executed in the same batch.
[0017] Secondly, this application provides a data processing apparatus, the apparatus comprising:
[0018] The merging module is used to obtain a target request based on multiple first requests. The first requests are operation requests for metadata. The first requests carry the path of the metadata to be operated on, and the multiple first requests are requests of the same operation type. The operation path included in the target request is an aggregation of the paths carried by the multiple first requests.
[0019] An execution module is used to execute the target request during a single request execution process to obtain the execution results of the plurality of first requests.
[0020] In one possible implementation, the aggregation of the paths carried by the plurality of first requests includes a first character, a second character, and a separator;
[0021] The first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
[0022] In one possible implementation, the merging module is specifically used for:
[0023] If there is no free queue, place the plurality of first requests in the waiting list;
[0024] When an idle queue exists, the paths carried by the multiple first requests are aggregated to obtain the target request;
[0025] The execution module is specifically used for:
[0026] The target request is executed during a single request execution process using the idle queue.
[0027] In one possible implementation, before executing the target request during a single request execution, the execution module is further configured to:
[0028] The target request is treated as a request to perform request execution preparation; wherein the request execution preparation process includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
[0029] In one possible implementation, before executing the target request during a single request execution, the execution module is further configured to:
[0030] The target request is treated as a single request, and path resolution is performed on the target request.
[0031] In one possible implementation, the paths carried by the plurality of first requests include a first path and a second path, the first path and the second path including a common directory node, and the directory node is parsed only once during the execution path resolution.
[0032] In one possible implementation, the paths carried by the plurality of first requests are sorted according to a preset rule during the execution path resolution, and the directory nodes included on the paths carried by the plurality of first requests are locked according to the sorting.
[0033] In one possible implementation, the M first requests, including the target request, are requests executed in the same batch, and the execution module is further configured to:
[0034] If the first N first requests out of the M first requests succeed and the (N+1)th first request fails, the N first requests are re-executed, and the (N+1)th first request is skipped.
[0035] The M second requests following the M first requests are executed as a single batch of requests.
[0036] Thirdly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory to cause the computing device or the computing device cluster to perform a method as described in the first aspect or any implementation thereof.
[0037] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to execute the method of the database system described in the first aspect or any implementation thereof.
[0038] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to execute the database system execution method described in the first aspect or any implementation thereof.
[0039] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0040] Figures 1A to 1D This is a schematic diagram of the application framework of this application;
[0041] Figure 2A and Figure 2B This is a schematic diagram of the application framework of this application;
[0042] Figure 3 This is a flowchart illustrating the data processing method according to an embodiment of this application;
[0043] Figure 4 An illustration of a private protocol;
[0044] Figure 5 This is an illustration of the execution of the request;
[0045] Figure 6 This is an illustration of the execution of the request;
[0046] Figure 7 This is a diagram illustrating the parsing of a request;
[0047] Figure 8 This is an illustration of the execution of the request;
[0048] Figure 9 This is an illustration of the execution of the request;
[0049] Figures 10 to 13 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application. Detailed Implementation
[0050] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0051] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0052] The method provided in this application embodiment can be applied to a database system. Figure 1A This illustrates a typical logical architecture of a database system, based on Figure 1A Database system 100 includes database 110 and database management system (DBMS) 130.
[0053] Database 110 is an organized collection of data stored in data storage 120; that is, a related collection of data organized, stored, and used according to a specific data model. Depending on the data model used to organize the data, the data can be divided into various types, such as relational data, graph data, and time series data. Relational data is data modeled using a relational model, usually represented as a table, where rows represent a set of related values for an object or entity. Graph data, simply called a "graph," is used to represent relationships between objects or entities, such as social relationships. Time series data is a series of data columns recorded and indexed in chronological order, used to describe the state changes of an object over time.
[0054] The database management system 130 is the core of the database system, serving as system software for organizing, storing, and maintaining data. Clients 200 can access the database 110 through the database management system 130, and database administrators also use the database management system for database maintenance. The database management system 130 provides various functions for clients 200 to create, modify, and query the database; clients 200 can be applications or user devices. The functions provided by the database management system 130 may include, but are not limited to, the following: (1) Data definition function: The database management system 130 provides a data definition language (DDL) to define the structure of the database 110. The DDL is used to describe the database framework and can be stored in the data dictionary; (2) Data access function: The database management system 130 provides a data manipulation language (DML) to implement basic access operations on the database 110, such as retrieval, insertion, modification and deletion; (3) Database operation management function: The database management system 130 provides data control function to effectively control and manage the operation of the database 110 to ensure that the data is correct and valid; (4) Database creation and maintenance function: including loading the initial data of the database, database dumping, recovery, reorganization, system performance monitoring and analysis, etc.; (5) Database transmission: The database management system provides data transmission to realize communication between the client and the database management system, which is usually coordinated with the operating system.
[0055] Data storage 120 includes, but is not limited to, solid-state drives (SSDs), disk arrays, cloud storage, or other types of non-transitory computer-readable storage media. Those skilled in the art will understand that a database system may include more than […]. Figure 1A The components shown are fewer or more components, or include those with Figure 1A The components shown are different components. Figure 1A Only components more relevant to the implementation disclosed in the embodiments of the present invention are shown.
[0056] The database system provided in this application embodiment can be a distributed database system (DDBS). In order to achieve concurrency control between transactions during transaction processing, DDBS typically uses a global transaction manager (GTM) to manage transactions. The following describes how... Figure 1B and Figure 1C Introducing DDBS.
[0057] Figure 1B This is a schematic diagram of a distributed database system using a shared-storage architecture, including one or more coordinator nodes (CNs), multiple data nodes (DNs), and one or more GTMs (such as...). Figure 1B The system comprises a first GTM and a second GTM. The first GTM acts as the master GTM, while the second GTM backs up the data of the first GTM and takes over its operation when the first GTM fails, ensuring high reliability of DDBS. CNs and DNs communicate via a network channel. In one embodiment, this network channel can consist of network devices such as switches, routers, and gateways. CNs, DNs, and GTMs jointly implement the functions of the database management system, providing clients with services such as database retrieval, insertion, modification, and deletion. In one embodiment, a database management system is deployed on each CN, DN, and GTM. A shared data storage device stores data that can be shared by multiple DNs, and DNs can perform read and write operations on the data in the data storage device through the network channel. The shared data storage device can be a shared disk array. In the distributed database system, the CN, DN, first GTM, or second GTM can be a physical machine, such as a database server, or a virtual machine (VM) or container running on abstract hardware resources. In one embodiment, the CN, DN, first GTM, or second GTM is a virtual machine or container, and the network channel is a virtual switching network, which includes virtual switches. The database management system deployed in the CN, DN, first GTM, or second GTM is a DBMS instance. This DBMS instance can be a process or a thread, and these DBMSs work together to perform the functions of a database relational system. In another embodiment, the CN, DN, first GTM, or second GTM are physical machines, and the network channel includes one or more switches, which are storage area network (SAN) switches, Ethernet switches, fiber optic switches, or other physical switching devices.
[0058] Figure 1C This diagram illustrates a distributed database system employing a shared-nothing architecture. Each Data Domain (DN) has its own dedicated hardware resources (such as data storage), operating system, and database. The CN, DN, and either the first or second GTM communicate via a network channel, which can be found in the above description. Figure 1BPlease refer to the relevant introductions for understanding. In this system, data is allocated to various Data Domains (DNs) based on the database model and application characteristics. Query tasks are divided into several parts by the Data Network (CN), executed in parallel on all DNs, and collaboratively computed to provide database services as a whole. All communication functions are implemented on a high-bandwidth network interconnection system. (Similar to...) Figure 1B Similar to the distributed database system with the described shared-storage architecture, CN, DN, first GTM, or second GTM can be either physical machines or virtual machines.
[0059] In all embodiments of this application, the data storage of the database system includes, but is not limited to, solid-state drives (SSDs), disk arrays, or other types of non-transient computer-readable media. Figures 1B-1C Although a database is not shown, it should be understood that a database is stored in a data storage device. Those skilled in the art will understand that a database system may include more than [a certain number of data storage devices]. Figures 1A-1C The components shown are fewer or more components, or include those with Figures 1A-1C The components shown are different components. Figures 1A-1C Only components more relevant to the implementation disclosed in the embodiments of this application are shown. However, those skilled in the art will understand that a distributed database system can contain any number of CNs and DNs. The database management system functions of each CN and DN can be implemented by appropriate combinations of software, hardware, and / or firmware running on each CN and DN.
[0060] The above Figure 1B and Figure 1C The described distributed database system includes multiple DNs and multiple CNs, where each DN has essentially the same function, and each CN also has essentially the same function.
[0061] Figure 1D This application illustrates an application data management system provided by an embodiment of this application. Specifically, as shown, a data node cluster may include multiple data nodes (DNs). Application developers can deploy relevant data of their developed applications in the corresponding data node DNs. The application can send service requests to an application server, which can convert these service requests into data operation requests. The application server can then send the data operation requests to a distributed database system. Subsequently, one or more data nodes (DNs) in the distributed database system perform the operations corresponding to the operation requests on the data in the database storage. Specifically, the application server can send the data operation requests to a coordinating node in a coordinating node cluster. The coordinating node then forwards the operation requests to the relevant DNs, which then perform the data operations corresponding to the operation requests.
[0062] Reference Figure 2A , Figure 2A This is a system architecture diagram of an embodiment of this application: This embodiment can be used for training large AI models. Currently, large model training requires input of multiple modalities of data, which are generally stored in a distributed file system in the form of files and read by the GPU cluster during training. Due to the insufficient metadata processing capabilities of current distributed file systems, the GPUs cannot reach full capacity during training, and a large amount of time is consumed waiting for data to be read, resulting in a significant waste of computing power. This embodiment can improve the throughput of the distributed file system, thus aiding in the training of large AI models.
[0063] Reference Figure 2B , Figure 2B The following is a system architecture diagram of an embodiment of this application: This embodiment can be applied to a distributed file system to improve its metadata processing capabilities and meet the storage needs of massive small files. In this type of system, multiple clients typically initiate requests concurrently. The client first requests metadata information from the metadata node (depending on the metadata service architecture, there may be only one metadata node, or the client may request the corresponding metadata node according to predefined distribution rules), and then requests file data from the data node.
[0064] In such systems, data nodes can generally be expanded indefinitely, thereby gaining larger storage space and higher transmission bandwidth. However, because metadata nodes need to ensure data consistency among nodes, their expansion is accompanied by increased synchronization costs, and their number is generally far less than that of data nodes, thus limiting metadata processing capabilities.
[0065] The following describes the relevant terms and concepts that may be involved in the embodiments of this application.
[0066] 1) Metadata: File metadata is the control information of a file, containing some predefined inherent file attributes that facilitate user operations on the file, such as file size, last modified time, and access permissions. Operation types on metadata include Create (create file), Stat (query file metadata), Mkdir (create directory), and Rmdir (delete directory).
[0067] 2) Data consistency: This means that there should be no contradictions between the data of different nodes. For example, if node 1 believes that file A exists, while node 2 believes that file A does not exist, this creates a contradiction and violates data consistency.
[0068] 3) Data Locality Assumption: A fundamental assumption about data distribution in computer systems, which states that if a piece of data has been used, it is more likely to be used again in the future, along with adjacent data. For example, if we are currently accessing file / A / a1, we believe there is a higher probability that we will continue to access / A / a1 itself and other files under / A, such as / A / a2, in the future.
[0069] 4) Remote procedure call (RPC) is commonly used in distributed services to facilitate clients initiating requests for execution on a remote end.
[0070] 5) Socket: A socket is an abstraction of an endpoint for bidirectional communication between application processes on different hosts in a network. It is used as a basic method for network communication.
[0071] Distributed file systems allow files to be shared across multiple hosts over a network, enabling multiple users on multiple machines to share files and storage space. Metadata and data have different characteristics. Metadata needs to be accessed frequently, but regardless of the file size, its metadata size is generally similar and relatively small. Data, on the other hand, is the actual data stored in the file, and its space occupied is approximately the same as the file size, which may occupy a large amount of storage space. Therefore, distributed file systems generally manage metadata and data separately. Data storage is similar, and it is generally distributed to different physical nodes according to a certain algorithm, thereby utilizing the storage space of different physical nodes. Since metadata is file control information, data consistency needs to be guaranteed. Current distributed file systems can handle metadata in two ways: (1) Single physical node maintains file metadata, such as GFS, HDFS, LustreFS, etc. This method has a single point of bottleneck, that is, the system performance is limited by the performance of the metadata node. (2) Metadata is distributed to different physical nodes according to a certain distribution method (such as CephFS based on subtree partitioning).
[0072] Current distributed file systems primarily address the issue of insufficient storage space on a single machine, but their metadata throughput is relatively low. As mentioned earlier, systems like GFS and HDFS, which maintain metadata using a single physical node, suffer from single-point bottlenecks. CephFS can distribute metadata across different nodes, alleviating metadata throughput issues, but because its distribution method is based on subtree partitioning, it still exhibits single-point performance bottlenecks when data is highly localized.
[0073] In the era of the booming development of distributed file systems, the performance bottleneck of metadata has not been a particularly important issue. With the development of AI and large language models, massive amounts of files are used in the training and inference process of models (such as multimodal models, where a modality is usually used as part of the input in the form of a file; in addition, the training of large language models depends on massive amounts of high-quality data). These files may not be very large in size, but they are numerous. Therefore, improving the throughput of metadata is an urgent problem to be solved.
[0074] To address the aforementioned problems, this application provides a data processing method, referring to... Figure 3 , Figure 3 A flowchart illustrating a data processing method provided in this application embodiment includes:
[0075] 301. Based on multiple first requests, a target request is obtained, wherein the first request is an operation request for metadata, the first request carries the path of the metadata to be operated, and the multiple first requests are requests of the same operation type, and the operation path included in the target request is an aggregation of the paths carried by the multiple first requests.
[0076] The execution entity in step 301 can be a metadata node. In some scenarios, multiple clients initiate requests concurrently. The client first requests metadata information from the metadata node (depending on the metadata service architecture, there may be only one metadata node, or the client may request the corresponding metadata node according to predefined distribution rules). Then, the metadata node can obtain multiple first requests.
[0077] For example, a client can send a request to a metadata node agent via a socket and a private protocol.
[0078] Each first request can be an operation request for metadata, carrying the path of the metadata to be operated on, and multiple first requests can be requests of the same operation type.
[0079] For example, the operation type of multiple first requests can all be the creation of metadata.
[0080] After obtaining multiple first requests, the operation parameters (paths) for metadata contained in the multiple first requests can be collected to obtain a merged request (target request). During execution, the target request can be executed as a single request, thereby improving the system's throughput for metadata requests and thus enhancing the processing performance for metadata.
[0081] Requests with the same operation type (optionally, the operation type can be one of several preset types, such as Mkdir, Create, Stat, etc.) can be merged.
[0082] In one possible implementation, the aggregation of the paths carried by the plurality of first requests includes a first character, a second character, and a separator; wherein the first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
[0083] In this embodiment, the transmitted information can be specially processed, and the content follows a private communication protocol format. The specific form of the private communication protocol can be customized, but it must include two parts: (1) the operation type of the metadata, which the metadata node agent needs to allocate queues through its content, and therefore should be easy to parse; (2) the operation parameters of the metadata (e.g., path), which the metadata node agent does not need to know its content, but needs to concatenate or split it, and therefore should be easy to concatenate and split. An example could be as follows: Figure 4 As shown.
[0084] In one possible implementation, when there is no free queue, the plurality of first requests are placed in a waiting list; when there is a free queue, the paths carried by the plurality of first requests are aggregated to obtain the target request; then, the target request can be executed in a single request execution process through the free queue.
[0085] For example, the metadata node proxy determines the request type: (1) If the request supports shared computing (e.g., Mkdir, Create, Stat, etc. in the overall architecture diagram), it is placed in the corresponding queue. If the queue is not in the Task list at this time, the entire queue is inserted into the Task list; (2) For others, it is directly inserted into the Task list. The Task list management thread can monitor in the background, take an idle proxy thread from the proxy thread pool (if there is none, it will block and wait), and take a Task from the Task list to assign to the thread.
[0086] These two steps belong to different modules in implementation, but they need to work together, so they are explained together. A clear example can be found here. Figure 5 .from Figure 5 As can be seen, for operations that can be merged between different clients (such as Create in the figure), the metadata processing proxy first merges them into a single Create request containing two sets of parameters and sends it to the metadata processing thread pool. The processing thread pool then returns two sets of processing results after processing, and the metadata processing proxy further splits the results and returns them to the corresponding clients. For operations that cannot be merged (such as Rmdir in the figure), the processing proxy directly forwards them to the processing thread pool and then directly forwards the results.
[0087] To effectively balance latency and throughput, this application provides an adaptive aggregation method, see [link to relevant documentation]. Figure 6 Taking Create as an example, this method follows the following process:
[0088] 1) When the first request arrives, insert the request into the Create queue and insert the entire Create queue into the Task linked list. For easy distinction, this queue will be referred to as Create queue 1.
[0089] 2) When subsequent requests arrive, Create queue 1 is already in the Task linked list, so you only need to insert the request into Create queue 1.
[0090] 3) Wait for an idle agent thread to appear, retrieve Create queue 1 from the Task linked list, and hand it over to the agent thread for processing.
[0091] 4) Create a new Create queue 2. If a new Create request arrives later, insert it into Create queue 2 and repeat the process from 1) to 4).
[0092] The key feature of this scheme is that aggregation occurs in step 2), specifically between a new request and an idle proxy thread. Therefore, when the distributed file system is under low pressure, idle proxy threads are always present, and step 2) is skipped, allowing new requests to be processed immediately without introducing additional latency. Conversely, when the distributed file system is under high pressure, the waiting time for idle proxy threads increases, leading to more requests being aggregated in step 2, thus better utilizing subsequent shared computation to increase system throughput. In summary, this aggregation strategy balances latency and throughput. Under low pressure, throughput requirements can always be met, and latency becomes more critical; this aggregation strategy minimizes latency as much as possible. Under high pressure, if throughput is low, latency cannot be improved due to request backlog; in this case, throughput becomes more important, and this aggregation strategy can be combined with shared computation to enhance throughput.
[0093] A task may contain multiple requests, which the proxy thread concatenates into a single metadata request and sends to an idle thread in the metadata node processing thread pool.
[0094] Based on the private protocol described in step 1), metadata operation parameters should be easy to concatenate. Figure 4 Taking the protocol shown as an example, if you need to merge different Create requests, simply place the paths of the Create requests into the same string sequentially.
[0095] A proxy deployed on the server side aggregates metadata requests arriving concurrently from multiple clients, and then shares the basis for subsequent computing, enabling aggregation that adaptively balances latency and throughput based on metadata request pressure.
[0096] 302. Execute the target request during a single request execution process to obtain the execution results of the plurality of first requests.
[0097] In one possible implementation, before executing the target request during a request execution process, the target request may be treated as a request to perform request execution preparation; wherein the request execution preparation process includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
[0098] The preparation phase is an unavoidable overhead for a single execution. For example, if a distributed file system is implemented as a distributed database, the preparation phase includes network transmission, SQL parsing, transaction management, and resource management. However, although these overheads are unavoidable for a single execution, if an execution contains multiple requests, these overheads can be amortized across the average time of each request, which is one of the benefits of shared computing.
[0099] In one possible implementation, before executing the target request during a single request execution process, the target request may be treated as a single request for target request path resolution.
[0100] In one possible implementation, the paths carried by the plurality of first requests include a first path and a second path, the first path and the second path including a common directory node, and the directory node is parsed only once during the execution path resolution.
[0101] After preparation is complete, the shared path resolution phase begins. Distributed file systems need to resolve directories level by level, starting from the root node. If an execution contains only one request, each node on the path needs to access shared resources (such as the underlying file system, database tables, etc., depending on the distributed file system implementation). However, if an execution contains multiple requests, shared path resolution can reduce the number of accesses to shared resources, thereby reducing lock contention and improving system throughput. See an example below. Figure 7 Suppose we need to resolve three paths: / a / 1, / a / 2, and / a / 3. If we resolve them separately, we need to access the shared resource 9 times. If we build a local path parse tree in one execution, we only need to access the shared resource 5 times.
[0102] By using the above method, redundant computations between multiple concurrent metadata operations are eliminated by merging shared execution, significantly improving the maximum throughput of metadata. In the test, the maximum IOPS of Create was increased by about 650%, and the maximum IOPS of Stat was increased by about 770%.
[0103] In one possible implementation, the paths carried by the plurality of first requests are sorted according to a preset rule during the execution path resolution, and the directory nodes included on the paths carried by the plurality of first requests are locked according to the sorting.
[0104] To prevent local path parse tree failures, embodiments of this application can add shared locks to intermediate nodes. Target nodes are generally also locked with shared locks, but some operations (such as renaming) require directory access, which may require exclusive locks on the target node, potentially leading to deadlocks. For example... Figure 8 In the context of Rename( / a; / b / 2) and Rename( / b; / a / 1) being performed simultaneously.
[0105] To address this issue, the embodiments of this application can follow the following locking order: (1) for a single path, locking begins from the root node downwards; (2) if an operation involves multiple paths, the paths are sorted according to a special lexicographical order ( / is less than any non-zero character), and locked one by one according to the rules of a single path. It has been rigorously proven that this locking order will never result in a deadlock.
[0106] In one possible implementation, the M first requests, including the target request, are executed in the same batch. If the first N first requests succeed and the (N+1)th first request fails, the N first requests can be re-executed, and the (N+1)th first request can be skipped. The M second requests following the M first requests are executed as requests in the same batch.
[0107] Batch execution refers to the batch execution of core code, excluding execution preparation and path resolution. For example, if the underlying distributed file system is a distributed database, this refers to inserting data into a table in the database. If the underlying implementation does not support batch execution, it can be achieved by running the core code in a loop. Errors in the business logic of a single execution should not affect each other. If the underlying implementation throws an error during a single execution, it can be caught using try-catch. If a single execution incurs additional execution preparation overhead, it can be mitigated by splitting it into smaller batches and retrying. For example... Figure 9In this process, multiple requests are executed in batches of 8. When an error occurs while executing the fourth request, the execution of the first 4 requests is canceled due to the try-catch block. Therefore, the first 3 requests are re-executed (which will definitely succeed). Then, starting from the 5th request, 8 requests are fetched again to start a new loop.
[0108] After obtaining the execution result, it can be split into parts, and the split information can be returned to the client. For example, it can be returned to the corresponding client via sockets and private protocols. This step must ensure that the returned execution result corresponds one-to-one with the request to avoid returning the execution result to the wrong client.
[0109] The following describes a data processing apparatus provided in the embodiments of this application from the perspective of the apparatus itself, referring to... Figure 10 , Figure 10 This application provides a schematic diagram of the structure of a data processing apparatus, as shown in the embodiments below. Figure 10 As shown, an embodiment of this application provides a data processing apparatus 1000 comprising:
[0110] The merging module 1001 is used to obtain a target request based on multiple first requests. The first requests are operation requests for metadata. The first requests carry the path of the metadata to be operated on, and the multiple first requests are requests of the same operation type. The operation path included in the target request is an aggregation of the paths carried by the multiple first requests.
[0111] The execution module 1002 is used to execute the target request during a single request execution process to obtain the execution results of the plurality of first requests.
[0112] In one possible implementation, the aggregation of the paths carried by the plurality of first requests includes a first character, a second character, and a separator;
[0113] The first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
[0114] In one possible implementation, the merging module is specifically used for:
[0115] If there is no free queue, place the plurality of first requests in the waiting list;
[0116] When an idle queue exists, the paths carried by the multiple first requests are aggregated to obtain the target request;
[0117] The execution module is specifically used for:
[0118] The target request is executed during a single request execution process using the idle queue.
[0119] In one possible implementation, before executing the target request during a single request execution, the execution module is further configured to:
[0120] The target request is treated as a request to perform request execution preparation; wherein the request execution preparation process includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
[0121] In one possible implementation, before executing the target request during a single request execution, the execution module is further configured to:
[0122] The target request is treated as a single request, and path resolution is performed on the target request.
[0123] In one possible implementation, the paths carried by the plurality of first requests include a first path and a second path, the first path and the second path including a common directory node, and the directory node is parsed only once during the execution path resolution.
[0124] In one possible implementation, the paths carried by the plurality of first requests are sorted according to a preset rule during the execution path resolution, and the directory nodes included on the paths carried by the plurality of first requests are locked according to the sorting.
[0125] In one possible implementation, the M first requests, including the target request, are requests executed in the same batch, and the execution module is further configured to:
[0126] If the first N first requests out of the M first requests succeed and the (N+1)th first request fails, the N first requests are re-executed, and the (N+1)th first request is skipped.
[0127] The M second requests following the M first requests are executed as a single batch of requests.
[0128] This application also provides a computing device 1000. For example... Figure 11 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.
[0129] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).
[0130] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0131] Memory 1006 may include volatile memory, such as random access memory (RAM). Memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). Memory 1006 stores executable program code, which processor 1004 executes to implement the methods performed by the aforementioned data processing device. Specifically, memory 1006 stores instructions for the methods executed by the data processing device (e.g., merging module and execution module).
[0132] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.
[0133] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0134] like Figure 12 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store instructions for methods executed by data processing devices (e.g., merging modules and execution modules).
[0135] In some possible implementations, one or more computing devices 1000 in the computing device cluster may also be used to execute partial instructions of the methods executed by the data processing apparatus (e.g., merging module and execution module). In other words, a combination of one or more computing devices 1000 can jointly execute the instructions of the methods executed by the data processing apparatus (e.g., merging module and execution module).
[0136] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions for executing some functions of the heterogeneous database system 100.
[0137] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 13 One possible implementation is shown. For example... Figure 13 As shown, the two computing devices 120A and 120B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 116 in computing device 120A may store instructions for executing the functions of the merging module. Simultaneously, the memory 116 in computing device 120B may store instructions for executing the functions of the execution module. Alternatively, the memory 116 in computing device 120A may store instructions for executing a portion of the functions of the first database system. Simultaneously, the memory 116 in computing device 120B may store instructions for executing another portion of the functions of the first database system.
[0138] It should be understood that Figure 13 The functions of computing device 120A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 120B can also be performed by multiple computing devices.
[0139] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 12 and Figure 13 The connection method of the computing device cluster. The difference is that the memory 116 of one or more computing devices in the computing device cluster can store the same instructions for executing the template generation method.
[0140] In some possible implementations, the memory 116 of one or more computing devices in the computing device cluster may also store partial instructions for executing the template generation method. In other words, a combination of one or more computing devices can jointly execute the instructions for executing the template generation method.
[0141] It should be noted that the memory 116 in different computing devices within the computing device cluster can store different instructions for executing parts of the template generation method. That is, the instructions stored in the memory 116 of different computing devices can implement the functions of one or more modules in the second database system and the first database system.
[0142] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the methods described above for database system execution, training sample generation, and model training methods applied to the optimization problem solving system 100.
[0143] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the methods of the database system described above, the training sample generation method, and the model training method.
[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data processing method, characterized in that, The method includes: Based on multiple first requests, a target request is obtained. The first requests are operation requests for metadata. Each first request carries the path of the metadata to be operated on, and the multiple first requests are requests of the same operation type. The operation path included in the target request is an aggregation of the paths carried by the multiple first requests. The target request is executed during a single request execution process, resulting in the execution results of the plurality of first requests.
2. The method according to claim 1, characterized in that, The aggregation of paths carried by the multiple first requests includes a first character, a second character, and a separator; The first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
3. The method according to claim 1 or 2, characterized in that, The step of obtaining the target request based on multiple first requests includes: If there is no free queue, place the plurality of first requests in the waiting list; When an idle queue exists, the paths carried by the multiple first requests are aggregated to obtain the target request; The execution of the target request during a single request execution includes: The target request is executed during a single request execution process using the idle queue.
4. The method according to any one of claims 1 to 3, characterized in that, Before executing the target request during a single request execution, the method further includes: The target request is treated as a request to perform request execution preparation; wherein the request execution preparation process includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
5. The method according to any one of claims 1 to 4, characterized in that, Before executing the target request during a single request execution, the method further includes: The target request is treated as a single request, and path resolution is performed on the target request.
6. The method according to claim 5, characterized in that, The paths carried by the plurality of first requests include a first path and a second path, the first path and the second path include a common directory node, and the directory node is parsed only once during the execution path parsing.
7. The method according to claim 5 or 6, characterized in that, Furthermore, during the execution path resolution, the paths carried by the plurality of first requests are sorted according to a preset rule, and the directory nodes included in the paths carried by the plurality of first requests are locked according to the sorting.
8. The method according to any one of claims 1 to 7, characterized in that, The method further includes M first requests, including the target request, which are executed in the same batch. If the first N first requests out of the M first requests succeed and the (N+1)th first request fails, the N first requests are re-executed, and the (N+1)th first request is skipped. The M second requests following the M first requests are executed as a single batch of requests.
9. A data processing apparatus, characterized in that, The device includes: The merging module is used to obtain a target request based on multiple first requests. The first requests are operation requests for metadata. The first requests carry the path of the metadata to be operated on, and the multiple first requests are requests of the same operation type. The operation path included in the target request is an aggregation of the paths carried by the multiple first requests. An execution module is used to execute the target request during a single request execution process to obtain the execution results of the plurality of first requests.
10. The apparatus according to claim 9, characterized in that, The aggregation of paths carried by the multiple first requests includes a first character, a second character, and a separator; The first character is used to indicate the operation type, the second character is used to indicate the plurality of paths, and the separator is used to separate different paths among the plurality of paths.
11. The apparatus according to claim 9 or 10, characterized in that, The merging module is specifically used for: If there is no free queue, place the plurality of first requests in the waiting list; When an idle queue exists, the paths carried by the multiple first requests are aggregated to obtain the target request; The execution module is specifically used for: The target request is executed during a single request execution process using the idle queue.
12. The apparatus according to any one of claims 9 to 11, characterized in that, Before executing the target request during a single request execution process, the execution module is further configured to: The target request is treated as a request to perform request execution preparation; wherein the request execution preparation process includes at least one of the following: network transmission, SQL parsing, transaction management, and resource management.
13. The apparatus according to any one of claims 9 to 12, characterized in that, Before executing the target request during a single request execution process, the execution module is further configured to: The target request is treated as a single request, and path resolution is performed on the target request.
14. The apparatus according to claim 13, characterized in that, The paths carried by the plurality of first requests include a first path and a second path, the first path and the second path include a common directory node, and the directory node is parsed only once during the execution path parsing.
15. The apparatus according to claim 13 or 14, characterized in that, Furthermore, during the execution path resolution, the paths carried by the plurality of first requests are sorted according to a preset rule, and the directory nodes included in the paths carried by the plurality of first requests are locked according to the sorting.
16. The apparatus according to any one of claims 9 to 15, characterized in that, The execution module, which includes the target request and M first requests, is configured to execute requests in the same batch. If the first N first requests out of the M first requests succeed and the (N+1)th first request fails, the N first requests are re-executed, and the (N+1)th first request is skipped. The M second requests following the M first requests are executed as a single batch of requests.
17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.
18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.
19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.