Distributed computing method and system based on ray

Abstract

The invention discloses a ray-based distributed computing method and system. The method comprises the following steps: acquiring all data, and calculating a fragment id of each piece of data according to a total fragment number; writing the data of the same fragment id into the LMDB file corresponding to the id identifier; all the LMDB files are distributed to the corresponding ray nodes in parallel; performing parallel calculation on data in each LMDB file in each ray node according to the issued calculation task; and collecting calculation results of the ray nodes and carrying out merging processing. According to the method, all data are fragmented and then written into a plurality of different LMDB files, the problem of write lock competition caused by the fact that a plurality of threads are written into the same file at the same time in the prior art is avoided, and the query delay is reduced to 28 ms through fragmented storage; according to the method, all the LMDB files are distributed to the corresponding ray nodes in parallel, each ray node only stores part of data, the memory occupancy rate is low, each ray node only processes the calculation task of the stored data, and waste of calculation resources is avoided.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a ray-based distributed computing method and system. Background Art

[0002] In existing technologies, data is directly stored without being sharded during the data storage phase: traditional LMDB systems write all data directly into a single storage pool, causing lock conflicts when multiple threads concurrently write to the same file (tests show that lock wait time accounts for 35% of the total write time). When processing 10TB of key-value data, unsharded storage causes query delays of up to 120ms. During the computation and distribution phase, all data is distributed to each ray node, forcing each ray node to load a full copy of the data, resulting in high memory usage, repeated computation of large amounts of data, and a waste of computing resources. Summary of the Invention

[0003] The embodiment of the present invention provides a ray-based distributed computing method and system to solve the problems in the prior art of high query latency due to unsharded storage, high memory usage due to all data being distributed to each ray node, and waste of computing resources.

[0004] To achieve the above-mentioned purpose, on the one hand, the present invention provides a distributed computing method based on ray, which includes: S1, obtaining all data, and calculating the shard ID of each data according to the total number of shards; S2, writing data with the same shard ID into the LMDB file of the corresponding ID identifier; S3, distributing all LMDB files in parallel to the corresponding ray nodes; S4, performing parallel computing on the data in each LMDB file in each ray node according to the issued computing tasks; S5, collecting the computing results of each ray node and merging them.

[0005] Optionally, the S1 includes: obtaining all data, taking each data as the current data in turn, and calculating the shard id of the current data according to the total number of shards: S11, calculating the key value of the current data through a hash algorithm to obtain the current hash value; S12, initializing the intermediate value b and the shard id of the current data; S13, assigning the integer part of the shard id of the current data to the intermediate value b to update the intermediate value b; S14, performing a mixed calculation on the current hash value and a preset magic number constant to update the current hash value; S15, performing an offset operation on the updated current hash value according to the updated intermediate value b to obtain the updated shard id of the current data; S16, judging whether the updated shard id of the current data is less than the total number of shards; if so, looping S13~S15 until the updated shard id of the current data is greater than or equal to the total number of shards and terminating the loop; S17, taking the latest intermediate value b generated in the last loop as the final shard id of the current data.

[0006] Optionally, S3 includes: taking each LMDB file as the current LMDB file in turn: taking the modulus of the preset total number of ray nodes according to the ID identifier of the current LMDB file to determine the target ray node to which the current LMDB file is pre-allocated; and transmitting each LMDB file in parallel to the corresponding target ray node via tcp file transfer.

[0007] Optionally, before S2, the step includes: creating a number of LMDB files equal to the total number of shards, and associating a unique ID identifier with each LMDB file; and numbering the ID identifiers of all LMDB files consecutively in an integer sequence in the range of [0, N-1], where N is the total number of shards.

[0008] Optionally, all data is written in parallel to the corresponding LMDB files.

[0009] On the other hand, the present invention provides a distributed computing system based on Ray, which includes: a shard ID calculation unit for obtaining all data and calculating the shard ID of each data based on the total number of shards; a writing unit for writing data with the same shard ID into the LMDB file of the corresponding ID identifier; a distribution unit for distributing all LMDB files in parallel to the corresponding Ray nodes; a parallel computing unit for performing parallel computing on the data in each LMDB file in each Ray node according to the issued computing tasks; and a collection and merging unit for collecting the computing results of each Ray node and performing merging processing.

[0010] Optionally, the shard ID calculation unit includes: obtaining all data, taking each data as the current data in turn, and calculating the shard ID of the current data according to the total number of shards: a hash calculation subunit, used to calculate the key value of the current data through a hash algorithm to obtain the current hash value; an initialization subunit, used to initialize the intermediate value b and the shard ID of the current data; a first update subunit, used to assign the integer part of the shard ID of the current data to the intermediate value b to update the intermediate value b; a second update subunit, used to perform a mixed calculation on the current hash value and a preset magic number constant to update the current hash value; a third update subunit, used to offset the updated current hash value according to the updated intermediate value b to obtain the updated shard ID of the current data; a judgment subunit, used to judge whether the updated shard ID of the current data is less than the total number of shards; if so, looping the first update subunit, the second update subunit, and the third update subunit until the updated shard ID of the current data is greater than or equal to the total number of shards and terminating the loop; an output subunit, used to use the latest intermediate value b generated in the last loop as the final shard ID of the current data.

[0011] Optionally, the distribution unit includes: a determination subunit, used to take each LMDB file as the current LMDB file in turn: modulo the preset total number of ray nodes according to the ID identifier of the current LMDB file, and determine the target ray node for pre-allocation of the current LMDB file; a transmission subunit, used to transmit each LMDB file in parallel to the corresponding target ray node via TCP file transmission.

[0012] Optionally, it also includes: a creation unit, used to: create a number of LMDB files equal to the total number of shards, each LMDB file is associated with a unique ID identifier; the ID identifiers of all LMDB files are numbered consecutively in an integer sequence, and the numbering range is [0, N-1], where N is the total number of shards.

[0013] Optionally, all data is written in parallel to the corresponding LMDB files.

[0014] Beneficial effects of the present invention:

[0015] The present invention provides a Ray-based distributed computing method and system, wherein the method shards all data and writes them in parallel to multiple different LMDB files, avoiding the write lock contention problem caused by multiple threads writing the same file simultaneously in the prior art. The sharded storage reduces query latency to 28ms. The method distributes all LMDB files in parallel to corresponding Ray nodes, each Ray node only stores part of the data, and each Ray node only loads the corresponding data, resulting in low memory usage. In addition, each Ray node only processes the computing tasks of the stored data, avoiding waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a ray-based distributed computing method provided by an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of calculating the shard ID of current data provided by an embodiment of the present invention;

[0018] Figure 3 is a schematic structural diagram of a ray-based distributed computing system provided by an embodiment of the present invention;

[0019] Figure 4 It is a structural diagram of the shard ID calculation unit provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0021] In existing technology, data is stored directly without sharding during the data storage phase. Traditional LMDB systems write all data directly to a single storage pool, leading to lock conflicts when multiple threads concurrently write to the same file. LMDB is a single-write, multiple-read database, meaning only one writer can hold the write lock while other writers must wait. If multiple threads attempt to write to the same LMDB file simultaneously, write lock contention occurs; other threads must queue for the lock, causing lock conflicts. When processing 10TB of key-value data, unsharded storage results in query latency of up to 120ms. During the computation distribution phase, all data is distributed to each ray node, forcing each ray node to load a full copy of the data, resulting in high memory usage and repeated computation of large amounts of data, wasting computing resources.

[0022] Therefore, the present invention provides a distributed computing method based on ray, Figure 1 This is a flow chart of a ray-based distributed computing method provided by an embodiment of the present invention; Figure 1 As shown, the method includes:

[0023] S1. Get all the data and calculate the shard ID of each data based on the total number of shards;

[0024] In an optional embodiment, the S1 includes:

[0025] Get all the data, take each data as the current data in turn, and calculate the shard ID of the current data based on the total number of shards. Figure 2 This is a flowchart of calculating the shard ID of the current data provided by an embodiment of the present invention; Figure 2 As shown:

[0026] S11. Calculate the key value of the current data through a hash algorithm to obtain the current hash value;

[0027] S12. Initialize the intermediate value b and the shard ID of the current data;

[0028] S13. Assign the integer part of the shard ID of the current data to the intermediate value b to update the intermediate value b;

[0029] S14, performing mixed calculation on the current hash value and a preset magic number constant to update the current hash value;

[0030] S15. Perform an offset operation on the updated current hash value according to the updated intermediate value b to obtain the updated shard ID of the current data;

[0031] S16. Determine whether the updated shard ID of the current data is less than the total number of shards; if so, loop through S13 to S15 until the updated shard ID of the current data is greater than or equal to the total number of shards, terminating the loop;

[0032] S17. The latest intermediate value b generated in the last cycle is used as the final shard ID of the current data.

[0033] In this application, we need to shard all data. We use the LMDB file system for data storage. The advantage of the LMDB file system is that data is stored in a key / value format, which makes random reads very efficient. When sharding all data, we first perform a hash operation on the key value of each data item to obtain a hash value for each data item. Then, based on the hash value, we use a specific algorithm to determine the shard ID for each data item.

[0034] The following is described through a specific embodiment:

[0035] Take the total number of shards = 8 and the key value of the current data = "test_key_to_partition" as an example:

[0036] First, serialize the key value of the current data to get the current serialized key value serialized_key:

[0037] serialized_key ='\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00\x8c\x15test_key_to_partition\x94.'

[0038] Hash the current serialized key value serialized_key and convert it into an integer to get the current hash value hashed_key:

[0039] hashed_key = 462316576431466514995110735505649961797345757849

[0040] Initialize the intermediate value b to -1 and the shard id of the current data to 0; and define the magic number constant magic_num = 2862933555777941757;

[0041] At this point, the shard ID of the current data is less than the total number of shards, and the first round of iteration begins:

[0042] Update the intermediate value b = int(id) = 0

[0043] Update the current hash value hashed_key = ((hashed_key *magic_num) + 1) &0xFFFFFFFFFFFFFFFF = 2935928542770030902

[0044] Update the shard id of the current data = float(b + 1) * (float(1 << 31) / float((hashed_key >> 33) + 1)) = 6.283103883506511

[0045] At this point, the shard ID of the current data is less than the total number of shards, and the second round of iteration begins:

[0046] Update the intermediate value b = int(id) = 6

[0047] Update the current hash value hashed_key = ((hashed_key *magic_num) + 1) &0xFFFFFFFFFFFFFFFF = 9650281935219956319

[0048] Update the shard id of the current data = float(b + 1) * (float(1 << 31) / float((hashed_key >> 33) + 1)) = 13.380666937182063

[0049] At this point, the shard ID of the current data is greater than the total number of shards, so the iteration ends and the latest intermediate value b (i.e., b is 6) generated by the last loop is returned as the final shard ID of the current data.

[0050] It should be noted that in the above embodiment, the shard IDs of all data include: shard ID 0, shard ID 1, shard ID 2, shard ID 3, shard ID 4, shard ID 5, shard ID 6, and shard ID 7.

[0051] S2. Write the data of the same shard ID into the LMDB file of the corresponding ID identifier;

[0052] In an optional embodiment, before S2, the following steps are included:

[0053] Create a number of LMDB files equal to the total number of shards, and associate a unique ID identifier with each LMDB file; the ID identifiers of all LMDB files are numbered consecutively as integers in the range [0, N-1], where N is the total number of shards.

[0054] LMDB (Lightning Memory-Mapped Database) is a memory-mapped database that is used to efficiently store and access large amounts of data.

[0055] In the above embodiment, eight LMDB files are created, and the ID identifiers of all LMDB files are consecutively numbered according to an integer sequence, namely 0, 1, 2, 3, 4, 5, 6, and 7.

[0056] The shard IDs of all data correspond one-to-one to the IDs of all files. Data with the same shard ID is written to the LMDB file with the corresponding ID identifier.

[0057] In an optional embodiment, all data is written in parallel to the corresponding LMDB files. That is, different LMDB files are written in parallel by multiple threads to improve writing efficiency.

[0058] The method of the present application shards all data and writes them in parallel to multiple different LMDB files, avoiding the write lock contention problem caused by multiple threads writing to the same file at the same time in the prior art. Sharded storage reduces query latency to 28ms.

[0059] S3, distribute all LMDB files to the corresponding ray nodes in parallel;

[0060] In an optional embodiment, the S3 includes:

[0061] Take each LMDB file as the current LMDB file in turn: modulo the total number of preset ray nodes according to the id identifier of the current LMDB file, and determine the target ray node for pre-allocation of the current LMDB file;

[0062] Each LMDB file is transferred in parallel to the corresponding target ray node via TCP file transfer.

[0063] The following is described through a specific embodiment:

[0064] Take the example of 8 LMDB files and 3 ray nodes:

[0065] The ID identifiers of the LMDB files are 0, 1, 2, 3, 4, 5, 6, and 7 respectively; the ray nodes are ray node 0, ray node 1, and ray node 2 respectively.

[0066] The LMDB file's id identifier 0 % 3=0 is assigned to the target ray node 0;

[0067] The LMDB file's ID identifier 1 % 3 = 1 is assigned to the target ray node 1;

[0068] The LMDB file's ID identifier 2 % 3 = 2 is assigned to the target ray node 2;

[0069] The LMDB file's ID identifier 3 % 3=0 is assigned to the target ray node 0;

[0070] The LMDB file's ID identifier 4 % 3=1 is assigned to the target ray node 1;

[0071] The LMDB file's ID identifier 5 % 3 = 2 is assigned to the target ray node 2;

[0072] The LMDB file's id identifier 6 % 3=0 is assigned to the target ray node 0;

[0073] The LMDB file's ID identifier 7 % 3=1 is assigned to the target ray node 1.

[0074] By taking the modulus of the total number of Ray nodes, multiple LMDB files can be evenly distributed to each Ray node; each Ray node only stores part of the data, and each Ray node only loads the corresponding data, so the memory usage is low.

[0075] S4. Perform parallel computation on the data in each LMDB file in each ray node according to the computing task issued.

[0076] S5. Collect the calculation results of each ray node and merge them.

[0077] In this application, each ray node only processes the computing tasks of stored data, realizing automatic diversion of computing tasks, avoiding excessive load on a certain node, and improving the overall computing efficiency of the system.

[0078] Figure 3 This is a schematic diagram of the structure of a ray-based distributed computing system provided by an embodiment of the present invention. Figure 3 As shown, the system includes:

[0079] The shard ID calculation unit 201 is used to obtain all data and calculate the shard ID of each data according to the total number of shards;

[0080] A writing unit 202 is used to write data with the same shard ID into the LMDB file of the corresponding ID identifier;

[0081] The distribution unit 203 is used to distribute all LMDB files to corresponding ray nodes in parallel;

[0082] A parallel computing unit 204 is configured to perform parallel computing on the data in each LMDB file in each ray node according to the computing task issued;

[0083] The collecting and merging unit 205 is used to collect the calculation results of each ray node and perform merging processing.

[0084] In an optional embodiment, Figure 4 This is a schematic diagram of the structure of the shard ID calculation unit provided by the embodiment of the present invention. Figure 4 As shown:

[0085] The shard ID calculation unit 201 includes:

[0086] Get all the data, take each data as the current data in turn, and calculate the shard ID of the current data based on the total number of shards:

[0087] The hash calculation subunit 2011 is used to calculate the key value of the current data through a hash algorithm to obtain the current hash value;

[0088] Initialization subunit 2012, used to initialize the intermediate value b and the shard ID of the current data;

[0089] The first updating subunit 2013 is configured to assign the integer portion of the shard ID of the current data to the intermediate value b to update the intermediate value b;

[0090] A second updating subunit 2014 is configured to perform a mixed calculation on the current hash value and a preset magic number constant to update the current hash value;

[0091] The third updating subunit 2015 is used to perform an offset operation on the updated current hash value according to the updated intermediate value b to obtain an updated shard ID of the current data;

[0092] The judgment subunit 2016 is used to determine whether the shard ID of the updated current data is less than the total number of shards; if so, the first updating subunit, the second updating subunit, and the third updating subunit are executed in a loop until the shard ID of the updated current data is greater than or equal to the total number of shards, terminating the loop;

[0093] The output subunit 2017 is used to use the latest intermediate value b generated by the last cycle as the final shard ID of the current data.

[0094] In an optional embodiment, the distribution unit 203 includes:

[0095] A determination subunit is used to sequentially use each LMDB file as the current LMDB file: modulo the total number of preset ray nodes according to the id identifier of the current LMDB file, and determine the target ray node for pre-allocation of the current LMDB file;

[0096] The transmission subunit is used to transmit each LMDB file in parallel to the corresponding target ray node through TCP file transmission.

[0097] In an optional embodiment, the system further includes: a creation unit, configured to:

[0098] Create a number of LMDB files equal to the total number of shards, and associate a unique ID identifier with each LMDB file; the ID identifiers of all LMDB files are numbered consecutively as integers in the range [0, N-1], where N is the total number of shards.

[0099] In an optional implementation, all data are written in parallel to corresponding LMDB files.

[0100] The system and method of the present application correspond to each other, and the specific implementation of the system will not be repeated here.

[0101] Beneficial effects of the present invention:

[0102] The present invention provides a Ray-based distributed computing method and system, wherein the method shards all data and writes them in parallel to multiple different LMDB files, avoiding the write lock contention problem caused by multiple threads writing the same file simultaneously in the prior art. The sharded storage reduces query latency to 28ms. The method distributes all LMDB files in parallel to corresponding Ray nodes, each Ray node only stores part of the data, and each Ray node only loads the corresponding data, resulting in low memory usage. In addition, each Ray node only processes the computing tasks of the stored data, avoiding waste of computing resources.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A distributed computing method based on ray, characterized in that: include: S1. Get all the data and calculate the shard ID of each data based on the total number of shards; S2. Write the data of the same shard ID into the LMDB file of the corresponding ID identifier; S3, distribute all LMDB files to the corresponding ray nodes in parallel; S4. Perform parallel computation on the data in each LMDB file in each ray node according to the computing task issued. S5. Collect the calculation results of each ray node and merge them.

2. The method according to claim 1, characterized in that Said S1 comprises: Get all the data, take each data as the current data in turn, and calculate the shard ID of the current data based on the total number of shards: S11. Calculate the key value of the current data through a hash algorithm to obtain the current hash value; S12. Initialize the intermediate value b and the shard ID of the current data; S13. Assign the integer part of the shard ID of the current data to the intermediate value b to update the intermediate value b; S14, performing mixed calculation on the current hash value and a preset magic number constant to update the current hash value; S15. Perform an offset operation on the updated current hash value according to the updated intermediate value b to obtain the updated shard ID of the current data; S16. Determine whether the updated shard ID of the current data is less than the total number of shards; if so, loop through S13 to S15 until the updated shard ID of the current data is greater than or equal to the total number of shards, terminating the loop; S17. The latest intermediate value b generated in the last cycle is used as the final shard ID of the current data.

3. The method according to claim 1, characterized in that The S3 includes: Take each LMDB file as the current LMDB file in turn: modulo the total number of preset ray nodes according to the id identifier of the current LMDB file, and determine the target ray node for pre-allocation of the current LMDB file; Each LMDB file is transferred in parallel to the corresponding target ray node via TCP file transfer.

4. The method according to claim 1, wherein Before said S2, it includes: Create a number of LMDB files equal to the total number of shards, and associate a unique ID identifier with each LMDB file; the ID identifiers of all LMDB files are numbered consecutively as integers in the range [0, N-1], where N is the total number of shards.

5. The method according to claim 4, characterized in that: All data is written in parallel to the corresponding LMDB files.

6. A distributed computing system based on ray, characterized in that, include: The shard ID calculation unit is used to obtain all data and calculate the shard ID of each data based on the total number of shards; The write unit is used to write data with the same shard ID into the LMDB file with the corresponding ID identifier; The distribution unit is used to distribute all LMDB files to the corresponding ray nodes in parallel; The parallel computing unit is used to perform parallel computing on the data in each LMDB file in each ray node according to the computing tasks issued; The collection and merging unit is used to collect the calculation results of each ray node and merge them.

7. The system according to claim 6, characterized in that The shard ID calculation unit includes: Get all the data, take each data as the current data in turn, and calculate the shard ID of the current data based on the total number of shards: The hash calculation subunit is used to calculate the key value of the current data through the hash algorithm to obtain the current hash value; Initialization subunit, used to initialize the intermediate value b and the shard id of the current data; A first updating subunit is configured to assign the integer portion of the shard ID of the current data to the intermediate value b to update the intermediate value b; A second updating subunit, configured to perform mixed calculation on the current hash value and a preset magic number constant to update the current hash value; The third updating subunit is used to perform an offset operation on the updated current hash value according to the updated intermediate value b to obtain an updated shard ID of the current data; A judgment subunit is used to judge whether the shard ID of the updated current data is less than the total number of shards; if so, the first update subunit, the second update subunit, and the third update subunit are executed in a loop until the shard ID of the updated current data is greater than or equal to the total number of shards, terminating the loop; The output subunit is used to use the latest intermediate value b generated by the last cycle as the final shard ID of the current data.

8. The system according to claim 6, wherein: The distribution unit includes: A determination subunit is used to sequentially use each LMDB file as the current LMDB file: modulo the total number of preset ray nodes according to the id identifier of the current LMDB file, and determine the target ray node for pre-allocation of the current LMDB file; The transmission subunit is used to transmit each LMDB file in parallel to the corresponding target ray node through TCP file transmission.

9. The system according to claim 6, wherein: Also includes: Create units for: Create a number of LMDB files equal to the total number of shards, and associate a unique ID identifier with each LMDB file; the ID identifiers of all LMDB files are numbered consecutively as integers in the range [0, N-1], where N is the total number of shards.

10. The system according to claim 9, characterized in that: All data is written in parallel to the corresponding LMDB files.

Images