Data storage method and device, computer equipment and storage medium
By creating and renaming temporary directories on local disks, frequent HTTP requests caused by the lack of renaming function of object stores are solved, and the execution efficiency of distributed jobs is improved.
Patent Information
- Application Number
- CN202411304276.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-31
AI Technical Summary
In big data distributed jobs, object storage lacks renaming function, resulting in frequent HTTP requests being sent, reducing the efficiency of small tasks being executed.
Create a temporary directory on the local disk, execute tasks and write the results to the directory. After completing the task, rename the directory with the job name, and upload the renamed directory to the object store at one time to avoid the temporary file storage and renaming process of object store. .
It reduces the number of interactions between object storage and computing nodes, reduces the number of HTTP requests, and improves the execution efficiency of splitting tasks.
Smart Images
Figure CN119225650A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data storage method, apparatus, computer equipment and storage medium. Background Art
[0002] During the execution of distributed big data jobs, the job is broken down into multiple small tasks and then distributed to different processing units for processing. During the execution of these small tasks, the execution results will be temporarily stored in the "_temporary" directory. When all small tasks are completed, the "_temporary" directory will be renamed to the directory name corresponding to the job execution results (usually the directory name with the "task_" prefix).
[0003] As the amount of stored data continues to increase, object storage has become the preferred storage tool for companies due to its stable performance, easy expansion, and low cost. However, unlike traditional storage systems, object storage does not have a rename function, but uses copy+delete to achieve renaming in disguise. At the same time, object storage uses HTTP communication for interaction. When object storage is used as the storage medium for the execution results of distributed jobs, object storage will frequently send HTTP requests to create temporary directories, write temporary results, and rename directories, resulting in low execution efficiency of small tasks. Summary of the invention
[0004] The present application provides a data storage method, apparatus, computer equipment and storage medium to solve the problem of generating a large number of HTTP requests by using object storage as the storage medium for the execution results of distributed jobs.
[0005] In a first aspect, the present application provides a data storage method, the method comprising:
[0006] Get multiple split tasks after different distributed jobs are split;
[0007] Create a temporary directory corresponding to each of the distributed jobs on the local disk;
[0008] Execute each of the split tasks to obtain corresponding task execution results;
[0009] Writing each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs;
[0010] When each of the split tasks belonging to the target distributed job is executed, renaming the temporary directory corresponding to the target distributed job with the job name corresponding to the target distributed job to obtain a renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs;
[0011] The renamed directory corresponding to the target distributed job is uploaded to the object storage for storage.
[0012] In a second aspect, the present application provides a data storage method, the method comprising:
[0013] Receive and save renamed directories corresponding to different distributed jobs uploaded by different computing nodes;
[0014] The different renamed directories corresponding to the same distributed job are aggregated to form the job calculation results.
[0015] In a third aspect, the present application provides a data storage device, the device comprising:
[0016] The acquisition module is used to obtain multiple split tasks after different distributed jobs are split;
[0017] A creation module, used to create a temporary directory corresponding to each of the distributed jobs on a local disk;
[0018] An execution module, used for executing each of the split tasks to obtain corresponding task execution results;
[0019] A writing module, used to write the execution results of each task into a temporary directory corresponding to the distributed job to which the corresponding split task belongs;
[0020] A renaming module, used for renaming a temporary directory corresponding to the target distributed job with a job name corresponding to the target distributed job when each of the split tasks belonging to the target distributed job is executed, so as to obtain a renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs;
[0021] The upload module is used to upload the renamed directory corresponding to the target distributed job to the object storage for storage.
[0022] In a fourth aspect, the present application provides a data storage device, the device comprising:
[0023] A storage module, used to receive and save renamed directories corresponding to different distributed jobs uploaded by different computing nodes;
[0024] The aggregation module is used to aggregate different renamed directories corresponding to the same distributed job to form the job calculation result.
[0025] In a fifth aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data storage method when executing the computer program.
[0026] In a sixth aspect, the present application also provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the above-mentioned data storage method.
[0027] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains multiple split tasks after splitting different distributed jobs; creates a temporary directory corresponding to each of the distributed jobs on the local disk; executes each of the split tasks to obtain the corresponding task execution results; writes each of the task execution results into the temporary directory corresponding to the distributed job to which the corresponding split task belongs; when the execution of each of the split tasks belonging to the target distributed job is completed, renames the temporary directory corresponding to the target distributed job with the job name corresponding to the target distributed job to obtain the renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs; uploads the renamed directory corresponding to the target distributed job to the object storage for storage.
[0028] Based on the above method, the temporary files generated by the computing splitting task are stored in the temporary directory of the local disk of the computing node, and the data storage is renamed in the local disk, and the renamed directory is sent to the object storage at one time, thereby avoiding the storage and renaming process of temporary files by the object storage, reducing the number of interactions between the object storage and the computing node, that is, reducing the HTTP requests of the object storage, improving the execution efficiency of the splitting task, and solving the problem of generating a large number of HTTP requests when using the object storage as the storage medium for the corresponding execution results of distributed jobs. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0031] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0032] Figure 1 An application environment diagram of a data storage method provided in an embodiment of the present application;
[0033] Figure 2 A schematic diagram of a data storage method provided in an embodiment of the present application;
[0034] Figure 3 A schematic diagram of a data storage method provided in an embodiment of the present application;
[0035] Figure 4 A structural block diagram of a data storage device provided in an embodiment of the present application;
[0036] Figure 5 A structural block diagram of a data storage device provided in an embodiment of the present application;
[0037] Figure 6 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0039] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.
[0040] Figure 1 FIG. 1 is an application environment diagram of a data storage method in an embodiment. Figure 1The data storage method is applied to a data storage system. The data storage system includes a computing node 110 and an object storage 120. The computing node 110 and the object storage 120 are connected via a network. The computing node 110 and the object storage 120 can be implemented by independent servers, or integrated into one server.
[0041] In one embodiment, Figure 2 A schematic diagram of a data storage method in an embodiment is shown in FIG. Figure 2 , provides a data storage method. This embodiment mainly applies this method to the above Figure 1 Taking the computing node 110 in FIG. 1 as an example, the data storage method specifically includes the following steps:
[0042] Step S210, obtaining multiple split tasks after different distributed jobs are split.
[0043] Specifically, computing node 110 receives split tasks issued by the scheduling system. The scheduling system decomposes the distributed job into multiple split tasks and then distributes them to different computing nodes 110 for processing. Therefore, computing node 110 will receive multiple split tasks from different distributed jobs after decomposition distributed by the scheduling system. The execution results of all split tasks belonging to the same distributed job are combined to form the execution result of the distributed job.
[0044] Step S220: creating a temporary directory corresponding to each of the distributed jobs on the local disk.
[0045] Specifically, Figure 3 The small task execution node in is the computing node 110, refer to Figure 3 In step ②, a temporary directory corresponding to each distributed job is created in the local disk of the computing node 110. Each distributed job corresponds to a temporary directory, which is recorded as a _temporary directory and is used to store temporary files generated during the execution of the split tasks belonging to the distributed job.
[0046] Step S230, executing each of the split tasks to obtain corresponding task execution results.
[0047] Specifically, the computing node 110 can execute each split task separately through multiple different execution threads, or use one execution thread to execute each split task in sequence, and the specific execution method adopted depends on the configuration of the maximum number of online threads and the hardware configuration of the computing node 110. The execution result of each split task is the task execution result.
[0048] Step S240, writing each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs.
[0049] Specifically, refer to Figure 3 In step ③, the computing node 110 writes the task execution result into the temporary directory corresponding to the distributed job to which the corresponding split task belongs, that is, the task execution results of multiple split tasks belonging to the same distributed job are written into the same temporary directory.
[0050] Step S250, when the execution of each of the split tasks belonging to the target distributed job is completed, the temporary directory corresponding to the target distributed job is renamed with the job name corresponding to the target distributed job to obtain the renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs.
[0051] Specifically, refer to Figure 3 In steps ④ and ⑤, when computing node 110 completes executing each split task belonging to the same distributed job assigned by the scheduling system, it is necessary to rename the temporary directory corresponding to the distributed job, specifically, to the job name corresponding to the distributed job, and to rename it to the real name corresponding to the distributed job, that is, the directory name after the rename is no longer the default directory name, and is generally a directory name with a "task_" prefix.
[0052] Step S260: Upload the renamed directory corresponding to the target distributed job to the object storage 120 for storage.
[0053] Specifically, refer to Figure 3 In step ⑥, computing node 110 sends the renamed directory to object storage 120 for storage at one time, that is, the storage and renaming process of the temporary directory is no longer implemented by object storage 120, thereby saving the interaction process between computing node 110 and object storage 120, and only object storage 120 needs to save the final renamed directory, thereby avoiding the storage and renaming process of temporary files by object storage 120, reducing the number of interactions between object storage 120 and computing node 110, that is, reducing the HTTP requests of object storage 120, improving the execution efficiency of split tasks, and solving the problem of generating a large number of HTTP requests when using object storage 120 as the storage medium for the execution results corresponding to distributed jobs.
[0054] In one embodiment, after obtaining a plurality of split tasks after splitting different distributed jobs, the method further includes:
[0055] Acquire relevant data related to each of the split tasks;
[0056] The executing of each of the split tasks to obtain corresponding task execution results includes:
[0057] The corresponding split tasks are executed based on the relevant data corresponding to each of the split tasks to obtain the task execution results corresponding to each of the split tasks.
[0058] Specifically, relevant data related to each split task can be obtained from the local computing node 110, the source database, or different data storage sources, and then the corresponding split task is executed based on the relevant data. For example, if the split task is to count the total value of a parameter in a specified table, it is necessary to obtain the specified table from the object storage 120 to execute the split task. By obtaining the relevant data of each split task to execute the split task, the execution success rate of the split task is ensured, and the execution success rate of the distributed job is improved.
[0059] In one embodiment, after obtaining a plurality of split tasks after splitting different distributed jobs, the method further includes:
[0060] Sending data requests corresponding to each of the split tasks to the object storage 120;
[0061] Obtain relevant data related to each of the splitting tasks provided by the object storage 120 according to each of the data requests.
[0062] Specifically, after receiving the split task, the computing node 110 may also generate a data request according to the split task, and send the data request to the object storage 120 to request to obtain the relevant data required for executing the split task from the object storage 120, that is, corresponding to Figure 3 Small and medium task execution nodes download data from object storage 120, and object storage 120 is also used to provide storage function for related data to save storage space of computing node 110 and reduce the number of interactive objects of computing node 110, thereby simplifying the computing process of computing node 110 to execute split tasks.
[0063] In one embodiment, creating a temporary directory corresponding to each of the distributed jobs on the local disk includes:
[0064] A temporary directory corresponding to each of the distributed jobs is created on the local disk, and a temporary subdirectory corresponding to each of the split tasks belonging to the same distributed job is created in the temporary directory.
[0065] Specifically, refer to Figure 3In step ②, after creating temporary directories corresponding to each distributed job in the local disk of computing node 110, temporary sub-directories corresponding to each split task belonging to the same distributed job are also created in each temporary directory. The temporary sub-directories are used to store temporary files generated during the execution of the corresponding split tasks, as well as the task execution results generated after the task execution is completed, so as to refine and accurately determine the storage location of the files generated during the execution of each split task and at the end of the execution.
[0066] For example, the scheduling system assigns three split tasks of distributed job A1 to computing node 1101, namely a1, a2, and a3. A temporary directory B1 is created for distributed job A1 in the local disk of computing node 1101, and three temporary sub-directories are created separately for a1, a2, and a3 in the temporary directory B1. The three temporary sub-directories are recorded as b1, b2, and b3 respectively. b1 is used to store temporary files and task execution results generated during the execution of split task a1, b2 is used to store temporary files and task execution results generated during the execution of split task a2, and b3 is used to store temporary files and task execution results generated during the execution of split task a3.
[0067] In one embodiment, writing each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs includes:
[0068] The execution results of each of the tasks are written into a temporary directory corresponding to the distributed job to which the corresponding split task belongs, and into a temporary subdirectory created for the corresponding split task.
[0069] Specifically, refer to Figure 3 In step ③, when a temporary subdirectory corresponding to each split task is created in the temporary directory, the execution results of each split task and the temporary files generated during the execution process can be directly written into the corresponding temporary subdirectories to accurately determine the storage location of the files generated by each split task during the execution process and at the end of the execution, which provides convenience for subsequent data management.
[0070] In one embodiment, a data storage method is provided, which is applied to the object storage 120, and includes:
[0071] Step S310 , receiving and saving renaming directories corresponding to different distributed jobs uploaded by different computing nodes 110 .
[0072] Specifically, refer to Figure 3In step ⑥, the object storage 120 only receives the rename directories corresponding to different distributed jobs provided by different computing nodes 110, that is, different computing nodes 110 may provide different rename directories corresponding to the same distributed job, and each computing node 110 will provide rename directories corresponding to different distributed jobs. The execution process of the split task and the renaming of the execution results are completed on the computing node 110, and the computing node 110 only uploads the renamed rename directory to the object storage 120, thereby saving the object storage 120 the steps of creating a temporary directory, writing temporary results, and renaming the directory, thereby avoiding the tedious renaming steps of using the object storage 120, and reducing the number of HTTP interactions between the object storage 120 and the computing node 110, which can improve the execution efficiency of the split task and reduce the service pressure of the object storage 120.
[0073] Step S320, aggregating different rename directories corresponding to the same distributed job to form a job calculation result.
[0074] Specifically, object storage 120 can also aggregate different rename directories corresponding to the same distributed job to form a job calculation result corresponding to the distributed job, so that the scheduling system can pull the job calculation results corresponding to different distributed jobs in units of distributed jobs, thereby reducing the workload of the scheduling system in aggregating different rename directories and alleviating the data processing pressure of the scheduling system.
[0075] In one embodiment, before receiving and saving the rename directories corresponding to different distributed jobs uploaded by different computing nodes 110, the method further includes:
[0076] Upon receiving the data request sent by the computing node 110, querying the relevant data requested by the data request;
[0077] The queried relevant data is sent to the computing node 110, wherein the relevant data is used to participate in the execution of the corresponding splitting task.
[0078] Specifically, the object storage 120 also provides the computing node 110 with the relevant data required to perform the splitting task based on the data request sent by the computing node 110, that is, the object storage 120 is also used to provide the storage function of the relevant data to save the storage space of the computing node 110 and reduce the number of interactive objects of the computing node 110, thereby simplifying the computing process of the computing node 110 to perform the splitting task.
[0079] Figure 2 and Figure 3 FIG. 1 is a flow chart of a data storage method in one embodiment. It should be understood that although Figure 2 and Figure 3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0080] In one embodiment, Figure 4 As shown, a data storage device is provided, comprising:
[0081] An acquisition module 410 is used to acquire multiple split tasks after different distributed jobs are split;
[0082] A creation module 420, configured to create a temporary directory corresponding to each of the distributed jobs on a local disk;
[0083] An execution module 430 is used to execute each of the split tasks to obtain corresponding task execution results;
[0084] A writing module 440 is used to write each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs;
[0085] A renaming module 450 is used to rename the temporary directory corresponding to the target distributed job with the job name corresponding to the target distributed job when the execution of each of the split tasks belonging to the target distributed job is completed, so as to obtain a renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs;
[0086] The upload module 460 is used to upload the renamed directory corresponding to the target distributed job to the object storage 120 for storage.
[0087] In one embodiment, the acquisition module 410 is further used for:
[0088] Acquire relevant data related to each of the split tasks;
[0089] The execution module 430 is also used for:
[0090] The corresponding split tasks are executed based on the relevant data corresponding to each of the split tasks to obtain the task execution results corresponding to each of the split tasks.
[0091] In one embodiment, the acquisition module 410 is further used for:
[0092] Sending data requests corresponding to each of the split tasks to the object storage 120;
[0093] Obtain relevant data related to each of the splitting tasks provided by the object storage 120 according to each of the data requests.
[0094] In one embodiment, the creation module 420 is further configured to:
[0095] A temporary directory corresponding to each of the distributed jobs is created on the local disk, and a temporary subdirectory corresponding to each of the split tasks belonging to the same distributed job is created in the temporary directory.
[0096] In one embodiment, the writing module 440 is further used for:
[0097] The execution results of each of the tasks are written into a temporary directory corresponding to the distributed job to which the corresponding split task belongs, and into a temporary subdirectory created for the corresponding split task.
[0098] In one embodiment, Figure 5 As shown, a data storage device is provided, the device comprising:
[0099] The storage module 510 is used to receive and save the rename directories corresponding to different distributed jobs uploaded by different computing nodes 110;
[0100] The aggregation module 520 is used to aggregate different rename directories corresponding to the same distributed job to form a job calculation result.
[0101] In one embodiment, the device further comprises a query module, configured to:
[0102] Upon receiving the data request sent by the computing node 110, querying the relevant data requested by the data request;
[0103] The queried relevant data is sent to the computing node 110, wherein the relevant data is used to participate in the execution of the corresponding splitting task.
[0104] like Figure 6 As shown, an embodiment of the present application provides a computer device, including a processor 711, a communication interface 712, a memory 713 and a communication bus 714, wherein the processor 711, the communication interface 712, and the memory 713 communicate with each other through the communication bus 714;
[0105] Memory 713, used for storing computer programs;
[0106] The processor 711 is used to implement the data storage method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 713.
[0107] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0108] In one embodiment, the data storage device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 6 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the data storage device, such as: Figure 4 The obtained module 410, the created module 420, the executed module 430, the written module 440, the renamed module 450 and the uploaded module 460 are shown. The computer program composed of each program module enables the processor to execute the data storage method of each embodiment of the present application described in this specification.
[0109] Figure 6 The computer device shown can be Figure 4 The acquisition module 410 in the data storage device shown is executed to acquire multiple split tasks after different distributed jobs are split. The computer device can execute the creation module 420 to create a temporary directory corresponding to each of the distributed jobs on the local disk. The computer device can execute each of the split tasks through the execution module 430 to obtain the corresponding task execution results. The computer device can execute the writing module 440 to write each of the task execution results into the temporary directory corresponding to the distributed job to which the corresponding split task belongs. The computer device can execute the renaming module 450 to rename the temporary directory corresponding to the target distributed job with the job name corresponding to the target distributed job when each of the split tasks belonging to the target distributed job is executed, and obtain the rename directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs. The computer device can execute the uploading module 460 to upload the rename directory corresponding to the target distributed job to the object storage 120 for storage.
[0110] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data storage method provided by any one of the aforementioned method embodiments is implemented.
[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or partly contributed to the relevant technology in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the data storage method described in each embodiment or some parts of the embodiment.
[0113] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative can be used.
[0114] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A data storage method, characterized in that: The method comprises: Get multiple split tasks after different distributed jobs are split; Create a temporary directory corresponding to each of the distributed jobs on the local disk; Execute each of the split tasks to obtain corresponding task execution results; Writing each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs; When each of the split tasks belonging to the target distributed job is executed, renaming the temporary directory corresponding to the target distributed job with the job name corresponding to the target distributed job to obtain a renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs; The renamed directory corresponding to the target distributed job is uploaded to the object storage for storage.
2. The data storage method according to claim 1, characterized in that: After obtaining the multiple split tasks after splitting different distributed jobs, the method further includes: Acquire relevant data related to each of the split tasks; The executing of each of the split tasks to obtain corresponding task execution results includes: The corresponding split tasks are executed based on the relevant data corresponding to each of the split tasks to obtain the task execution results corresponding to each of the split tasks.
3. The data storage method according to claim 2, characterized in that: Obtain relevant data related to each of the split tasks, including: Sending data requests corresponding to each of the split tasks to the object storage; Relevant data related to each of the splitting tasks provided by the object storage according to each of the data requests is obtained.
4. The data storage method according to claim 1, characterized in that: The step of creating a temporary directory corresponding to each of the distributed jobs on the local disk includes: A temporary directory corresponding to each of the distributed jobs is created on the local disk, and a temporary subdirectory corresponding to each of the split tasks belonging to the same distributed job is created in the temporary directory.
5. The data storage method according to claim 4, characterized in that: The step of writing each of the task execution results into a temporary directory corresponding to the distributed job to which the corresponding split task belongs includes: The execution results of each of the tasks are written into a temporary directory corresponding to the distributed job to which the corresponding split task belongs, and into a temporary subdirectory created for the corresponding split task.
6. A data storage method, characterized in that: The method comprises: Receive and save renamed directories corresponding to different distributed jobs uploaded by different computing nodes; The different renamed directories corresponding to the same distributed job are aggregated to form the job calculation results.
7. The data storage method according to claim 6, characterized in that: Before receiving and saving the rename directories corresponding to different distributed jobs uploaded by different computing nodes, the method further includes: Upon receiving a data request sent by the computing node, querying the relevant data requested by the data request; The queried relevant data is sent to the computing node, wherein the relevant data is used to participate in the execution of the corresponding splitting task.
8. A data storage device, characterized in that: The device comprises: The acquisition module is used to obtain multiple split tasks after different distributed jobs are split; A creation module, used to create a temporary directory corresponding to each of the distributed jobs on a local disk; An execution module, used for executing each of the split tasks to obtain corresponding task execution results; A writing module, used to write the execution results of each task into a temporary directory corresponding to the distributed job to which the corresponding split task belongs; A renaming module, used for renaming a temporary directory corresponding to the target distributed job with a job name corresponding to the target distributed job when each of the split tasks belonging to the target distributed job is executed, so as to obtain a renamed directory corresponding to the target distributed job, wherein the target distributed job is any one of the distributed jobs; The upload module is used to upload the renamed directory corresponding to the target distributed job to the object storage for storage.
9. A data storage device, characterized in that: The device comprises: A storage module, used to receive and save renamed directories corresponding to different distributed jobs uploaded by different computing nodes; The aggregation module is used to aggregate different renamed directories corresponding to the same distributed job to form the job calculation result.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data storage method according to any one of claims 1 to 5 or 6 to 7 is implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 5 or 6 to 7 is implemented.