Data processing method and device, equipment and medium
By splitting and parallel processing of output files in big data jobs, the problem of low data processing efficiency in the existing technology is solved, and efficient massive data processing is achieved.
Patent Information
- Application Number
- CN202411784959.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-03-28
AI Technical Summary
The existing data processing technology is inefficient, especially when processing big data operations, serial processing methods cannot meet the efficiency requirements.
By judging the dependencies between output files, the pending jobs are split into multiple job steps and sent to multiple threads for processing in parallel.
It improves data processing efficiency, can meet the needs of massive data processing, and reduces resource waste.
Smart Images
Figure CN119718574A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium. Background Art
[0002] Currently, data processing is usually performed on a single machine, and all operations can only be processed serially, resulting in low efficiency. For example, for big data jobs, when serial processing is performed, multiple job steps are executed sequentially, and the next job step can only be executed after the previous job step is completed. Since disk reading and writing of a single file takes a long time, and big data jobs need to process massive amounts of data, the existing serial processing method cannot meet the efficiency requirements of big data jobs. Summary of the invention
[0003] The embodiments of the present application provide a data processing method, apparatus, device and medium for improving data processing efficiency.
[0004] In a first aspect, a data processing method is provided, the method comprising:
[0005] Obtaining original job parameters of the job to be processed; the original job parameters include multiple output file information, and one output file information corresponds to one output file;
[0006] If the job to be processed supports splitting, determine whether there is dependency between the output files;
[0007] If there is no dependency between any two output files, the job to be processed is split into multiple job steps according to the number of output files; wherein a job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to an output file information;
[0008] Send multiple job steps to multiple threads for execution.
[0009] The data processing method provided in the present application, when processing a divisible job to be processed, such as a big data job, splits the job to be processed into multiple job steps according to the dependency relationship between the output files. The multiple job steps can be sent to multiple threads for parallel processing, thereby greatly improving the data processing efficiency and meeting the needs of massive data processing.
[0010] In some embodiments, before sending the multiple job steps to multiple threads for execution, the method further includes:
[0011] Save the sub-job parameters to the database and send the identity of each job step to the message container;
[0012] Send multiple job steps to multiple threads for execution, including:
[0013] Get the identity in the message container;
[0014] According to the obtained identity, read the sub-job parameters of the corresponding job step from the database;
[0015] Send the read sub-job parameters to the corresponding thread for execution.
[0016] By building a message middleware, asynchronous communication between different modules is realized, thereby achieving parallel execution of multiple job steps.
[0017] In some embodiments, the data processing method further includes:
[0018] Synchronize the completion status of each job step after execution to the message container;
[0019] Get the job step completion status recorded in the message container, and clear the identity of the completed job step from the message container.
[0020] By using the above method, the identity of the completed job step is deleted from the message container, thereby avoiding the waste of resources caused by reprocessing the completed job step.
[0021] In some embodiments, before sending the multiple job steps to the multiple threads for execution, the method further includes:
[0022] Establish a pending job slicing table; the pending job slicing table records the number of job steps into which the pending job is split;
[0023] The method also includes:
[0024] Record the number of completed job steps;
[0025] When the number of completed job steps is the same as the number of job steps recorded in the pending job shard table, the pending job is marked as completed.
[0026] Through the above method, the completion status of the tasks to be executed can be monitored and managed.
[0027] In some embodiments, the data processing method further comprises:
[0028] If there are dependencies between at least some of the multiple output files, the pending jobs are sent to a single thread for execution.
[0029] By using the above method, when there are dependencies between at least some of the multiple output files, the job to be processed is not split, thereby reducing the difficulty of splitting.
[0030] In some embodiments, the data processing method further includes:
[0031] If, among multiple output files, some output files are dependent on each other, while the rest of the output files are not dependent on each other, the job to be processed is split into multiple job steps according to the dependency relationship between the output files; the output file information corresponding to the multiple dependent output files is all assigned to the sub-job parameters of the same job step, and the output file information corresponding to the non-dependent output files is assigned one by one to the sub-job parameters of different job steps.
[0032] The above method is helpful to save computing resources, avoid confusion in the processing process, improve stability, and take into account processing efficiency.
[0033] In some embodiments, the dependencies between the output files are preconfigured in the original job parameters;
[0034] If the job to be processed supports splitting, determine whether there is dependency between the output files, including:
[0035] Get dependencies from the original job parameters;
[0036] Determine whether there is dependency between output files based on dependency relationships.
[0037] By using the above method, the dependency relationship between the output files is pre-configured in the original job parameters, and the dependency relationship between the output files can be directly obtained from the original job parameters, thereby simplifying the dependency relationship judgment process between the output files and improving processing efficiency.
[0038] In a second aspect, the present application provides a data processing device, including:
[0039] The initiating module is used to obtain the original job parameters of the job to be processed; determine whether the job to be processed supports splitting; determine whether there is a dependency between the output files corresponding to the multiple output file information in the original job parameters; if there is no dependency between any two output files, split the job to be processed into multiple job steps according to the number of output files; wherein the job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to an output file information;
[0040] The scheduling module is used to send multiple job steps to multiple threads for execution.
[0041] In some embodiments, the initiating module is further used to send each sub-job parameter to a database and send an identity of each job step to a message container before sending the multiple job steps to the multiple threads for execution;
[0042] The scheduling module is specifically used to obtain the identity in the message container; read the sub-job parameters of the corresponding job step from the database according to the obtained identity; and send the read sub-job parameters to the corresponding thread for execution.
[0043] In some embodiments, the scheduling module is further used to synchronize the completion status of each job step after execution to the message container;
[0044] The initiating module is further used to obtain the completion status of the operation step recorded in the message container, and clear the identity of the completed operation step from the message container.
[0045] In some embodiments, the initiating module is also used to establish a pending job slice table; the pending job slice table records the number of job steps into which the pending job is split; records the number of completed job steps; when the number of completed job steps is the same as the number of job steps recorded in the pending job slice table, the pending job is marked as executed.
[0046] In some embodiments, the scheduling module is further used to send the pending job to a single thread for execution if there is dependency between at least some of the output files among the multiple output files.
[0047] In some embodiments, the initiating module is also used to, if among multiple output files, some output files have dependencies, while the remaining output files do not have dependencies, split the job to be processed into multiple job steps according to the dependency relationship between the output files; wherein the output file information corresponding to the multiple dependent output files is all allocated to the sub-job parameters of the same job step, and the output file information corresponding to the output files without dependencies is allocated one by one to the sub-job parameters of different job steps.
[0048] In a third aspect, the present application provides an electronic device, including:
[0049] A memory for storing program instructions;
[0050] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.
[0051] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods in the first aspect.
[0052] In a fifth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 One of the flow charts of the data processing method provided in the embodiment of the present application;
[0054] Figure 2 One of the schematic diagrams of the structure of splitting jobs to be processed provided in the embodiment of the present application;
[0055] Figure 3 The second flowchart of the data processing method provided in the embodiment of the present application;
[0056] Figure 4 Flowchart 3 of the data processing method provided in the embodiment of the present application;
[0057] Figure 5 Flow chart 4 of the data processing method provided in the embodiment of the present application;
[0058] Figure 6 The second schematic diagram of the structure of splitting jobs to be processed provided in the embodiment of the present application;
[0059] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0060] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is exemplarily shown. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0062] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0063] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0064] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.
[0065] Currently, data processing is usually performed on a single machine, and all operations can only be processed serially, resulting in low efficiency. For example, for big data jobs, when serial processing is performed, multiple job steps are executed sequentially, and the next job step can only be executed after the previous job step is completed. Since disk reading and writing of a single file takes a long time, and big data jobs need to process massive amounts of data, the existing serial processing method cannot meet the efficiency requirements of big data jobs.
[0066] In view of this, embodiments of the present application provide a page method, apparatus, device and medium for solving the problem of fixed data processing.
[0067] Figure 1 This is one of the flow charts of the data processing method provided in the embodiment of the present application. The method can be applied to the case of batch data processing, such as big data job processing in banking business, etc. The method can be executed by a data processing device, which can be composed of hardware and / or software. Figure 1 As shown, the flow method includes the following steps:
[0068] S110: Acquire original job parameters of the job to be processed; the original job parameters include multiple output file information, and one output file information corresponds to one output file.
[0069] In this step, the original job parameters of the job to be processed are obtained. Figure 2 One of the schematic diagrams of the splitting structure of the jobs to be processed provided in the embodiment of the present application is as follows: Figure 2As shown, the original job parameters contain pre-configured input file information, data processing strategies, output file information and other information required to complete the job to be processed. One of the output file information is used to define an output file. When the output results of the job to be processed are multiple, the number of output file information in the original job parameters can be multiple, thereby defining multiple output files. For example, when a big data job needs to output multiple results, multiple output file information can be configured in the original job parameters to define multiple output files that need to be output. During specific implementation, the number of output file information in the original job parameters can be one or more. In the embodiment of the present application, the original job parameters contain multiple output file information as an example to illustrate the method for parallel processing of splitting the job to be processed provided by the present application. When the number of output file information in the original job parameters is one, data processing can be performed with reference to the prior art, which will not be repeated here.
[0070] S120: If the job to be processed supports splitting, determine whether there is dependency between the output files;
[0071] In this step, it is necessary to determine whether the pending job is of a splittable type. Only when the pending job is of a splittable type can it be split to implement the subsequent parallel processing process. For example, when the type of the pending job is a big data job, it can be considered that the pending job is of a splittable type.
[0072] In this step, if the job to be processed supports splitting, it is necessary to determine whether there is a dependency between the output files. In the present application, there is a dependency between output files, which specifically refers to a specific association relationship between two or more output files. For example, two or more files need to be output in a specific order, then it can be considered that the two or more output files are dependent on each other. For example, the calculation of the next output file needs to be processed based on the previous output file, then it can be considered that the two output files are dependent on each other. The specific association relationship may also include other situations not illustrated. During the specific implementation, it can be defined according to actual needs and is not limited here. In some embodiments of the present application, output files that do not have dependencies can be processed in parallel in multiple threads to improve processing efficiency, and output files that have dependencies can be processed through one thread to reduce resource usage, save computing resources, and help improve the stability of the processing process. The specific content will be elaborated in detail later and will not be repeated here.
[0073] In some embodiments, the dependency relationship between output files can be pre-configured in the original job parameters, where the dependency relationship includes two situations: the existence of dependency and the non-existence of dependency. When determining whether there is dependency between output files, the dependency relationship between output files can be directly obtained from the original job parameters, and then the dependency relationship between output files can be determined based on the obtained dependency relationship, thereby simplifying the dependency relationship determination process between output files and improving processing efficiency. For example, the dependency relationship between output files can be configured in the following manner: if one output file among multiple output files does not have the aforementioned specific association relationship with other output files, it can be considered that there is no dependency between the output file and other output files among the multiple output files; if there is the aforementioned specific association relationship between one output file among multiple output files and another output file, and there is no aforementioned specific association relationship between these two output files and other output files among the multiple output files, it can be considered that there is a dependency between the two output files, and there is no dependency between these two files and other output files; if there is the aforementioned specific association relationship between the first output file and the second output file among multiple output files, and there is the aforementioned specific association relationship between the second output file and the third output file, then regardless of whether there is the aforementioned specific association relationship between the first output file and the third output file, it can be considered that there is a dependency between the first output file, the second output file and the third output file; and so on. The association relationship between multiple output files can be defined. In specific implementation, the dependency relationship between output files can also be configured in other ways, which are not limited here.
[0074] S130: If there is no dependency between any two output files, the job to be processed is split into multiple job steps according to the number of output files; wherein a job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to an output file information.
[0075] In this step, if there is no dependency between the output files, it means that each output file is not restricted by the output order and content of other output files, and can be output independently. Then, the job to be processed can be split into multiple job steps according to the number of output files, and each job step can be used to process one output file independently, so that multiple job steps can be sent to multiple threads for parallel processing to maximize the processing efficiency. When splitting, the original job parameters can be split into multiple sub-job parameters, one sub-job parameter corresponds to one job step, and the sub-job parameters contain the necessary information to complete the corresponding job step. Specifically, each sub-job parameter contains an output file information to define the output file that the job step corresponding to the sub-job parameter needs to output, and each sub-job parameter also contains the same input file information, data processing strategy and other necessary information as the original job parameters.
[0076] For example, if Figure 2 As shown, the original job parameters include output file information 1, output file information 2 and output file information 3, and there is no dependency between the output files defined by output file information 1, output file information 2 and output file information 3. Then the original job parameters can be split into sub-job parameters 1, sub-job parameters 2 and sub-job parameters 3, wherein sub-job parameter 1 includes output file information 1, input file information, data processing strategy and other information, sub-job parameter 2 includes output file information 2, input file information, data processing strategy and other information, and sub-job parameter 3 includes output file information 3, input file information, data processing strategy and other information, thereby splitting the job 10 to be processed into a first job step 11, a second job step 12 and a third job step 13.
[0077] S140: Send multiple job steps to multiple threads for execution.
[0078] In this step, multiple job steps are sent to multiple threads for execution, wherein each thread corresponds to an execution unit for executing the corresponding job step, and multiple execution units execute in parallel to improve data processing efficiency.
[0079] The data processing method provided in the present application, when processing a divisible job to be processed, such as a big data job, splits the job to be processed into multiple job steps according to the dependency relationship between the output files. The multiple job steps can be sent to multiple threads for parallel processing, thereby greatly improving the data processing efficiency and meeting the needs of massive data processing.
[0080] Figure 3 The second flowchart of the data processing method provided in the embodiment of the present application.
[0081] In some embodiments, Figure 3 As shown, before executing step S140, step S150 is also included: saving the sub-job parameters to the database, and sending the identity of each job step to the message container.
[0082] In this step, the sub-job parameters corresponding to each job step are saved in the database, and when split, each job step generates a corresponding identity. In this step, the identity of each job step is also sent to the message container for subsequent calls.
[0083] When executing step S140, the following steps are specifically included:
[0084] S141: Obtaining the identity in the message container;
[0085] S142: Reading the sub-job parameters of the corresponding job step from the database according to the acquired identity identifier;
[0086] S143: Send the read operation step parameters to the corresponding thread for execution.
[0087] In the data processing method provided by the embodiment of the present application, the splitting process, the scheduling process and the execution process need to be carried out in different modules, for example, the splitting process, the scheduling process and the execution process are respectively carried out in the initiating module, the scheduling module and the execution module, and the message middleware composed of the database and the message container can realize asynchronous communication between different modules, thereby realizing the parallel execution of multiple job steps. Specifically, the initiating module splits the pending job into multiple job steps, and generates a corresponding identity for each job step, and then sends the sub-job parameters corresponding to each job step to the database for storage, and sends the identity of each job step to the message container, the scheduling module obtains the identity from the message module, and then reads the sub-job parameters of the job step corresponding to the identity from the database according to the obtained identity, and sends the read sub-job parameters to a thread of the execution module for execution. The scheduling module continuously obtains the identity from the message module, and reads the sub-job parameters from the database according to the identity, and sends them to the corresponding thread in the execution module for execution until the pending job is executed.
[0088] Figure 4 The third flowchart of the data processing method provided in the embodiment of the present application.
[0089] In some embodiments, Figure 4 As shown, the data processing method also includes the following steps:
[0090] S410: Synchronize the completion status of each job step after execution into the message container.
[0091] S420: Acquire the completion status of the operation step recorded in the message container, and clear the identity of the completed operation step from the message container.
[0092] Each job step usually does not need to be executed again after completion to avoid wasting computing resources. Therefore, after each job step is completed, the completion status of the job step can be synchronized to the message container through the scheduling module, and the initiating module can continuously scan and obtain the completion status of the job step recorded in the message container, and clear the identity of the completed job step from the message container to avoid re-executing the completed job step and causing waste of resources. Among them, whether the job step is completed can be defined according to actual needs. For example, in some embodiments, only the successfully executed job step can be defined as completed, and for the job step that failed to execute or timed out, it can be executed multiple times until it is successfully executed. In some embodiments, both the successfully executed and failed job steps can be defined as completed, and for the job step that timed out, it can continue to be executed until it succeeds or fails, or the job step that timed out can be directly defined as completed. No limitation is made here.
[0093] In some embodiments, before sending multiple job steps to multiple threads for execution, a pending job slice table may be established, in which the pending job slice table records the number of job steps into which the pending job is split, so as to manage the completion of the pending job. For example, the initiating module may establish a pending job slice table while splitting the pending job, and record the number of job steps into which the pending job is split in the pending job slice table when the splitting is completed. Figure 5 The fourth flowchart of the data processing method provided in the embodiment of the present application is as follows: Figure 5 As shown, the data processing method also includes the following steps:
[0094] S510: Record the number of completed operation steps;
[0095] S520: When the number of completed job steps is the same as the number of job steps recorded in the pending job slice table, the pending job is marked as completed.
[0096] For example, the number of completed job steps can be recorded through the initiating module. When the number of completed job steps is the same as the number of job steps recorded in the pending job shard table, the pending job is marked as completed, so as to monitor and manage the completion status of the pending job.
[0097] In some embodiments, if there is a dependency between at least some of the multiple output files, the job to be processed is sent to a single thread for execution. Specifically, when there is a dependency between at least some of the multiple output files, the job to be processed does not need to be split, so that the entire job to be processed can be sent to a single thread for execution. This processing method is simple to operate, can reduce the difficulty of splitting, and avoid errors in the splitting process. For example, the multiple output files include a first output file, a second output file, and a third output file, wherein there is a dependency between the first output file and the second output file, and there is no dependency between the third output file and the first output file and the second output file. At this time, the job to be processed does not need to be split. In specific implementation, there may be other situations where there is a dependency between at least some of the multiple output files, which are not described one by one here.
[0098] In some embodiments, if there are dependencies between some of the multiple output files, and there are no dependencies between the other output files, then the job to be processed is split into multiple job steps according to the dependency relationship between the output files; wherein the output file information corresponding to the multiple output files with dependencies is all assigned to the sub-job parameters of the same job step, and the output file information corresponding to the output files without dependencies is assigned one by one to the sub-job parameters of different job steps. When there are dependencies between some of the multiple output files, by assigning the output files with dependencies to the same job step for processing, it is beneficial to save computing resources, and since there is a certain correlation relationship between the multiple output files with dependencies in logical order, assigning the output files with dependencies to the same job step for processing can avoid confusion in the processing process and improve stability, and at the same time, the output files without dependencies are assigned to different job steps for processing, so that the processing efficiency can be taken into account at the same time.
[0099] Figure 6 The second schematic diagram of the structure of splitting jobs to be processed provided in an embodiment of the present application.
[0100] For example, if Figure 6As shown, the original job parameters corresponding to the pending job 10 include output file information 1, output file information 2 and output file information 3, wherein there is a dependency between the output file defined by output file information 1 and the output file defined by output file information 2, and there is no dependency between the output file defined by output file information 3 and the output file defined by output file 1 and the output file defined by output file 2. Then, according to the dependency relationship between the output files defined by output file information 1, output file information 2 and output file information 3, the pending job can be split into a first job step 11 and a second job step 12, wherein the first output file information 1 and the second output file information 2 are all allocated to the sub-job parameter 1 corresponding to the first job step 11, and the third output file information 3 is allocated to the sub-job parameter 2 corresponding to the second job step 12. In specific implementation, there may be other cases where some of the output files among the multiple output files are dependent on each other, which will not be described one by one here.
[0101] Based on the same technical concept, an embodiment of the present application also provides a data processing device, which can implement the process of the above-mentioned data method in the embodiment of the present application.
[0102] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 7 As shown, the device includes: an initiating module 710 and a scheduling module 720. Further, it may also include an executing module 730, a message container 740 and a database 750.
[0103] Initiate module 710, used to obtain the original job parameters of the job to be processed; determine whether the job to be processed supports splitting; determine whether there is a dependency between the output files corresponding to the multiple output file information in the original job parameters; if there is no dependency between any two output files, split the job to be processed into multiple job steps according to the number of output files; wherein the job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to an output file information.
[0104] The scheduling module 720 is used to send multiple job steps to multiple threads for execution.
[0105] The execution module 730 includes a plurality of execution units, one execution unit corresponds to one thread, and the execution module 730 is used to receive the sub-job parameters sent by the scheduling module 720 and call the corresponding execution unit to execute the job step.
[0106] In some embodiments, the dependency relationship between the output files is preconfigured in the original job parameters. The initiating module 710 is specifically used to obtain the dependency relationship from the original job parameters; and determine whether there is a dependency between the output files according to the dependency relationship.
[0107] In some embodiments, the initiating module 710 is further used to send each sub-job parameter to the database 750 and send the identity of each job step to the message container 740 before the scheduling module 720 sends the multiple job steps to the multiple threads for execution. The scheduling module is specifically used to obtain the identity in the message container 740; read the sub-job parameters of the corresponding job step from the database 750 according to the obtained identity; and send the read sub-job parameters to the corresponding thread for execution.
[0108] In some embodiments, the scheduling module 720 is further used to synchronize the completion status of each job step after execution to the message container 740. The initiating module 710 is further used to obtain the completion status of the job step recorded in the message container 740, and clear the identity of the completed job step from the message container 740.
[0109] In some embodiments, the initiation module 710 is also used to establish a pending job slice table; the pending job slice table records the number of job steps into which the pending job is split; records the number of completed job steps; when the number of completed job steps is the same as the number of job steps recorded in the pending job slice table, the pending job is marked as executed.
[0110] In some embodiments, the scheduling module 720 is further configured to send the pending job to a single thread for execution if there is dependency between at least some of the output files among the multiple output files.
[0111] In some embodiments, the initiating module is also used to, if among multiple output files, some output files have dependencies, while the remaining output files do not have dependencies, split the job to be processed into multiple job steps according to the dependency relationship between the output files; wherein the output file information corresponding to the multiple dependent output files is all allocated to the sub-job parameters of the same job step, and the output file information corresponding to the output files without dependencies is allocated one by one to the sub-job parameters of different job steps.
[0112] Based on the same technical concept, an embodiment of the present application provides an electronic device that can implement the functions of the data processing device discussed above.
[0113] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is exemplarily shown.
[0114] The electronic device provided in the embodiment of the present application includes at least one processor 801 and a memory 802 connected to the at least one processor 801. The specific connection medium between the processor 801 and the memory 802 is not limited in the embodiment of the present application. Figure 8In the example, the processor 801 and the memory 802 are connected via a bus 800. Figure 8 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 800 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 801 can also be called a controller, and there is no limitation on the name.
[0115] In the embodiment of the present application, the memory 802 stores instructions that can be executed by at least one processor 801. The at least one processor 801 can execute a data processing method discussed above by executing the instructions stored in the memory 802. The processor 801 can implement Figure 7 The functions of each module in the device shown.
[0116] Among them, the processor 801 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 802 and calling the data stored in the memory 802, the various functions of the device and processing data, the device can be monitored as a whole.
[0117] In the embodiment of the present application, the processor 801 may include one or more processing units, and the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0118] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of a data processing method disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.
[0119] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 802 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 802 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0120] By programming the processor 801, the code corresponding to the data processing method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The embodiment shown is a data processing method. How to design and program the processor 801 is a technology well known to those skilled in the art and will not be described in detail here.
[0121] It should be noted here that the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0122] Based on the same technical concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data processing method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.
[0123] Based on the same technical concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data processing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.
[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data processing method, characterized in that: The method comprises: Obtaining original job parameters of the job to be processed; the original job parameters include a plurality of output file information, and each output file information corresponds to one output file; If the job to be processed supports splitting, determining whether there is dependency between the output files; If there is no dependency between any two of the output files, the to-be-processed job is split into a plurality of job steps according to the number of the output files; wherein the job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to one of the output file information; The multiple job steps are sent to multiple threads for execution.
2. The method according to claim 1, characterized in that Before sending the multiple job steps to the multiple threads for execution, the method further includes: The sub-job parameters are saved in a database, and the identity of each of the job steps is sent to a message container; The sending the multiple job steps to multiple threads for execution specifically includes: Obtaining the identity in the message container; Reading the sub-job parameters of the corresponding job step from the database according to the acquired identity identifier; Send the read sub-job parameters to the corresponding thread for execution.
3. The method according to claim 2, characterized in that Also includes: Synchronize the completion status of each job step after execution to the message container; The completion status of the operation step recorded in the message container is obtained, and the identity of the completed operation step is cleared from the message container.
4. The method according to claim 1, characterized in that Before sending the multiple job steps to multiple threads for execution, the method further includes: Establishing a pending job slice table; the pending job slice table records the number of job steps into which the pending job is split; The method further comprises: Record the number of completed job steps; When the number of completed job steps is the same as the number of job steps recorded in the pending job slice table, the pending job is marked as completed.
5. The method according to claim 1, characterized in that Also includes: If there are dependencies between at least some of the multiple output files, the to-be-processed jobs are sent to a single thread for execution.
6. The method according to claim 1, characterized in that Also includes: If, among the multiple output files, some of the output files are dependent on each other, while the remaining output files are not dependent on each other, the job to be processed is split into multiple job steps according to the dependency relationship between the output files; wherein the output file information corresponding to the multiple dependent output files is all allocated to the sub-job parameters of the same job step, and the output file information corresponding to the output files without dependencies is allocated one by one to the sub-job parameters of different job steps.
7. The method according to any one of claims 1 to 6, characterized in that: The dependency relationship between the output files is preconfigured in the original job parameters; If the job to be processed supports splitting, determining whether there is a dependency between the output files specifically includes: Acquire the dependency relationship from the original job parameters; It is determined whether there is a dependency between the output files according to the dependency relationship.
8. A data processing device, characterized in that: include: An initiating module, used to obtain the original job parameters of the job to be processed; Determine whether the pending job supports splitting; Determine whether there is a dependency between output files corresponding to the multiple output file information in the original job parameters; If there is no dependency between any two of the output files, the job to be processed is split into multiple job steps according to the number of output files; wherein the job step corresponds to a sub-job parameter, and the sub-job parameter corresponds to one of the output file information; The scheduling module is used to send the multiple job steps to multiple threads for execution.
9. The data processing device according to claim 8, characterized in that The initiating module is further used to send each of the sub-job parameters to a database and send an identity of each of the job steps to a message container before sending the multiple job steps to the multiple threads for execution; The scheduling module is specifically used to obtain the identity in the message container; read the sub-job parameters of the corresponding job step from the database according to the obtained identity; and send the read sub-job parameters to the corresponding thread for execution.
10. The data processing device according to claim 9, characterized in that The scheduling module is also used to synchronize the completion status of each job step after execution into the message container; The initiating module is further used to obtain the completion status of the operation step recorded in the message container, and clear the identity of the completed operation step from the message container.
11. The data processing device according to claim 8, characterized in that The initiation module is also used to establish a pending job slice table; the pending job slice table records the number of job steps into which the pending job is split; records the number of completed job steps; when the number of completed job steps is the same as the number of job steps recorded in the pending job slice table, the pending job is marked as executed.
12. The data processing device according to any one of claims 8 to 11, characterized in that: The scheduling module is also used to send the pending job to a single thread for execution if at least some of the multiple output files are dependent on each other.
13. The data processing device according to any one of claims 8 to 11, characterized in that: The initiating module is also used to, if among the multiple output files, some of the output files are dependent on each other, while the remaining output files are not dependent on each other, split the to-be-processed job into multiple job steps according to the dependency relationship between the output files; wherein the output file information corresponding to the multiple dependent output files is all allocated to the sub-job parameters of the same job step, and the output file information corresponding to the output files without dependencies is allocated one by one to the sub-job parameters of different job steps.
14. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 7 according to the obtained program instructions.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 7.
16. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.