Data processing method and device, equipment and storage medium

Abstract

The invention relates to a data processing method and device, equipment and a storage medium. The method comprises the steps that a to-be-processed target task of a terminal is acquired; determining a process set for executing the target task and an execution sequence corresponding to the process set; according to the execution sequence, executing each process in the process set in sequence to obtain an execution result corresponding to each process; and storing the execution result in a local storage device, and uploading the execution result to the target object based on a preset storage strategy, so that the target object stores the execution result in an object form. According to the method, even if the data is abnormal when the data is transmitted to the target object, retransmission can still be performed based on the execution result stored in the local storage device after the problem is solved, and the problem of resource waste caused by failure of the whole task is avoided.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a data processing method, apparatus, device, and storage medium. Background Art

[0002] In data storage, object storage is a technology for storing and managing a large amount of unstructured data, which has advantages such as high scalability, high reliability, and high flexibility, and has been extremely widely used in the field of data storage.

[0003] In the prior art, after obtaining a Spark task, execute the Spark task to obtain its corresponding execution result. Then, send the execution result to a target object so that the target object stores the execution result.

[0004] However, in the above method, if the target object returns an error message, it may cause the entire task to fail, and the process of re-executing the task will consume a large amount of resources, especially for large-scale data processing tasks, thus resulting in resource waste. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium. Even if an exception occurs when data is transmitted to a target object, after the problem is solved, retransmission can be performed based on the execution result stored in the local storage device, avoiding the problem of resource waste caused by the failure of the entire task.

[0006] In a first aspect, this application provides a data processing method, and the method includes:

[0007] Obtain a target task to be processed by a terminal;

[0008] Determine a process set for executing the target task and an execution order corresponding to the process set;

[0009] Execute each process in the process set in sequence according to the execution order to obtain an execution result corresponding to each process;

[0010] Store the execution result in a local storage device, and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in an object form.

[0011] Optionally, the uploading the execution result to the target object based on a preset storage policy includes:

[0012] During the process of executing each process in sequence according to the execution order, detect whether there is a target process that has been executed;

[0013] When there is a target process that has been executed, obtain the target execution result corresponding to the target process, and upload the target execution result based on the target object.

[0014] Optionally, the uploading the target execution result based on the target object includes:

[0015] Detect whether the target execution result meets a preset condition;

[0016] When the target execution result meets the preset condition, send the target execution result to the target object;

[0017] When the target execution result does not meet the preset condition, analyze the target execution result to obtain the number of data sending times; according to the number of data sending times, send the target execution result to the target object.

[0018] Optionally, the uploading the execution result to the target object based on a preset storage policy includes:

[0019] Detect whether the target task has been executed;

[0020] When the target task has been executed, obtain all execution results of the target task, and send all execution results to the target object.

[0021] Optionally, the method further includes:

[0022] Receive data returned by the target object;

[0023] When the data is an abnormal status code, obtain the required execution result in the local storage device, and send the execution result to the target object;

[0024] When the data is not an abnormal status code, mark the status of the corresponding process as successful.

[0025] Optionally, the method further includes:

[0026] According to a data division strategy, divide the execution result to obtain result elements corresponding to the execution result;

[0027] Create a corresponding temporary file for the result element, and use the temporary file to store the corresponding result element to obtain a temporary file corresponding to the execution result;

[0028] The uploading the execution result to the target object based on a preset storage policy includes:

[0029] Upload the temporary file to the target object based on a preset storage policy.

[0030] Optionally, the method further includes:

[0031] Create a request connection between the terminal and the target object according to the temporary file in the execution result, obtain the request connection corresponding to the target execution result, and the request connection and the temporary file are in one-to-one correspondence.

[0032] In a second aspect, the present application provides a data processing device, and the device includes:

[0033] An acquisition unit, configured to acquire a target task to be processed by a terminal;

[0034] A determination unit, configured to determine a process set for executing the target task and an execution order corresponding to the process set;

[0035] An execution unit, configured to sequentially execute the processes according to the execution order to obtain an execution result corresponding to each process;

[0036] A storage unit, configured to store the execution result in a local storage device and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in an object form.

[0037] Optionally, the device further includes a storage unit, configured to:

[0038] During the process of executing each process according to the execution order, detect whether there is a target process that has been executed;

[0039] When there is a target process that has been executed, obtain the target execution result corresponding to the target process and upload the target execution result based on the target object.

[0040] Optionally, the storage unit is configured to:

[0041] Detect whether the target execution result meets a preset condition;

[0042] When the target execution result meets the preset condition, send the target execution result to the target object;

[0043] When the target execution result does not meet the preset condition, analyze the target execution result to obtain the number of data sending times; according to the number of data sending times, send the target execution result to the target object.

[0044] Optionally, the storage unit is configured to:

[0045] Detect whether the target task has been executed;

[0046] When the target task is completed, obtain all the execution results of the target task and send all the execution results to the target object.

[0047] Optionally, the device further includes a data return unit, and the data return unit is configured to:

[0048] Receive data returned by the target object;

[0049] When the data is an abnormal status code, obtain the required execution results in the local storage device and send the execution results to the target object;

[0050] When the data is not an abnormal status code, mark the status of the corresponding process as successful.

[0051] Optionally, the device further includes a partitioning unit, and the partitioning unit is configured to:

[0052] Partition the execution results according to a data partitioning strategy to obtain result elements corresponding to the execution results;

[0053] Create corresponding temporary files for the result elements and use the temporary files to store the corresponding result elements to obtain temporary files corresponding to the execution results;

[0054] The storage unit is configured to:

[0055] Upload the temporary files to the target object based on a preset storage strategy.

[0056] Optionally, the device further includes a creation unit, and the creation unit is configured to:

[0057] Create a request connection between the terminal and the target object according to the temporary files in the execution results to obtain a request connection corresponding to the target execution result, and the request connection corresponds to the temporary file one by one.

[0058] In a third aspect, the present application provides a data processing device, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0059] Obtain a target task to be processed by the terminal;

[0060] Determine a process set for executing the target task and an execution order corresponding to the process set;

[0061] Execute each process in the process set in sequence according to the execution order to obtain the execution result corresponding to each process;

[0062] Store the execution result in a local storage device, and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in the form of an object.

[0063] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above data processing method is implemented.

[0064] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: In the embodiments of the present application, a target task to be processed by a terminal is obtained; at least one process included in the target task and the process execution order between the processes are determined; the processes are executed in sequence according to the execution order to obtain the execution result corresponding to each process; the execution result is stored in a local storage device, and based on a preset storage policy, the execution result is stored in the target object. It can be seen that when the execution result is stored in the target object in the present application, the execution result is also stored in a local storage device. Even if an exception occurs when the data is transmitted to the target object, after the problem is solved, the execution result stored in the local storage device can still be retransmitted to avoid the problem of resource waste caused by the failure of the entire task. Description of the Drawings

[0065] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present invention and used together with the description to explain the principles of the present invention.

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0067] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not limit the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, the drawings in the drawings do not constitute a proportional limitation.

[0068] Figure 1 It is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0069] Figure 2 It is a schematic flowchart of a data storage method provided by an embodiment of the present application;

[0070] Figure 3 A flowchart of a data upload method provided by an embodiment of the present application;

[0071] Figure 4 A flowchart of a data sending method provided by an embodiment of the present application;

[0072] Figure 5 A flowchart of another data upload method provided by an embodiment of the present application;

[0073] Figure 6 A flowchart of a data detection method provided by an embodiment of the present application;

[0074] Figure 7 A schematic diagram of another data processing provided by an embodiment of the present application;

[0075] Figure 8 A flowchart of a data processing device provided by an embodiment of the present application;

[0076] Figure 9 A schematic diagram of a data processing device provided by an embodiment of the present application. Detailed implementation manners

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0078] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. Such repetition is for the purpose of simplification and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0079] In data storage, object storage is a technology for storing and managing a large amount of unstructured data. It has advantages such as high scalability, high reliability, and high flexibility, and has been extremely widely used in the field of data storage. In the prior art, after obtaining a Spark task, the Spark task is executed to obtain its corresponding execution result. Then, the execution result is sent to a target object so that the target object stores the execution result. However, in the above manner, if the target object returns an error message, it may cause the entire task to fail, and the process of re-executing the task will consume a large amount of resources, especially for large-scale data processing tasks, thus resulting in waste of resources.

[0080] In summary, the embodiments of the present application provide a data processing method. Even if an exception occurs when transmitting to the target object, after the problem is solved, retransmission can be performed based on the execution result stored in the local storage device, avoiding the problem of waste of resources caused by the failure of the entire task, as Figure 1 shown. The specific steps include:

[0081] Step 101, obtain a target task to be processed by a terminal.

[0082] Among them, the target task is a Spark task to be processed. Spark is a unified analysis engine for large-scale data processing. A Spark task is a program written by a user using the API provided by Spark, aiming to perform operations such as reading, transforming, calculating, and storing large-scale data. A Spark task usually consists of one or more operations, and these operations can be related operations such as data reading and transformation.

[0083] In this step, when the terminal needs to execute the target task, the terminal needs to obtain the target task.

[0084] Step 102, determine a process set for executing the target task and the execution order corresponding to the process set.

[0085] In this step, analyze the dependency relationships between the various operations in the target task, and then find all the operations that generate dependency relationships. Then, according to the dependency relationships between the operations in the target task, the target task is divided into multiple processes to obtain a process set. Since there is a data dependency between processes, they need to be executed in sequence. That is, after the previous process is completed, the subsequent process can start to execute. Therefore, according to the dependency relationships between these processes, the execution order between these processes can be obtained.

[0086] Step 103, sequentially execute each process in the process set according to the execution order to obtain the execution result corresponding to each process.

[0087] In this step, each process in the process set is sequentially executed according to the execution order to obtain the execution result corresponding to each process.

[0088] Step 104: Store the execution result in the local storage device and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in the form of an object.

[0089] Among them, the local storage device refers to devices such as the terminal hard disk, or other devices that can be set separately, such as other storage devices connected to the terminal, NAS (Network Attached Storage), etc. The data in the target object is stored in the form of an object, that is, the target object is a device with data storage capabilities and supports the object storage method, such as an object storage system, a specific storage engine of a database, etc. The preset storage policy is set by technicians according to experience. It can be to store these execution results in the target object after obtaining the execution results corresponding to all processes, or to store the execution result corresponding to each process in the target object as soon as the execution result corresponding to a process is obtained. There is no limitation here.

[0090] In this step, the execution result can be stored in the local storage device and the target object simultaneously, or the execution result can be first written to the local storage device and then stored in the target object. There is no limitation here.

[0091] In addition, after writing the execution result to the local storage device, multiple threads can also be started to upload the data to the target object asynchronously. Of course, the data can also be uploaded to the target object synchronously. There is no limitation here.

[0092] It should be noted that technicians can determine whether to execute this application according to their own needs. For example, when the user closes the function corresponding to this application, it means that the user does not need to execute this application, and directly uploads the execution result of the target task to the target object after obtaining it. When the user enables the function corresponding to this application, it means that the user needs to execute this application. After obtaining the execution result of the target task, not only the execution result needs to be uploaded to the target object, but also the execution result needs to be stored in the local storage device.

[0093] In an embodiment of the present application, a target task to be processed by a terminal is obtained; a set of processes for executing the target task and an execution order corresponding to the set of processes are determined; each process in the set of processes is sequentially executed according to the execution order to obtain an execution result corresponding to each process; the execution result is stored in a local storage device, and based on a preset storage policy, the execution result is uploaded to the target object, so that the target object stores the execution result in an object form. It can be seen that when storing the execution result in the target object, the present application also stores the execution result in the local storage device. Even if an exception occurs when the data is transmitted to the target object, after the problem is solved, the execution result stored in the local storage device can still be retransmitted to avoid the problem of resource waste caused by the failure of the entire task.

[0094] In practice, after obtaining the execution results of the processes, if the amount of data of these execution results is very large, for the convenience of storage, these execution results can also be divided into multiple data blocks, and each data block is stored in a temporary file. Based on the above method, an execution result may include multiple temporary files. Therefore, an embodiment of the present application provides a data storage method, and the method is as Figure 2 shown, and the specific steps include:

[0095] Step 201, divide the execution result according to a data division policy to obtain result elements corresponding to the execution result.

[0096] Among them, the data division policy is used to divide the execution result to obtain its corresponding result elements. For example, the execution result can be divided according to a preset size to obtain at least one data block, and each data block is a result element. The preset size is set by a technician according to experience or can also be determined according to other rules, which is not limited here.

[0097] Step 202, create a corresponding temporary file for the result element and store the corresponding result element using the temporary file to obtain temporary files corresponding to the execution result.

[0098] In this step, after obtaining all the result elements corresponding to a certain execution result, a corresponding temporary file is created for each result element, and the corresponding result element is stored using the temporary file to obtain temporary files corresponding to the execution result.

[0099] Among them, the temporary file ends with a.temp suffix, and its file name is set based on a preset rule. For example, the corresponding process name and the creation time are concatenated to obtain the name of the temporary file. After all the processes in the target task are executed, spark driver can also be used for unified naming to achieve unified naming.

[0100] Among them, the spark driver is a key component in Apache Spark. It is responsible for coordinating and managing the execution of the entire Spark application and is the control center of the entire Spark application.

[0101] Step 203: Upload the temporary file to the target object based on the preset storage policy.

[0102] Among them, the preset storage policy is the same as the preset storage policy involved in step 104, and will not be elaborated here one by one.

[0103] In this step, when obtaining the temporary file of a certain process or obtaining all the temporary files of the target task, use the preset storage policy to upload the temporary file to the target object so that the target object stores these temporary files.

[0104] In addition, when uploading the execution result to the target object, a request connection between the terminal and the target object can be established based on each execution result, or a request connection between the terminal and the target object can be created according to the temporary file in the execution result. For example, multiple temporary files correspond to one request connection, or the temporary file corresponds to the request connection one by one. When the temporary file corresponds to the request connection one by one, create the request connection between the terminal and the target object according to the temporary file in the execution result to obtain the request connection corresponding to the target execution result.

[0105] In the embodiment of the present application, after obtaining the execution results of all processes, these execution results can be sent to the target object in batches so that the target object stores these execution results. To further reduce the request pressure on the target object, the present application can also store the execution result of each process in the local storage device and upload it to the target object every time a process is executed, so that the target object stores the execution result. Therefore, the embodiment of the present application provides a data upload method, which is a further limitation of the preset storage policy in step 104, as Figure 3 shown. The specific steps are as follows:

[0106] Step 301: During the process of executing each process in the execution order, detect whether there is a target process that has been executed.

[0107] In this step, during the process of executing each process in the execution order, detect whether there is a process that has been executed, and determine this process as the target process.

[0108] Step 302: When there is a target process that has been executed, obtain the target execution result corresponding to the target process and upload the target execution result based on the target object.

[0109] In this step, when there is a target process, obtain the target execution result corresponding to the target process, and store the target execution result based on the target object. That is, every time a process is executed and the execution result of the process is obtained, and the execution result is stored in the target object, which can avoid excessive request pressure on the target object caused by uploading a large amount of data to the target object.

[0110] In practice, the data volume of some execution results is very large, while that of some execution results is very small. For an execution result with a large data volume, if it is uploaded to the target object at one time, it may cause excessive request pressure on the target object. To avoid the above situation, this application can detect the data volume of the previous result and select an appropriate upload method according to the detection result, so as to avoid excessive request pressure on the target object as much as possible while saving the calculation amount. The embodiment of this application provides a data sending method, which is a further limitation on storing the target execution result based on the target object in step 302. The specific steps include:

[0111] Step 401, detect whether the target execution result meets a preset condition.

[0112] Among them, the preset condition is that the data volume of the execution result is less than the first value, or the number of temporary files in the execution result is less than the second value. The first value and the second value can be set by technicians according to experience, or can be set according to other methods, which is not limited here.

[0113] In this step, the data volume of the target execution result can be obtained, and it is detected whether the data volume is less than the first value. When the data volume is less than the first value, step 402 is executed. When the data volume is not less than the first value, step 403 is executed. Or obtain the number of temporary files in the target execution result, and detect whether the number is less than the preset value. When it is less than the preset value, step 402 is executed. When it is not less than the preset value, step 403 is executed.

[0114] Step 402, when the target execution result meets the preset condition, send the target execution result to the target object.

[0115] In this step, when the target execution result meets the preset condition, it means that directly sending the target execution result to the target object will not cause excessive request pressure on the target object. Therefore, the terminal can directly establish a request connection with the target object, and then use the request connection to send the target execution result to the target object, so that the target object stores the target execution result.

[0116] Step 403, in case the target execution result does not meet the preset condition, analyze the target execution result to obtain the number of data transmissions, and send the target execution result to the target object according to the number of data transmissions.

[0117] In this step, when the target execution result does not meet the preset condition, it means that sending the target execution result to the target object will cause excessive request pressure on the target object. Therefore, it is necessary to divide the target execution result into multiple execution sub-results. Then, upload each execution sub-result to the target object in batches so that the target object can store the target execution result.

[0118] Furthermore, when the preset condition is that the data volume of the target execution result is not less than the preset value, the number of transmissions can be determined according to the ratio of the data volume to the preset value, that is, how many execution sub-results need to be divided. Then, divide the execution result according to the number of transmissions to obtain multiple execution sub-results, and then upload the multiple execution sub-results to the target object multiple times.

[0119] In practice, after a certain execution sub-result is successfully uploaded, the next execution sub-result can be uploaded. Or, upload an execution sub-result every preset time interval until all execution sub-results are uploaded to the target object.

[0120] Among them, the preset time interval is set by technicians according to experience. For example, the preset time interval is 10 seconds. If the time point of the last upload of the execution sub-result is 10:10:10, then the next execution sub-result will be uploaded at 10:10:20. It can also be determined according to the data transmission duration and the object processing duration. For example, the data transmission duration is 4 seconds and the object processing duration is 3 seconds, then the preset time interval is the sum of the data transmission duration and the object processing duration, that is, 7 seconds.

[0121] Among them, since after the target object stores the execution result, it will also return data to the target object to indicate whether the target object has successfully stored the execution result. Therefore, the data transmission duration generally includes the duration from the terminal to the target object and the duration from the target object to the terminal. The object processing duration is the duration for the target object to receive the execution result and start returning data.

[0122] In the embodiments of the present application, in order to avoid the terminal from frequently establishing a request connection between the terminal and the target object, after all processes are completed, the execution results stored in the local storage device can be obtained and sent to the target object so that the target object can store these execution results. Therefore, the embodiments of the present application provide a data upload method, and the strategy is as Figure 5 shown, and the specific steps include:

[0123] Step 501, detect whether the target task is completed.

[0124] In this step, by detecting whether the target task is completed, it is determined whether all processes of the target task are completed. When all these processes are completed, the execution results of these processes are obtained to send these execution results to the target object.

[0125] Step 502, when the target task is completed, obtain all execution results of the target task and send the execution results to the target object.

[0126] In this step, when all processes of the target task are completed, a request connection between the terminal and the target object is established, and all execution results of the target task are obtained. Through this request connection, the execution results are sent to the target object.

[0127] In the embodiment of the present application, after the target object receives the execution results, the correctness of the execution results can also be verified. After the verification passes, the target object returns a message of successful storage to the terminal. After the verification fails, the target object returns a message of failed storage to the terminal, so that the terminal resends the execution results to the target object according to this message. Therefore, the embodiment of the present application provides a data processing method, and the method is as Figure 6 shown, and the specific steps include:

[0128] Step 601, receive the data returned by the target object.

[0129] In this step, the terminal receives the data sent by the target object and detects whether the data is an abnormal status code. Herein, the abnormal status code is a set of codes used to identify abnormal situations during the data storage process.

[0130] Step 602, when the data is an abnormal status code, obtain the required execution results in the local storage device and send the execution results to the target object.

[0131] In this step, when the data sent by the target object is an abnormal status code, it indicates that the target object fails to successfully store the execution results. The terminal can resend the execution results. Therefore, the terminal needs to re-obtain the execution results sent last time in the local storage device and send the execution results to the target object so that the target object stores the execution results.

[0132] Furthermore, when the terminal receives the abnormal status code returned by the target object, the terminal can immediately re-obtain the execution results sent last time in the local storage device and send the execution results to the target object. To avoid the failure of storage again due to the same reason, the terminal can obtain the required execution results from the local storage device again after a preset time period and send the execution results to the target object.

[0133] Step 603: When the data is not an abnormal status code, mark the status of the corresponding process as successful.

[0134] In this step, when the data sent by the target object is not an abnormal status code, it indicates that the target object has successfully stored the execution result. Obtain the execution result sent last time, and mark the status of the process corresponding to this execution result as successful.

[0135] After that, it is also possible to detect whether there is an execution result to be sent in the local storage device. If so, send the execution result to be sent to the target object.

[0136] It should be noted that when the terminal sends a certain execution result, or when the terminal successfully sends a certain execution result, the status of the process corresponding to this execution result can be modified to the success status.

[0137] As Figure 7 shown, the embodiment of the present application provides a schematic diagram of data processing. In this diagram, the target task is divided into multiple processes. The execution result of each process is stored in the local storage device and uploaded to the target object in an asynchronous multi-threaded writing manner, so that the target object stores the execution result of each process.

[0138] As Figure 8 shown, the embodiment of the present application provides a data processing device. This device corresponds to the method embodiment and specifically includes:

[0139] An acquisition unit 801, configured to acquire a target task to be processed by the terminal;

[0140] A determination unit 802, configured to determine a process set for executing the target task and an execution order corresponding to the process set;

[0141] An execution unit 803, configured to sequentially execute the processes according to the execution order to obtain an execution result corresponding to each process;

[0142] A storage unit 804, configured to store the execution result in the local storage device and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in an object form.

[0143] Optionally, the device further includes a storage unit 804, configured to:

[0144] During the process of executing each process according to the execution order, detect whether there is a target process that has been executed;

[0145] When there is a target process that has been executed, obtain the target execution result corresponding to the target process and upload the target execution result based on the target object.

[0146] Optionally, the storage unit 804 is configured to:

[0147] Detect whether the target execution result meets a preset condition;

[0148] When the target execution result meets the preset condition, send the target execution result to the target object;

[0149] When the target execution result does not meet the preset condition, analyze the target execution result to obtain the number of data sending times; according to the number of data sending times, send the target execution result to the target object.

[0150] Optionally, the storage unit 804 is configured to:

[0151] Detect whether the target task is completed;

[0152] When the target task is completed, obtain all execution results of the target task and send all execution results to the target object.

[0153] Optionally, the device further includes a data return unit 805, and the data return unit 805 is configured to:

[0154] Receive data returned by the target object;

[0155] When the data is an abnormal status code, obtain the required execution result in the local storage device and send the execution result to the target object;

[0156] When the data is not an abnormal status code, mark the status of the corresponding process as successful.

[0157] Optionally, the device further includes a partitioning unit 806, and the partitioning unit 806 is used for:

[0158] Partition the execution result according to a data partitioning strategy to obtain result elements corresponding to the execution result;

[0159] Create a corresponding temporary file for the result element and use the temporary file to store the corresponding result element to obtain a temporary file corresponding to the execution result;

[0160] The storage unit 804 is configured to:

[0161] Upload the temporary file to the target object based on a preset storage strategy.

[0162] Optionally, the device further includes a creation unit 807, and the creation unit 807 is configured to:

[0163] Create a request connection between the terminal and the target object according to the temporary file in the execution result, and obtain the request connection corresponding to the target execution result. The request connection and the temporary file are in one-to-one correspondence.

[0164] As Figure 9 shown, an embodiment of the present application provides a data processing device, including a processor 901, a communication interface 902, a memory 903, and a communication bus 904. Among them, the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904.

[0165] The memory 903 is used to store computer programs.

[0166] In an embodiment of the present application, when the processor 901 executes the program stored on the memory 903, it implements the data processing method provided by any one of the foregoing method embodiments, including:

[0167] Obtain the target task to be processed by the terminal.

[0168] Determine the process set for executing the target task and the execution order corresponding to the process set.

[0169] Execute each process in the process set in sequence according to the execution order, and obtain the execution result corresponding to each process.

[0170] Store the execution result in the local storage device, and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in the form of an object.

[0171] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps executed by the data processing method provided by any one of the foregoing method embodiments.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0174] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "comprising", "including", "containing", and "having" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the particular order described or illustrated, unless the order of performance is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0175] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, The method includes: Obtaining a target task to be processed by the terminal; Determining a process set for executing the target task and an execution order corresponding to the process set; Sequentially executing each process in the process set according to the execution order to obtain an execution result corresponding to each process; Storing the execution result in a local storage device, and uploading the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in an object form.

2. The method according to claim 1, wherein The uploading the execution result to the target object based on a preset storage policy includes: During the process of executing each process according to the execution order, detecting whether there is a target process that has been executed; When there is a target process that has been executed, obtaining the target execution result corresponding to the target process, and uploading the target execution result based on the target object.

3. The method according to claim 2, wherein The uploading the target execution result based on the target object includes: Detecting whether the target execution result meets a preset condition; When the target execution result meets the preset condition, sending the target execution result to the target object; When the target execution result does not meet the preset condition, analyzing the target execution result to obtain the number of data sending times; and sending the target execution result to the target object according to the number of data sending times.

4. The method according to claim 1, characterized in that, The uploading the execution result to the target object based on a preset storage policy includes: Detecting whether the target task has been executed; When the target task has been executed, obtaining all execution results of the target task, and sending all execution results to the target object.

5. The method according to claim 1, characterized in that The method further includes: Receiving data returned by the target object; When the data is an abnormal status code, obtaining the required execution result in the local storage device, and sending the execution result to the target object; When the data is not an abnormal status code, marking the status of the corresponding process as successful.

6. The method according to claim 1, wherein The method further includes: Dividing the execution result according to a data division policy to obtain result elements corresponding to the execution result; Creating a corresponding temporary file for the result element, and using the temporary file to store the corresponding result element to obtain a temporary file corresponding to the execution result; The uploading the execution result to the target object based on a preset storage policy includes: Uploading the temporary file to the target object based on a preset storage policy.

7. The method according to claim 6, wherein The method further includes: Creating a request connection between the terminal and the target object according to the temporary file in the execution result to obtain a request connection corresponding to the target execution result, and the request connection corresponds to the temporary file one by one.

8. A data processing device, characterized in that, The device includes: An obtaining unit, configured to obtain a target task to be processed by the terminal; A determining unit, configured to determine a process set for executing the target task and an execution order corresponding to the process set; An executing unit, configured to sequentially execute each process in the process set according to the execution order to obtain an execution result corresponding to each process; A storage unit for storing the execution result in a local storage device and uploading the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in the form of an object.

9. A data processing device, characterized in that, Comprising: At least one communication interface; At least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: Obtain a target task to be processed by the terminal; Determine a process set for executing the target task and an execution order corresponding to the process set; Execute each process in the process set in sequence according to the execution order to obtain an execution result corresponding to each process; Store the execution result in a local storage device and upload the execution result to the target object based on a preset storage policy, so that the target object stores the execution result in the form of an object.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 7.

Images