A data processing method, device, apparatus, and storage medium

Abstract

The application provides a data processing method, device and equipment and a storage medium. The method comprises the following steps: obtaining to-be-processed data and a pre-designed calculation graph corresponding to the to-be-processed data; the pre-designed calculation graph represents the relationship between a plurality of operators, the plurality of operators comprise a connection operator and a task operator determined by a data processing task of the to-be-processed data, the task operator comprises an upstream task operator and at least two concurrent task operators; based on the upstream task operator, the to-be-processed data is processed to obtain an upstream data processing result; based on the at least two concurrent task operators, the upstream data processing result is processed concurrently to obtain a concurrent data processing result corresponding to each concurrent task operator; based on the connection operator, the concurrent data processing results corresponding to each concurrent task operator are combined to obtain a data combination result. The application can realize concurrent calculation of a single message among a plurality of operators, shorten the message processing time and improve the system response speed.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and specifically relates to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] In existing real-time computing frameworks, computing tasks are typically decomposed into several operators internally, and the logical relationships or order between these operators are constructed into a Directed Acyclic Graph (DAG) structure.

[0003] Figure 1 This is a real-time computation graph structure in the prior art. In this computation graph structure, data first flows into operator node A1. After processing, there are two branches: one part of the data flows to operator node B1, and the other part flows to operator node C1 for processing, and then flows to operator node D1.

[0004] but Figure 1 In the real-time computation graph structure, a message is processed only in one operator node at a time. It only considers the concurrent processing of multiple messages, but does not consider the concurrent operation of a single message across multiple operators, which results in a long message processing time and a slow system response speed. Summary of the Invention

[0005] To enable concurrent operation of a single message across multiple operators, shorten message processing time, and improve system response speed, this invention proposes a data processing method, apparatus, device, and storage medium.

[0006] On one hand, the present invention proposes a data processing method, the method comprising:

[0007] Obtain the data to be processed and the preset computation graph corresponding to the data to be processed; the preset computation graph represents the relationship between multiple operators, the multiple operators include connection operators and task operators determined by the data processing task of the data to be processed, the task operators include upstream task operators and at least two concurrent task operators;

[0008] Based on the upstream task operator, the data to be processed is processed to obtain the upstream data processing result;

[0009] Based on the at least two concurrent task operators, the upstream data processing results are processed concurrently to obtain the concurrent data processing results corresponding to each concurrent task operator.

[0010] Based on the connection operator, the concurrent data processing results corresponding to each concurrent task operator are combined to obtain the data combination result.

[0011] On the other hand, embodiments of the present invention provide a data processing apparatus, the apparatus comprising:

[0012] An acquisition module is used to acquire the data to be processed and the preset computation graph corresponding to the data to be processed; the preset computation graph represents the relationship between multiple operators, the multiple operators include connection operators and task operators determined by the data processing task of the data to be processed, the task operators include upstream task operators and at least two concurrent task operators;

[0013] The upstream processing module is used to process the data to be processed based on the upstream task operator to obtain the upstream data processing result;

[0014] The concurrent processing module is used to perform concurrent processing on the upstream data processing results based on the at least two concurrent task operators, so as to obtain the concurrent data processing results corresponding to each concurrent task operator.

[0015] The combination module is used to combine the concurrent data processing results corresponding to each concurrent task operator based on the connection operator to obtain the data combination result.

[0016] On the other hand, the present invention proposes an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0017] On the other hand, the present invention proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the data processing method as described above.

[0018] The data processing method, apparatus, device, and storage medium proposed in this invention process data to be processed through a preset computation graph. This preset computation graph includes a connection operator and task operators determined by the data processing tasks of the data to be processed. The connection operator is designed to combine the data processing results of at least two concurrent task operators. After the upstream task operator processes the data to be processed, the upstream data processing result is simultaneously sent to at least two concurrent task operators. These at least two concurrent task operators then concurrently process the upstream data processing result, obtaining the concurrent data processing result corresponding to each concurrent task operator. Next, the connection operator combines the concurrent data processing results corresponding to each concurrent task operator to obtain a combined data result. This invention introduces a concurrent execution mechanism for operators and additionally designs a connection operator capable of combining the data processing results of at least two concurrent task operators, enabling a single message to run concurrently among multiple concurrent task operators, thereby shortening message processing latency and improving the system's rapid response capability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 It is a real-time computing graph structure in existing technologies.

[0021] Figure 2 This is a schematic diagram of the implementation environment of a data processing method provided in an embodiment of the present invention.

[0022] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a process for constructing a preset computation graph provided by an embodiment of the present invention.

[0024] Figure 5 This is an example diagram of a DAG provided in an embodiment of the present invention.

[0025] Figure 6 This is an example diagram of a message storage structure provided in an embodiment of the present invention.

[0026] Figure 7 This is an example diagram of the task operator execution mechanism provided in an embodiment of the present invention.

[0027] Figure 8This is a schematic diagram of a process for storing concurrent data processing results provided by an embodiment of the present invention.

[0028] Figure 9 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention.

[0029] Figure 10 This is an optional structural diagram of the blockchain system provided in an embodiment of the present invention.

[0030] Figure 11 This is an optional schematic diagram of the block structure provided in an embodiment of the present invention.

[0031] Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0032] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0033] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. The backend services of cloud computing systems require substantial computing and storage resources, such as those for video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may eventually possess its own identification mark, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. Specifically, cloud technology includes technical fields such as security, big data, databases, industry applications, networking, storage, management tools, and computing.

[0034] Specifically, embodiments of the present invention relate to the field of computing technology in cloud technology.

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0037] Figure 2 This is a schematic diagram of the implementation environment of a data processing method provided in an embodiment of the present invention. For example... Figure 2 As shown, this implementation environment may include at least terminal 01 and server 02. Terminal 01 and server 02 can be directly or indirectly connected via wired or wireless communication, and this invention does not impose any limitations on this. For example, terminal 01 uploads data to be processed to server 02 via wired or wireless communication, and server 02 sends data combination results to terminal 01 via wired or wireless communication.

[0038] Specifically, terminal 01 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0039] Specifically, terminal 01 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Server 02 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 01 and server 02 can be directly or indirectly connected via wired or wireless communication, and this invention does not impose any limitations on this connection.

[0040] It should be noted that, Figure 2 This is just one example.

[0041] This invention applies to real-time computing framework scenarios, including but not limited to Storm, Spark Streaming, and Flink. Storm is a distributed, fault-tolerant real-time computing system that provides a set of general primitives for distributed real-time computing, which can be used in "stream processing" to process messages and update databases in real time. Spark Streaming is an extension of the core application programming interface (API) of the Spark computing engine, enabling high-throughput, fault-tolerant real-time streaming data processing. Flink is an open-source stream processing framework whose core is a distributed streaming data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-paradigm-paradigm manner, and its pipelined runtime system can execute both batch and stream processing programs. Furthermore, Flink's runtime itself also supports the execution of iterative algorithms.

[0042] Specifically, real-time computing is generally used for massive amounts of data, referring to the real-time generation, transmission, computation, and display of data. It emphasizes low latency from data generation to its impact on the final result.

[0043] Specifically, real-time computing can typically be divided into a three-layer structure:

[0044] The top layer is the application expression layer, which decomposes the computational task into a DAG structure formed by several operators through certain methods. The operators are the nodes of the DAG graph, representing the application logic written by the user; the edges between the graph nodes represent the channels for data flow. Its core is the convenience of expression, mainly to facilitate application developers to quickly describe or build applications.

[0045] The middle layer is the DAG execution engine layer. Its main purpose is to transform and map the DAG computation tasks expressed in a special way in the upper layer and deploy them to the physical machine cluster in the lower layer for execution. This layer is the core component of DAG computation. The scheduling of computation tasks, the fault tolerance of the underlying hardware, the transmission of data and management information, and the management and normal operation of the entire system all need to be completed by this layer.

[0046] The lowest layer is the physical machine cluster, which is a distributed computing environment built from a large number of physical machines. This is where computing tasks are ultimately executed.

[0047] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. This method can be used for... Figure 2In the implementation environment described herein, the steps of the methods described in the embodiments or flowcharts are provided. However, based on conventional or non-inventive labor, more or fewer steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 3 As shown, the method may include:

[0048] S101. Obtain the data to be processed and the preset computation graph corresponding to the data to be processed; the preset computation graph represents the relationship between multiple operators, including connection operators and task operators determined by the data processing task of the data to be processed, and task operators including upstream task operators and at least two concurrent task operators.

[0049] In this embodiment of the invention, when processing the data to be processed, the data to be processed and the preset calculation graph corresponding to the data to be processed are obtained.

[0050] In this embodiment of the invention, the preset computation graph can be a DAG, which refers to a directed graph without loops.

[0051] The DAG in this embodiment of the invention includes two types of operators. One type is a task operator obtained by decomposing the corresponding data processing task, which includes at least an upstream task operator and at least two concurrent task operators. The other type is an additionally designed connection operator, which is designed to combine the data processing results of the upstream task operators. The DAG is constructed by constructing the various operators according to their logical relationships or execution order.

[0052] In this embodiment of the invention, the DAG can be pre-constructed before processing the data to be processed, that is, before S101, the method may also include: S100. Constructing a preset computation graph.

[0053] Specifically, such as Figure 4 As shown, S100 may include:

[0054] S1001. Data processing task to obtain data to be processed.

[0055] S1003. Analyze the data processing task and determine the task operators required to process the data to be processed.

[0056] S1005. Determine the operator type and configuration parameters of the task operator; the configuration parameters of the task operator include the field offset parameters that record the corresponding field offset position information.

[0057] S1007. Based on the operator type and configuration parameters of the task operator, determine the task operators with mutual dependencies and the task operators without mutual dependencies.

[0058] S1009. Determine the execution order of interdependent task operators based on their dependencies.

[0059] S10011. Connect task operators with interdependencies according to their respective execution order, and combine the outputs of task operators without interdependencies through the connection operator to obtain a preset computation graph; the configuration parameters of the connection operator include statistical parameters of task operators without interdependencies, at least two concurrent task operators belong to task operators without interdependencies, and the upstream task operator is the task operator in the preset computation graph whose execution order is ahead of multiple concurrent task operators.

[0060] In the data processing process, different data processing tasks will be generated for one or more related data streams. Each data processing task needs to be implemented through one or more calculation steps. Each independent calculation operation involved is called an operator.

[0061] As described in S1003, after determining the data processing task of the data to be processed, the data processing task can be decomposed to obtain multiple task operators required to process the data to be processed, and there are upstream and downstream dependencies between each task operator.

[0062] As described in S1005, after determining the task operators, the type of each task operator can be determined, and corresponding configuration parameters can be obtained according to the type of each task operator. These configuration parameters are configured according to the corresponding data processing task and may include business logic parameters, including but not limited to: upstream operators or input streams of the current operator, logical parameters of output fields, etc. For example, the configuration parameters may include field offset parameters of the corresponding field offset position information.

[0063] As described in S1007, the dependencies between various task operators can be resolved based on the configuration parameters of each task operator, thereby determining the task operators with mutual dependencies and the task operators without mutual dependencies (i.e., those that can run concurrently).

[0064] As described in S1009, for task operators with interdependent relationships, the execution order between these interdependent task operators can be determined based on the dependency relationship. As described in S10011, for task operators with interdependent relationships, they can be connected according to the corresponding execution order. For task operators without interdependent relationships, an additional connection operator can be designed to combine the data processing results of task operators without interdependent relationships, thereby obtaining the DAG.

[0065] It should be noted that the connection operator in this embodiment of the invention differs from the task operator in that it is not directly decomposed from the data processing task, but is designed separately for the decomposed task operator. Similar to the task operator, the connection operator also requires parameter configuration. The configuration parameters of the connection operator may include statistical parameters for counting the number of upstream task operators (i.e., task operators that do not have mutual dependencies, i.e., at least two concurrent task operators).

[0066] In some feasible embodiments, after obtaining the DAG, if the number of decomposed task operators and the number of connection operators used are large, the DAG can be divided into multiple computational units, such that each computational unit includes an upstream task operator, at least two concurrent task operators, and one connection operator. For each computational unit, the operators before the connection operator can be considered concurrent task operators, the operators before the concurrent task operators can be considered upstream task operators, and if there are other task operators after the connection operator, then the operators after the connection operator can be considered downstream task operators. It should be noted that the connection operator or downstream task operator of the previous computational unit may be the upstream task operator of the next computational unit.

[0067] If there are multiple computing units, each computing unit can be traversed during data processing, and S103-S1011 can be executed during the traversal of each computing unit.

[0068] The following example, using an advertising company's advertising development business, illustrates S100:

[0069] In the commercial promotion of advertising, it is necessary to obtain and analyze user click behavior logs for advertising. These logs serve as data streams, i.e., data to be processed. The data processing task involves analyzing how much to charge for each advertisement. Depending on the data stream source and the uncertainty of user click behavior, this data processing task needs to be broken down. For example, if duplicate clicks are present in the user click behavior logs, deduplication or de-duplication calculations are required. If different data streams contain user click behavior data for the same advertisement, multiple aggregation calculations (e.g., first aggregation and second aggregation operations) are needed. Finally, summation (SUM) is required for user click behavior data for the same advertisement at different times. The deduplication, first aggregation, second aggregation, and summation involved are independent task operators that can be configured with parameters according to the data processing task, and these parameter configurations are recorded in the corresponding control structures.

[0070] After decomposing the above task operators, we can determine the task operators with interdependencies (the first and second aggregates depend on deduplication, and summation depends on the first and second aggregates) and the task operators without interdependencies (there is no interdependency between the first and second aggregates, and the first and second aggregates are task operators that can run concurrently) based on the operator type and configuration parameters of the task operators.

[0071] Based on the dependencies between interdependent task operators, the execution order of these operators is determined as: deduplication, aggregation (first aggregation and second aggregation), and summation. Deduplication, aggregation (first aggregation and second aggregation), and summation are then connected sequentially according to their respective execution order. Since the first aggregation and second aggregation do not have a dependency relationship, an additional connection operator is designed to connect them. This connection operator can combine the data processing results of the first and second aggregations and transmit the combined data processing result to the summation operator, thus obtaining the corresponding Directed Acyclic Graph (DAG). An example of this DAG can be shown below. Figure 5 The deduplication in this DAG is equivalent to the upstream task operator (A2 operator), the first aggregation and the second aggregation are equivalent to the concurrent task operators (B2 operator and C2 operator), and the summation is equivalent to the downstream task operator (D2 operator).

[0072] It should be noted that there can be multiple upstream task operators, which are connected sequentially according to their dependencies. Similarly, there can be multiple downstream task operators, which are connected sequentially according to their dependencies.

[0073] In this embodiment of the invention, the process of constructing a DAG is equivalent to designing the application expression layer of the real-time computing framework. During the design of the application expression layer, an additional connection operator is designed so that it can combine the calculation results of multiple upstream task operators (i.e. concurrently running task operators), thereby enabling a single message to be processed in multiple task operators. This introduces a message concurrent execution mechanism, thereby shortening message processing latency and improving the system's response speed.

[0074] In this embodiment of the invention, during the DAG construction process, the message storage structure can be pre-designed. In memory, the unified message format transmitted between graph nodes is a contiguous storage space. The original information when data flows into the system constitutes the initial fields of the message. After being processed by each node, new fields are generated, and all fields are stored in the contiguous storage space corresponding to the message structure. Figure 6 The diagram shown is an example of a message storage structure. Fields 1, 2, and 3 can be... Figure 5 The data transmitted from the upstream task operator A2 in the middle, Figure 5 After the B2 operator in the code processes this field, it can generate... Figure 6 The new field (i.e., field 5) is entered into the message storage structure. Based on the corresponding field offset information, the starting position of the field in the B2 operator is determined, and field 5 is stored in the message storage structure according to this starting position. Figure 5 After the C2 operator in the code processes this field, it can generate... Figure 6 The new field (i.e., field 4) is entered, and the starting position of the field of the C2 operator is determined according to the corresponding field offset information. Based on this starting position, field 4 is stored in the message storage structure.

[0075] In this embodiment of the invention, during the DAG construction process, a corresponding task operator node control structure can be designed for each task operator to record the configuration parameters corresponding to the task node, such as field offset parameters that record field offset position information. Similarly, a corresponding connection operator node control structure can be designed for connection operators to record the configuration parameters corresponding to the connection node, such as statistical parameters that record the number of upstream task operators (i.e., at least two concurrent task operators).

[0076] In this embodiment of the invention, during the DAG construction process, the execution mechanism of the task operators can be pre-designed: for each task operator (including upstream task operators, concurrent task operators, and downstream task operators), it processes data according to its own logic, writes the result to the storage location corresponding to the operator node, and then passes the processed data to the connection operator. Figure 7 The diagram shown is an example of a task operator execution mechanism. Figure 5Operator A2 passes the data (i.e., message fields 1, 2, and 3) to operator B2. Operator B2 processes the received data to obtain a newly generated field (i.e., field 5). Based on the corresponding field offset position information and the required storage space, it determines the starting position of the field and stores field 5 at that position. Finally, operator B2 transmits the data processing result to the connection operator.

[0077] In this embodiment of the invention, during the DAG construction process, the execution mechanism of the connection operator can be pre-designed: the connection operator is designed to receive the data processing results of the upstream concurrent task operators, and check whether the upstream concurrent task operators have all completed the processing operations on the corresponding data. If so, the data processing results of the upstream concurrent task operators are combined, and finally the combined data is passed to the next operator for processing.

[0078] In this embodiment of the invention, the design of the message storage structure, the control structure of the operator nodes, and the execution mechanism of the operators are essentially the design of the execution engine layer in a real-time computing framework. During the design of the execution engine layer, a field offset parameter is set in the control structure of each task operator to record field offset position information. This allows the corresponding storage location to be quickly determined based on this field offset parameter during data processing, improving the accuracy and efficiency of data storage. Furthermore, a statistical parameter is set in the control structure of the connection operator to record the number of upstream task operators (i.e., at least two concurrent task operators). This allows the connection operator to accurately determine whether the upstream concurrent task operators have finished processing the corresponding data based on this statistical parameter. If so, the data processing results of the upstream concurrent task operators are combined and transmitted to the next operator node for processing. This avoids errors in data combination caused by performing subsequent combination operations before the upstream concurrent task operators have finished processing all the corresponding data, further improving the accuracy of data processing.

[0079] S103. Based on the upstream task operator, process the data to be processed to obtain the upstream data processing result.

[0080] In one feasible embodiment, if the upstream task operator includes a preprocessing task operator, then S103 may include:

[0081] Obtain the operator type and configuration parameters of the preprocessing task operator.

[0082] Based on the operator type and configuration parameters of the preprocessing operator, the data to be processed is processed into data in a preset format.

[0083] Use data in a preset format as the result of upstream data processing.

[0084] In this embodiment of the invention, after the data to be processed flows into the DAG, if the DAG contains only one computing unit, the data to be processed can be directly processed through the upstream task operator in that computing unit. If the DAG contains multiple computing units, the data to be processed can be processed first through the upstream task operator in the first computing unit.

[0085] Hereinafter, the task operator and the connection operator are taken as the first computational unit in this DAG, and the preset computational graph is used as... Figure 5 Taking the DAG in the example, let's illustrate S103 with an example:

[0086] The data to be processed is fed into the DAG, and then processed by the A2 operator in the first computational unit of the DAG.

[0087] Since the data to be processed needs to be packaged into a unified data format when it flows into the DAG, the first computation unit can include a preprocessing task operator, that is, the A2 operator can be a preprocessing task operator. As mentioned above, the operator type and corresponding configuration parameters of the A2 operator can be obtained, and the data to be processed can be processed according to the operator type and corresponding configuration parameters of the A2 operator to obtain unified message format data (such as {a1,a2,a3}). This unified message format data is the upstream data processing result, where {a1,a2,a3} indicates that the upstream data processing result contains fields a1,a2,a3.

[0088] In this embodiment, the data to be processed is processed into a preset format by a preprocessing task operator. On the one hand, this ensures the consistency of the data flowing into multiple subsequent concurrent task operators, and on the basis of data consistency, it can effectively improve the accuracy of data processing. On the other hand, since the data has been processed into a unified preset data format before flowing into each concurrent task operator, each concurrent task operator does not need to process the data to be processed separately after receiving the data, reducing the number of times and the amount of data processing, thereby reducing the burden of system data processing and improving data processing efficiency.

[0089] In this embodiment of the invention, after S103, the method may further include:

[0090] Obtain the field offset parameters corresponding to the upstream task operator.

[0091] Obtain the first field offset position information corresponding to the upstream data processing result from the field offset parameters corresponding to the upstream task operator.

[0092] Determine the initial storage space required for the upstream data processing results.

[0093] Based on the first field offset position information and the first storage space, the upstream data processing results are stored in the preset storage structure.

[0094] In this embodiment of the invention, since a field offset parameter recording the corresponding field offset position information has been pre-configured for each task operator during DAG construction, and this field offset parameter is set in the corresponding node control structure, after obtaining the upstream data processing result, the corresponding first field offset position information can be directly obtained from the node control structure corresponding to the upstream task operator, and the first storage space required for the upstream data processing result can be calculated. Then, based on the first field offset position information and the first storage space, a storage location is allocated for the upstream data processing result in a preset storage structure, and the storage location is recorded in the node control structure corresponding to the upstream task operator. Finally, the upstream data processing result is stored in the preset storage structure.

[0095] S105. Based on at least two concurrent task operators, perform concurrent processing on the upstream data processing results to obtain the concurrent data processing results corresponding to each concurrent task operator.

[0096] Specifically, prior to S105, the method may further include:

[0097] The upstream data processing results are simultaneously transmitted to at least two concurrent task operators.

[0098] Specifically, S105 may include:

[0099] Obtain the operator type and configuration parameters corresponding to each concurrent task operator.

[0100] Based on the operator type and configuration parameters corresponding to each concurrent task operator, the upstream data processing results are processed concurrently to obtain the concurrent data processing results corresponding to each concurrent task operator. The concurrent data processing results include the fields corresponding to the upstream data processing results and the newly generated fields.

[0101] Specifically, after S105, the method may further include: S106. Storing the results of concurrent data processing. For example... Figure 8 As shown, S106 may include:

[0102] S10601. Calculate the second storage space required for the newly generated fields corresponding to each concurrent task operator.

[0103] S10603. Obtain the field offset parameters corresponding to each concurrent task operator.

[0104] S10605. Obtain the second field offset position information of the newly generated field corresponding to each concurrent task operator from the field offset parameters corresponding to each concurrent task operator.

[0105] S10607. Based on the second field offset position information of the newly generated field corresponding to each concurrent task operator and the corresponding second storage space, store the newly generated field corresponding to each concurrent task operator in the preset storage structure.

[0106] It should be noted that at least two concurrent task operators in S105 and the upstream task operators in S103 belong to the same computational unit in the DAG.

[0107] Hereinafter, the task operator and the connection operator are taken as the first computational unit in this DAG, and the preset computational graph is used as... Figure 5 Taking the DAG in the example, S105 and S106 are illustrated with examples:

[0108] In this embodiment of the invention, after operator A2 obtains the upstream data processing result (such as fields {a1, a2, a3}), the upstream data processing result can be simultaneously passed to operators B2 and C2. Then, operators B2 and C2 are obtained according to... Figure 7 The execution mechanism shown performs concurrent processing on the upstream data processing results. Specifically, it involves obtaining the operator types and configuration parameters corresponding to operators B2 and C2, and then performing concurrent processing on the upstream data processing results based on these parameters. This yields the concurrent data processing results corresponding to the B2 operator node (e.g., fields {a1,a2,a3,b1,b2,b3}) and the C2 operator node (e.g., fields {a1,a2,a3,c1,c2,c3}). From the concurrent data processing results corresponding to each concurrent task operator, it can be seen that the concurrent data processing results obtained by operators B2 and C2 include the fields corresponding to the upstream data processing results and newly generated fields (e.g., b1,b2,b3 or c1,c2,c3).

[0109] During DAG construction, a field offset parameter is pre-configured for each task operator to record the corresponding field offset position information, and this field offset parameter is set in the corresponding node control structure. Therefore, after obtaining the concurrent data processing results corresponding to each concurrent task operator, the second field offset position information of the newly generated field can be obtained from the field offset parameter corresponding to the concurrent task operator. The second storage space required for the newly generated field (such as b1, b2, b3 or c1, c2, c3) can be calculated. Then, based on the second field offset position information and the second storage space, the field start position information of the newly generated field (b1, b2, b3 or c1, c2, c3) is determined so as to allocate a storage location for the concurrent data processing result in the preset storage structure. This storage location is recorded in the node control structure corresponding to the concurrent task operator. Finally, based on this storage location, the newly generated field corresponding to the concurrent data processing result is stored in the preset storage structure.

[0110] In this embodiment, the corresponding offset position information is obtained from the pre-designed node control structure, and the corresponding storage location is determined based on the offset position information. On the one hand, this can reduce the error in determining the storage location and improve the accuracy of data storage. On the other hand, since the offset position information is recorded in the pre-designed node control structure, it can be directly obtained from the corresponding node control structure when storing data, without the need for calculation through other means, which reduces the computational load of the system, thereby reducing the system burden and improving the efficiency of data processing.

[0111] S107. Based on the connection operator, combine the concurrent data processing results corresponding to each concurrent task operator to obtain the data combination result.

[0112] Specifically, the method may also include:

[0113] After any concurrent task operator completes the processing operation on the upstream data processing result, the statistical quantity in the statistical parameter is incremented by 1.

[0114] Specifically, S107 may include:

[0115] Retrieve the statistical quantities from the statistical parameters.

[0116] When the number of statistics meets the preset threshold, it is determined that each concurrent task operator has completed the processing operation on the upstream data processing results. Then, based on the connection operator, the concurrent data processing results corresponding to each concurrent task operator are combined to obtain the data combination result.

[0117] It should be noted that the connection operator in S107, the upstream task operator in S103, and the concurrent task operator in S105 belong to the same computational unit in the DAG.

[0118] Hereinafter, the task operator and the connection operator are taken as the first computational unit in this DAG, and the preset computational graph is used as... Figure 5 Taking the DAG in the example, S107 is illustrated as follows:

[0119] After obtaining the concurrent data processing results corresponding to operators B2 and C2 in S105, the concurrent data processing results corresponding to operators B2 and C2 can be passed to the connection operator (i.e., operator C2). Operator C2 combines the concurrent data processing results to obtain the data combination result, which contains the field {a1,a2,a3,b1,b3,b3,c1,c2,c3}.

[0120] In this embodiment of the invention, when constructing a DAG, since a statistical parameter is pre-configured in the connection node to count the number of upstream task operators (i.e., at least two concurrent task operators), before data combination through the connection operator, relevant information can be extracted from the JoinInfo field corresponding to the concurrent data processing results of each concurrent task operator. This information determines whether the concurrent task operator has completed the processing operation on the upstream data processing results. If so, the statistical parameter in the connection node is incremented by 1. When the statistical quantity meets a preset threshold, the connection operator combines the concurrent data processing results to obtain the combined data result. If any concurrent task operator has not completed the processing operation on the upstream data processing result, the operation is terminated until all concurrent task operators have completed their processing, and then the processing results of all concurrent task operators are combined.

[0121] In this embodiment of the invention, since there may be multiple computing units in the DAG, each computing unit includes a connection operator, and each connection operator's node control structure is configured with statistical parameters, before combining data through the connection operator, the identity document (ID) of the connection node to be combined can be obtained, and the statistical parameters of the corresponding connection operator can be found by obtaining the ID, so as to perform subsequent operations based on the statistical parameters.

[0122] In this embodiment, statistical parameters are used to count whether all concurrent task operators have completed their corresponding data processing operations. Data is combined by connection operators only when the number of statistical parameters meets the preset threshold. If any concurrent operator has not completed its corresponding data processing operation, data combination is not performed. This avoids errors in data combination caused by performing subsequent combination operations before the upstream concurrent task operators have finished processing all the corresponding data, and further improves the accuracy of data processing.

[0123] In embodiments of the present invention, such as Figure 9 The task operator further includes a downstream task operator, which is a task operator in the preset computation graph that is executed after the connection operator. Therefore, after S107, the method may further include:

[0124] S109. Obtain the operator type and configuration parameters of the downstream task operator.

[0125] S1011. Based on the operator type and configuration parameters of the downstream task operator, process the data combination result to obtain the downstream data processing result.

[0126] It should be noted that this downstream task operator belongs to the same computational unit as the upstream task operator, concurrent task operator, and connection operator mentioned above.

[0127] Hereinafter, the task operator and the connection operator are taken as the first computational unit in this DAG, and the preset computational graph is used as... Figure 5 Taking the DAG in the example, S109-S1011 will be explained by way of example:

[0128] The connection operator passes the data combination result to the downstream task operator (i.e.) Figure 5 When using the D2 operator, the operator type and configuration parameters of the D2 operator can be obtained. Based on the operator type and configuration parameters, the data combination result is processed to obtain the downstream data processing result. Based on the corresponding field offset information and the required storage space, the storage location of the downstream data processing result in the preset storage structure is determined. Finally, the downstream data processing result is stored in the preset storage structure.

[0129] In a feasible embodiment, at least one of the data to be processed in S101, the field offset parameter in S1005, the upstream data processing result in S103, the concurrent data processing result in S105, the data combination result in S107, and the downstream data processing result in S1011 can be stored in the blockchain system. See also Figure 10 , Figure 10The diagram shown is an optional structural schematic of a blockchain system provided in an embodiment of the present invention. Multiple nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the blockchain system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0130] See Figure 10 The functions of each node in the blockchain system shown include:

[0131] 1) Routing: A basic function of nodes used to support communication between nodes.

[0132] In addition to routing capabilities, nodes can also have the following functions:

[0133] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0134] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0135] See Figure 11 , Figure 11 This is an optional schematic diagram of the block structure provided in an embodiment of the present invention. Each block includes the hash value of the transaction records stored in this block (the hash value of this block) and the hash value of the previous block. The blocks are connected through their hash values ​​to form a blockchain. Additionally, the block may also include information such as a timestamp when it was generated. A blockchain is essentially a decentralized database, a chain of data blocks linked together using cryptographic methods.

[0136] The data processing method provided in this embodiment of the invention has the following beneficial effects:

[0137] 1) In this embodiment of the invention, an additional connection operator is designed to combine the calculation results of multiple upstream task operators (i.e. concurrently running task operators), so that a single message can be processed in multiple task operators. That is, a message concurrent execution mechanism is introduced, which allows a single message to run concurrently among multiple concurrent task operators, thereby shortening the message processing latency and improving the system's fast response capability.

[0138] 2) In this embodiment of the invention, a field offset parameter is set in the control structure of each task operator to record the field offset position information. This allows the corresponding storage location to be quickly determined based on the field offset parameter during data processing, improving the accuracy and efficiency of output storage. Furthermore, a statistical parameter is set in the control structure of the connection operator to record the number of upstream concurrent task operators (i.e., at least two concurrent task operators). This allows the connection operator to accurately determine whether the upstream concurrent task operators have finished processing the corresponding data based on this statistical parameter. If so, the data processing results of the upstream concurrent task operators are combined and transmitted to the next operator node for processing. This avoids errors in data combination caused by directly performing subsequent combination operations before the upstream concurrent task operators have finished processing all the corresponding data, further improving the accuracy of data processing.

[0139] 3) In this embodiment of the invention, the corresponding offset position information is obtained from the pre-designed node control structure, and the corresponding storage position is determined according to the offset position information. On the one hand, the error in determining the storage position can be reduced and the accuracy of data storage can be improved. On the other hand, since the offset position information is recorded in the pre-designed node control structure, it can be directly obtained from the corresponding node control structure when storing data, without the need for calculation through other means, which reduces the computational load of the system, thereby reducing the system burden and improving the efficiency of data processing.

[0140] like Figure 12 As shown, embodiments of the present invention also provide a data processing apparatus, which may include at least:

[0141] The acquisition module 201 can be used to acquire the data to be processed and the preset computation graph corresponding to the data to be processed; the preset computation graph represents the relationship between multiple operators, including connection operators and task operators determined by the data processing task of the data to be processed, and the task operators include upstream task operators and at least two concurrent task operators.

[0142] The upstream processing module 203 can be used to process the data to be processed based on the upstream task operator to obtain the upstream data processing result.

[0143] The concurrent processing module 205 can be used to concurrently process the upstream data processing results based on at least two concurrent task operators to obtain the concurrent data processing results corresponding to each concurrent task operator.

[0144] The combination module 207 can be used to combine the concurrent data processing results corresponding to each concurrent task operator based on the connection operator to obtain the data combination result.

[0145] In this embodiment of the invention, the device may further include: a construction module for building the preset computational graph; specifically, the construction module may include:

[0146] The data processing task acquisition unit can be used to acquire data processing tasks that need to be processed.

[0147] The task operator determination unit can be used to analyze data processing tasks and determine the task operators required to process the data to be processed.

[0148] The first type of parameter determination unit can be used to determine the operator type and configuration parameters of the task operator; the configuration parameters of the task operator include the field offset parameters that record the corresponding field offset position information.

[0149] The dependency determination unit can be used to determine task operators with mutual dependencies and task operators without mutual dependencies based on the operator type and configuration parameters of the task operator.

[0150] The execution order determination unit can be used to determine the execution order of interdependent task operators based on the dependencies between them.

[0151] The preset computation graph determination unit can be used to connect task operators with interdependencies according to their respective execution order, and combine the output results of task operators without interdependencies through connection operators to obtain the preset computation graph. The configuration parameters of the connection operators include statistical parameters for counting at least two concurrent task operators; the configuration parameters of the connection operators include statistical parameters for counting task operators without interdependencies, at least two concurrent task operators are task operators without interdependencies, and the upstream task operator is the task operator in the preset computation graph whose execution order is ahead of multiple concurrent task operators.

[0152] In this embodiment of the invention, the upstream processing module 203 may include:

[0153] The second type of parameter determination unit can be used to obtain the operator type and configuration parameters of the preprocessing task operator.

[0154] The preprocessing unit can be used to process the data to be processed into a preset format based on the operator type and configuration parameters of the preprocessing operator.

[0155] The upstream data processing result acquisition unit can be used to obtain data in a preset format as the upstream data processing result.

[0156] In this embodiment of the invention, the device may further include:

[0157] The transmission module can be used to transmit upstream data processing results to at least two concurrent task operators simultaneously.

[0158] In this embodiment of the invention, the concurrent processing module 205 may include:

[0159] The third type of parameter determination unit can be used to obtain the operator type and configuration parameters corresponding to each concurrent task operator.

[0160] The concurrent data processing result acquisition unit can be used to perform concurrent processing on the upstream data processing result based on the operator type and configuration parameters corresponding to each concurrent task operator, and obtain the concurrent data processing result corresponding to each concurrent task operator. The concurrent data processing result includes the fields corresponding to the upstream data processing result and the newly generated fields.

[0161] Accordingly, the device may further include a concurrent data processing result storage module, which may include:

[0162] The second storage space determination unit can be used to calculate the second storage space required for the newly generated fields corresponding to each concurrent task operator.

[0163] The field offset parameter determination unit can be used to obtain the field offset parameters corresponding to each concurrent task operator.

[0164] The second field offset position information determination unit can be used to obtain the second field offset position information of the newly generated field corresponding to each concurrent task operator from the field offset parameters corresponding to each concurrent task operator.

[0165] The storage unit can be used to store the newly generated fields corresponding to each concurrent task operator in a preset storage structure based on the second field offset position information of the newly generated fields corresponding to each concurrent task operator and the corresponding second storage space.

[0166] In this embodiment of the invention, the device may further include:

[0167] The statistics module can be used to increment the statistical count in the statistics parameters by 1 after any concurrent task operator has completed its processing operation on the upstream data.

[0168] In this embodiment of the invention, the combination module 207 may further include:

[0169] The statistical quantity acquisition unit can be used to obtain statistical quantities from statistical parameters.

[0170] The data combination result acquisition unit can be used to determine that each concurrent task operator has completed the processing operation on the upstream data processing result when the statistical quantity meets the preset quantity threshold. Then, based on the connection operator, the concurrent data processing results corresponding to each concurrent task operator are combined to obtain the data combination result.

[0171] In this embodiment of the invention, the task operator further includes a downstream task operator, which is a task operator in a preset computation graph that is executed after the connecting operator. Therefore, the device may further include a downstream processing module, which may further include:

[0172] The third type of parameter determination unit can be used to obtain the operator type and configuration parameters of downstream task operators.

[0173] The downstream data processing result acquisition unit can be used to process the data combination results based on the operator type and configuration parameters of the downstream task operator to obtain the downstream data processing results.

[0174] It should be noted that the device embodiments provided in this invention are based on the same inventive concept as the method embodiments described above.

[0175] This invention also provides an electronic device for data processing, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the data processing method provided in the above method embodiments.

[0176] Embodiments of the present invention also provide a computer-readable storage medium that can be disposed in a terminal to store at least one instruction or at least one program related to implementing a data processing method in the method embodiments, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method provided in the above method embodiments.

[0177] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0178] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0179] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method provided in the above-described method embodiments.

[0180] The data processing method embodiments provided in this invention can be executed on a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 12 This is a hardware structure block diagram of a server for a data processing method provided in an embodiment of the present invention. For example... Figure 12As shown, the server 300 can vary significantly depending on its configuration or performance. It may include one or more Central Processing Units (CPUs) 310 (CPUs 310 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 330 for storing data, and one or more storage media 320 (e.g., one or more mass storage devices) for storing application programs 323 or data 322. The memory 330 and storage media 320 may be temporary or persistent storage. The program stored in the storage media 320 may include one or more modules, each module including a series of instruction operations on the server. Furthermore, the CPU 310 may be configured to communicate with the storage media 320 and execute a series of instruction operations stored in the storage media 320 on the server 300. The server 300 may also include one or more power supplies 360, one or more wired or wireless network interfaces 350, one or more input / output interfaces 340, and / or one or more operating systems 321, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0181] The input / output interface 340 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 300. In one example, the input / output interface 340 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 340 may be a radio frequency (RF) module for wireless communication with the Internet.

[0182] Those skilled in the art will understand that Figure 12 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 300 may also include... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown.

[0183] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0184] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0185] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0186] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized by, The method comprises: acquiring to-be-processed data and a pre-designed computation graph corresponding to the to-be-processed data; the pre-designed computation graph represents a relationship between a plurality of operators, the plurality of operators comprising a connection operator and a task operator determined by a data processing task of the to-be-processed data, the task operator comprising an upstream task operator and at least two concurrent task operators; the at least two concurrent task operators are task operators that do not have a mutual dependency relationship; a field offset parameter recording field offset position information is arranged in a control structure of each task operator; processing the to-be-processed data based on the upstream task operator to obtain an upstream data processing result, and simultaneously transmitting the upstream data processing result to the at least two concurrent task operators; concurrently processing the upstream data processing result based on the at least two concurrent task operators to obtain a respective concurrent data processing result corresponding to each concurrent task operator; the concurrent data processing result comprises a field corresponding to the upstream data processing result and a newly generated field; storing the respective newly generated field corresponding to each concurrent task operator in a pre-set storage structure based on field offset position information of the respective newly generated field corresponding to each concurrent task operator and required storage space; combining the respective concurrent data processing result corresponding to each concurrent task operator based on the connection operator to obtain a data combination result.

2. The method of claim 1, wherein, The method further comprises a step of constructing the pre-designed computation graph, and the step of constructing the pre-designed computation graph comprises: acquiring a data processing task of the to-be-processed data; analyzing the data processing task to determine the task operator required for processing the to-be-processed data; determining an operator type and configuration parameters of the task operator; the configuration parameters of the task operator comprise a field offset parameter recording corresponding field offset position information; determining task operators having a mutual dependency relationship and task operators not having a mutual dependency relationship based on the operator type and configuration parameters of the task operator; determining an execution order of the task operators having a mutual dependency relationship based on a dependency relationship between the task operators having a mutual dependency relationship; connecting the task operators having a mutual dependency relationship in the corresponding execution order, and combining output results of the task operators not having a mutual dependency relationship through the connection operator to obtain the pre-designed computation graph; the configuration parameters of the connection operator comprise a statistical parameter of the task operators not having a mutual dependency relationship, and the upstream task operator is a task operator in the pre-designed computation graph whose execution order is before the plurality of concurrent task operators.

3. The method of claim 2, wherein, When the upstream task operator comprises a preprocessing task operator, processing the to-be-processed data based on the upstream task operator to obtain an upstream data processing result comprises: acquiring an operator type and configuration parameters of the preprocessing task operator; processing the to-be-processed data into pre-set format data based on the operator type and configuration parameters of the preprocessing operator; taking the pre-set format data as the upstream data processing result. Correspondingly, after the upstream data processing result is obtained by processing the to-be-processed data based on the upstream task operator, the method further includes: obtaining a field offset parameter corresponding to the upstream task operator; obtaining first field offset position information corresponding to the upstream data processing result from the field offset parameter corresponding to the upstream task operator; determining a first storage space required by the upstream data processing result; storing the upstream data processing result in a preset storage structure based on the first field offset position information and the first storage space.

4. The method of claim 2, wherein, The concurrent processing of the upstream data processing result based on the at least two concurrent task operators to obtain respective concurrent data processing results corresponding to the respective concurrent task operators includes: obtaining an operator type and configuration parameters corresponding to each of the concurrent task operators; concurrently processing the upstream data processing result based on the operator type and configuration parameters corresponding to each of the concurrent task operators to obtain respective concurrent data processing results corresponding to each of the concurrent task operators.

5. The method of claim 2, wherein, The method further includes: after any one of the concurrent task operators completes the processing operation on the upstream data processing result, adding 1 to the statistical quantity in the statistical parameter.

6. The method of claim 5, wherein, The combination of the respective concurrent data processing results corresponding to each of the concurrent task operators based on the connection operator to obtain a data combination result includes: obtaining the statistical quantity in the statistical parameter; when the statistical quantity meets a preset quantity threshold, determining that each of the concurrent task operators has completed the processing operation on the upstream data processing result, and then combining the respective concurrent data processing results corresponding to each of the concurrent task operators based on the connection operator to obtain the data combination result.

7. The method of claim 2, wherein, The task operator further includes a downstream task operator, which is a task operator in the preset computation graph and has a processing sequence after the connection operator, and after the combination of the respective concurrent data processing results corresponding to each of the concurrent task operators based on the connection operator to obtain a data combination result, the method further includes: obtaining an operator type and configuration parameters of the downstream task operator; processing the data combination result based on the operator type and configuration parameters of the downstream task operator to obtain a downstream data processing result.

8. A data processing apparatus, characterized by, The device includes: an obtaining module, configured to obtain to-be-processed data and a preset computation graph corresponding to the to-be-processed data; the preset computation graph represents a relationship between a plurality of operators, the plurality of operators include a connection operator and a task operator determined by a data processing task of the to-be-processed data, the task operator includes an upstream task operator and at least two concurrent task operators; the at least two concurrent task operators are task operators without mutual dependency relationship; a field offset parameter recording field offset position information is arranged in a control structure of each of the task operators; an upstream processing module, configured to process the to-be-processed data based on the upstream task operator to obtain an upstream data processing result, and simultaneously transmit the upstream data processing result to the at least two concurrent task operators; The concurrent processing module is configured to perform concurrent processing on the upstream data processing result based on the at least two concurrent task operators, to obtain a concurrent data processing result corresponding to each of the concurrent task operators; the concurrent data processing result includes a field corresponding to the upstream data processing result and a newly generated field; The concurrent data processing result storage module is configured to store the newly generated field corresponding to each of the concurrent task operators in a preset storage structure based on field offset position information of the newly generated field corresponding to each of the concurrent task operators and required storage space. The combination module is configured to combine the concurrent data processing result corresponding to each of the concurrent task operators based on the connection operator, to obtain a data combination result.

9. The apparatus of claim 8, wherein, The device further includes a construction module configured to construct the preset design calculation graph, and the construction module includes: A data processing task acquisition unit is configured to acquire a data processing task of the data to be processed. A task operator determination unit is configured to analyze the data processing task, and determine the task operators required for processing the data to be processed. A first type parameter determination unit is configured to determine an operator type and configuration parameters of the task operators; the configuration parameters of the task operators include field offset parameters recording corresponding field offset position information. A dependency relationship determination unit is configured to determine task operators having a mutual dependency relationship and task operators not having a mutual dependency relationship based on the operator type and configuration parameters of the task operators. An execution order determination unit is configured to determine an execution order of the task operators having a mutual dependency relationship based on the dependency relationship between the task operators having a mutual dependency relationship. A preset design calculation graph determination unit is configured to connect the task operators having a mutual dependency relationship according to a corresponding execution order, and combine output results of the task operators not having a mutual dependency relationship through the connection operator, to obtain the preset design calculation graph; the configuration parameters of the connection operator include statistical parameters for the task operators not having a mutual dependency relationship; and the upstream task operator is a task operator in the preset design calculation graph, and the execution order of the upstream task operator is located before the multiple concurrent task operators.

10. The apparatus of claim 9, wherein, The upstream processing module includes: A second type parameter determination unit is configured to acquire an operator type and configuration parameters of a preprocessing task operator. A preprocessing unit is configured to process the data to be processed into preset format data based on the operator type and configuration parameters of the preprocessing operator. An upstream data processing result acquisition unit is configured to take the preset format data as the upstream data processing result.

11. The apparatus of claim 9, wherein, The concurrent processing module includes: A third type parameter determination unit is configured to acquire an operator type and configuration parameters of each of the concurrent task operators. A concurrent data processing result acquisition unit is configured to perform concurrent processing on the upstream data processing result based on the operator type and configuration parameters of each of the concurrent task operators, to obtain a concurrent data processing result corresponding to each of the concurrent task operators.

12. The apparatus of claim 9, wherein, The device further includes: The statistics module is configured to add one to a statistical number in the statistics parameter after any one of the concurrent task operators completes the processing operation on the upstream data processing result.

13. The apparatus of claim 12, wherein, The combination module includes: a statistical number acquisition unit configured to acquire a statistical number in the statistics parameter; a data combination result acquisition unit configured to, when the statistical number meets a preset number threshold, determine that each concurrent task operator completes the processing operation on the upstream data processing result, and then perform combination processing on the respective concurrent data processing results of the concurrent task operators based on the connection operator to obtain the data combination result.

14. The apparatus of claim 9, wherein, The task operator further includes a downstream task operator, which is a task operator in the pre-designed calculation graph and has a processing sequence after the connection operator. The device further includes a downstream processing module, which includes: a third type parameter determination unit configured to acquire an operator type and a configuration parameter of the downstream task operator; a downstream data processing result acquisition unit configured to process the data combination result based on the operator type and the configuration parameter of the downstream task operator to obtain the downstream data processing result.

15. An electronic device, comprising: The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method of any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method of any one of claims 1 to 7.

Images