Data processing method, electronic equipment and storage medium

Abstract

The invention provides a data processing method, electronic equipment and a storage medium, and the method comprises the steps: executing a plurality of workflows, the workflows comprise a plurality of nodes, and the nodes start to execute node work after applying for operation resources; receiving an intervention instruction input by a user during execution of the plurality of workflows; in response to the intervention instruction, pausing or terminating execution of the plurality of workflows; receiving update configuration information of the plurality of workflows input by a user; according to the update configuration information, restarting the plurality of workflows which are paused for execution, the update configuration information including priority information of the plurality of workflows; and according to the priority information and the time when each node of the plurality of workflows applies for the operation resource, allocating the operation resource to each node in the plurality of workflows.

Description

Technical Field

[0001] The embodiments in the present disclosure relate to the field of data processing, and particularly to a method, an electronic device, and a storage medium for data processing. Background Art

[0002] High-precision maps are an important part of an autonomous driving system. However, its production process, from point cloud data processing to generating maps with product specifications, requires a series of complex processes and also involves interactions of multiple tasks, multiple states, and multiple data.

[0003] Although some related automation systems in the prior art can perform automated transfer of some tasks, they cannot meet some specific requirements in the production of high-precision maps. For example, once a task is started, it cannot be dynamically adjusted. Therefore, a more intelligent workflow processing method is needed. Summary of the Invention

[0004] In view of this, the present disclosure proposes a data processing method, an electronic device, and a storage medium, which dynamically manage the production process of high-precision maps by performing multi-state adjustment on the workflow.

[0005] The first aspect of the present disclosure proposes a data processing method, including executing a plurality of workflows, where each workflow includes a plurality of nodes, and the nodes start to execute node work after obtaining computing resources; during the execution of the plurality of workflows, receiving an intervention instruction input by a user; in response to the intervention instruction, suspending or terminating the execution of the plurality of workflows; receiving updated configuration information of the plurality of workflows input by the user; according to the updated configuration information, restarting the plurality of workflows whose execution has been suspended, where the updated configuration information includes priority information of the plurality of workflows; and allocating computing resources to each node in the plurality of workflows according to the priority information and the time when each node in the plurality of workflows applies for computing resources.

[0006] The second aspect of the present disclosure proposes an electronic device, which includes a memory and a processor, and at least one computer program instruction is stored in the memory, and the at least one computer program instruction is loaded and executed by the processor to implement the data processing method proposed by the present disclosure.

[0007] The third aspect of the present disclosure proposes a computer-readable storage medium, and at least one computer program instruction is stored in the computer-readable storage medium, and when the at least one computer program instruction is executed by a processor, it can implement the data processing method proposed by the present disclosure. Description of the Drawings

[0008] The accompanying drawings exemplarily illustrate embodiments and form a part of the specification, and are used together with the written description of the specification to explain the exemplary embodiments of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements. Figure 1 It is a schematic diagram showing the process of a data processing method according to an embodiment of the present disclosure. Figure 2 It is a schematic diagram showing the allocation of computing resources to each node according to an embodiment of the present disclosure. Figure 3 It is a schematic diagram showing an aggregation node according to an embodiment of the present disclosure. Figure 4 It is a schematic diagram showing the process of a workflow according to an embodiment of the present disclosure. Figure 5 It is a schematic diagram showing the process of a data processing method according to an embodiment of the present disclosure. Figure 6 It is a schematic diagram showing the process of a data processing method according to an embodiment of the present disclosure. Figure 7 It is a schematic diagram showing the process of a data processing method according to an embodiment of the present disclosure. Figure 8 It is a schematic diagram showing the node state transition of a data processing method according to an embodiment of the present disclosure. Figure 9 It is a schematic diagram showing the workflow state transition of a data processing method according to an embodiment of the present disclosure. Figure 10 It is a block diagram showing a data processing device according to an exemplary embodiment of the present disclosure. Figure 11 It is a block diagram showing an electronic device according to an exemplary embodiment of the present disclosure. Detailed Embodiments

[0009] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0010] In the present disclosure, the term "plural" means two or more, unless otherwise specified. In the present disclosure, the term "and / or" describes the association relationship of associated objects and covers any one of the listed objects and all possible combination manners. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0011] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects and are not intended to limit their positional relationship, timing relationship or importance relationship. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in a manner other than those illustrated or described herein.

[0012] In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these process, method, system, product or device.

[0013] Figure 1 is a flowchart showing a data processing method 100 according to an embodiment of the present disclosure. The data processing method 100 can be executed by a data processing device. Note that although Figure 1 Steps 102, 104, 106, and 108 in are depicted as independent blocks and seemingly separate steps, but these steps should not be understood as necessarily being executed in the order shown. The method 100 may include steps 102, 104, 106, and 108.

[0014] Step 102, the data processing device executes a plurality of workflows. The workflows include a plurality of nodes, and each node starts to execute node work after obtaining computing resources. A workflow consists of a plurality of nodes and the forward and backward dependency relationships between the nodes. The workflow can be constructed in the form of a directed acyclic graph, where the vertices of the directed acyclic graph represent nodes, and the directed edges represent the forward and backward dependency relationships between the nodes. Each workflow is used to execute a workflow task, and each node in the workflow is respectively used to execute the corresponding node work, and the node work is a subtask in the task.

[0015] In some embodiments, the data processing device executes the plurality of workflows according to initial configuration information, where the initial configuration information may include initial priority information of the plurality of workflows. In some embodiments, the initial configuration information refers to the parameters, settings, and data required by the workflows before starting to execute tasks. These initial configuration information include the starting conditions of tasks, input data, permissions, environment settings, etc. Their role is to ensure that the workflows can operate as expected during execution and correctly process the corresponding data and tasks. Additionally, the initial configuration information may also include road segmentation parameters, analysis path parameters, output result saving parameters, etc. The road segmentation parameters may include length parameters for segmenting roads, and this parameter will affect the segmentation accuracy of roads. The analysis path parameters include specifying the specific task types for road network analysis, such as path planning, shortest path search, traffic flow simulation, etc. The output result saving parameters include specifying the location where the output results will be saved, such as being saved in the local file system or the cloud system.

[0016] Among them, the multiple nodes in the workflow at least include adjacent first and second nodes, and the first node is the upstream node of the second node. The first node work is executed by the first node to generate a first execution result. Then, after the first execution result generated by the first node, the data processing device releases the first computing resource of the first node and allocates a second computing resource to the second node to execute the second node work by the second node. In addition, the workflow can correspond to batch tasks, that is, batch tasks can be completed by executing the workflow. For example, multiple workflows are respectively used to generate maps of different roads in an area, so as to finally obtain the road map of the area.

[0017] In some embodiments, the multiple nodes in the workflow include an aggregation node and multiple inflow nodes of the aggregation node. If multiple nodes in the workflow flow to a certain node simultaneously, the node being flowed into is the aggregation node, and the multiple nodes flowing into the aggregation node are the inflow nodes. When allocating computing resources to the aggregation node, it is necessary to determine whether the multiple inflow nodes of the aggregation node in the workflow meet specific conditions. In some embodiments, the specific conditions include that all of the multiple inflow nodes have generated execution results, or the proportion of the nodes that have generated execution results among the multiple inflow nodes exceeds or is equal to a predetermined threshold. In some embodiments, the node proportion may be greater than or equal to 50% or 80%, and of course, it is not limited to this. After the multiple inflow nodes of the aggregation node in the workflow meet the specific conditions, that is, after the proportion of the nodes that have generated execution results among the multiple inflow nodes exceeds or is equal to the predetermined threshold, computing resources can be allocated to the aggregation node to execute the node work through the aggregation node.

[0018] In some embodiments, multiple nodes in the workflow include a road segmentation node. The road segmentation node is used to generate an extended area of the target road according to buffer parameters, and determine the sections of other roads within the extended area as associated sections of the target road. Therefore, the updated configuration parameters also include the updated buffer parameters. The buffer parameters include, but are not limited to, the minimum segmentation length, the maximum segmentation length, the extension width or extension radius for extending from the target road to obtain the extended area.

[0019] In some embodiments, multiple nodes in the workflow include a point cloud extraction node. The point cloud extraction node is used to extract and display point cloud points located between a preset first height and a second height. The updated configuration parameters also include the updated first height and second height, so that point cloud points located in different height intervals can be displayed in the interface.

[0020] Step 104, during the execution of the multiple workflows by the data processing device, receive an intervention instruction input by the user; in response to the intervention instruction, pause or terminate the execution of the multiple workflows.

[0021] In some embodiments, the data processing device pausing the execution of multiple workflows in response to an intervention instruction can include two states. These two states are, for example, a pause processing state or a pause transfer state. Correspondingly, the intervention instruction can be respectively referred to as a pause processing instruction or a pause transfer instruction.

[0022] In some embodiments, after receiving the intervention instruction, the data processing device causes the workflow to enter the pause processing state. In some embodiments, when the workflow enters the pause processing state, the currently running current node in the workflow stops executing its corresponding task.

[0023] In some embodiments, after receiving the intervention instruction, the data processing device causes the workflow to enter the pause transfer state. Among them, when the workflow enters the pause transfer state, once the currently running current node in the workflow completes its corresponding task, the workflow is suspended. Further, the workflow being suspended means that the workflow will no longer continue to flow to its next node.

[0024] In some embodiments, after receiving the intervention instruction, the data processing device causes the workflow to enter the termination state. Among them, when the workflow enters the termination state, the workflow is not allowed to be restarted.

[0025] In some embodiments, the data processing device determines the workflow status of the workflow based on a pre-set mapping relationship between node status and workflow status, and the actual status of each node in the workflow. Subsequently, a user interface is provided to display the workflow status of the workflow and the actual status of each node.

[0026] In some embodiments, the node status includes at least one of the following: not started, queuing, skipped, running, run completed, run failed. The workflow status includes at least one of the following: not started, running, run completed, run failed, paused for transfer, paused for processing, terminated.

[0027] In some embodiments, when the status of the workflow is represented as not started, for example, the first node in the workflow has not applied for computing resources and the status of the first node is not started. When the status of the workflow is represented as running, for example, any one of the multiple nodes in the workflow is running. At this time, the status of the downstream node of the any one node is queuing, and the downstream node of the any one node is waiting for the allocation of computing resources. When the status of the workflow is represented as run completed, for example, the first node to the last node in the workflow have all completed their corresponding tasks. When the status of the workflow is represented as run failed, for example, one of the multiple nodes in the workflow has run failed, and at this time the workflow status is represented as run failed. When the status of the workflow is represented as terminated, for example, the workflow will no longer be restarted to execute tasks.

[0028] In some embodiments, when the status of the workflow is represented as paused for transfer, for example, after one of the multiple nodes in the workflow has completed its corresponding task, the workflow is suspended, for example, it does not continue to transfer to the next node. Additionally, the workflow can be restarted after the status of the workflow is paused for transfer. The restarted node can be one of the multiple nodes. When the status of the workflow is represented as paused for processing, for example, one of the multiple nodes in the workflow (for example, the first node in the workflow) stops executing its corresponding task. Additionally, the workflow can be restarted after the status of the workflow is paused for processing. For example, one of the multiple nodes (for example, the third node in the workflow), and at this time, the next task execution will start from the third node in the workflow. Among them, the node status of the second node in the workflow after restart is skipped.

[0029] Step 106, the data processing device receives the update configuration information of the multiple workflows input by the user; according to the update configuration information, restart the multiple workflows that have been suspended from execution, where the update configuration information includes the priority information of the multiple workflows.

[0030] In some embodiments, after pausing or terminating the execution of the workflow, the updated configuration information is received through a user interface. The updated configuration information includes: priority information of the multiple workflows, identifiers and configuration parameters of newly added nodes in the workflow, identifiers of nodes deleted from the workflow, configuration parameters of existing nodes after modification when the workflow is paused, matching rules between multiple nodes with different naming methods, and default output values when a node fails to output a result.

[0031] In some embodiments, the identifiers and configuration parameters of newly added nodes in the workflow, for example, adding a new node in the workflow and defining the configuration parameters of the newly added node, such as the required resources or input and output paths of the node. In some embodiments, the identifier of a node deleted from the workflow, for example, when a node needs to be deleted from the workflow, the identifier of the node is used to specify the specific node to be deleted. In some embodiments, the configuration parameters of existing nodes after modification when the workflow is paused, for example, during the pause of the workflow process, a user may need to modify the configuration parameters of an existing node, such as adjusting the output path of the node. In some embodiments, the matching rules between multiple nodes with different naming methods, for example, when there are multiple nodes in the workflow, the matching rules between the multiple nodes define how they are associated and cooperate with each other. Different naming methods may require different matching rules to ensure that they work together as expected. In still some other embodiments, the current workflow may also need to call the processing results of other workflows, or the current workflow is an improved version of an old workflow. Therefore, the matching rules of the nodes also include the matching rules between the nodes running in the current workflow and the nodes running in other workflows, so that the processing results of the required nodes can be called from other workflows. In some embodiments, the default output value when a node fails to output a result, for example, during the execution of multiple nodes in the workflow, sometimes valid output results may not be generated for various reasons. To address this situation, the default output value when a node fails to output a result can be defined to ensure the continuity and consistency of the workflow process.

[0032] In some embodiments, for any workflow in the multiple workflows, the update configuration information also includes a node identifier. The node identifier is, for example, a unique identifier or name for each node in the workflow. According to the node identifier, when the configuration information of the workflow needs to be updated, the node identifier can be used to determine which specific node to update. After a restart node is determined from the multiple nodes of the workflow, after the restart node restarts the workflow that was suspended, the current node that was running when the workflow was suspended, and the intermediate nodes between the restart node and the current node can be determined. Then, the execution results generated by the restart node, the current node, and the intermediate node before the workflow was suspended are cleared, and the workflow is executed based on the update configuration information. In other words, the execution of the workflow based on the update configuration information will no longer use the execution results generated by the restart node, the current node, and the intermediate node before the workflow was suspended.

[0033] Step 108: The data processing device allocates computing resources to each node in the multiple workflows according to the priority information and the time when each node in the multiple workflows applies for computing resources.

[0034] In some implementations, multiple nodes of the same workflow are based on the execution order of the nodes, and the downstream node will apply for computing resources only after the upstream node completes the result output. When multiple nodes from different workflows apply for computing resources, the priority of the workflow and the application time of the computing resources are comprehensively considered to determine which node should be allocated computing resources first. For example, when the time difference between multiple nodes applying for computing resources is less than or equal to a first threshold, the node to which the workflow has a higher priority will be allocated computing resources first. When the time difference between multiple nodes applying for computing resources is greater than the first threshold, the resources are allocated according to the time of application.

[0035] In some implementations, node serial numbers are generated for each node in each workflow in order, and computing resources are allocated to each node in the multiple workflows according to the priorities of the multiple workflows, the time when each node applies for computing resources, and the node serial number of each node that applies for computing resources. The node serial number is the sequence number of the node in the workflow to which it belongs. The time when each node applies for computing resources refers to the time point when each node starts to request computing resources. If two parallel nodes in the same workflow (such as nodes with the same ranking serial number on two branches) both apply for computing resources, the computing resources are allocated according to the application time.

[0036] Figure 2 is a schematic diagram showing the allocation of computing resources to each node according to an embodiment of the present disclosure.

[0037] In some embodiments, each node may need to wait for a period of time until it is allocated sufficient computing resources to start executing tasks. For example, if there are two workflows A and B. The priority of workflow A is, for example, priority 1, while the priority of workflow B is, for example, priority 2. Among them, the priority order of priority 1 is higher than that of priority 2. There are multiple nodes in workflow A, and the time for each node to apply for computing resources is different. In addition, the multiple nodes in workflow A all have the same priority 1 as workflow A. There are multiple nodes in workflow B, and the time for each node to apply for computing resources is different. In addition, the multiple nodes in workflow B all have the same priority 2 as workflow B.

[0038] Workflow A, for example, includes node A1 (priority 1, application time for computing resources is T 11 , node serial number 1), node A2 (priority 1, application time for computing resources is T 12 , node serial number 2), node A3 (priority 1, application time for computing resources is T 13 , node serial number 3), and so on. Finally, it is node An (priority 1, application time for computing resources is T n , node serial number n); Workflow B, for example, includes node B1 (priority 2, application time for computing resources is T 21 , node serial number 1), node B2 (priority 2, application time for computing resources is T 22 , node serial number 2), node B3 (priority 2, application time for computing resources is T 23 , node serial number 3), and so on. Finally, it is node Bm (priority 1, application time for computing resources is T m , node serial number m). Among them, the times of T 11 and T 21 are not limited to being different times. For example, the times of T 11 and T 21 can be the same.

[0039] In some embodiments, node A1, node A2, node B1, and node B2 are used as the nodes applying for computing resources. When the times of T 11 and T 21 can be the same, the allocation method of allocating computing resources to each node in the multiple workflows can be to allocate to node A1 first because the priority of A1 is the highest. When the application time of T 12 for applying for computing resources is later than that of T 21 , at this time, the computing resources will be allocated to node B1 first. Although the priority of A2 is higher than that of B1, since the application time of B1 for computing resources is earlier than that of A2, node B1 will obtain resources prior to node A2. Then it is allocated to node A2, and finally to node B2.

[0040] In some embodiments, when the time difference between multiple nodes applying for computing resources is less than or equal to a first threshold, computing resources are preferentially allocated to the node with a smaller sequence number value and a higher priority. More preferably, computing resources are first allocated to the node with a smaller sequence number, and then to the node with a higher priority. For example, assume that nodes A2, B1, and B2 apply for computing resources within a short period of time. Since the sequence number of node B1 is smaller, computing resources are preferentially allocated to node B1. For the remaining nodes A2 and B2, since the priority of node A2 is higher, computing resources are first allocated to node A2.

[0041] In some other embodiments, after ensuring that computing resources have been allocated to nodes with the same node sequence number in each workflow, computing resources will be allocated to nodes with the next node sequence number. Specifically, only after computing resources have been allocated to nodes with a node sequence number of 1 in all three workflows will computing resources be allocated to nodes with a node sequence number of 2 in these three workflows. At this time, it may occur that although node A2 applies for computing resources earlier than node B1, since the node sequence number value of node B1 is smaller, computing resources are preferentially allocated to node B1.

[0042] Figure 3 FIG. is a schematic diagram of an aggregation node showing a data processing method according to an embodiment of the present disclosure.

[0043] In some embodiments, multiple nodes in a workflow include an aggregation node C3 and multiple inflow nodes C11, C21 of the aggregation node. Additionally, the downstream nodes of the aggregation node C3 can be represented as C4, C5. At this time, it is necessary to determine whether multiple inflow nodes C11, C21 of the aggregation node in the workflow meet specific conditions to allocate computing resources to them. Among them, the specific conditions include that all of the multiple inflow nodes have generated execution results, or the proportion of nodes among the multiple inflow nodes that have generated execution results exceeds or is equal to a predetermined threshold. In some embodiments, the node proportion can be greater than or equal to 50% or 80%, and of course, it is not limited thereto. When multiple inflow nodes of the aggregation node in the workflow meet the specific conditions, that is, when the proportion of nodes among the multiple inflow nodes that have generated execution results exceeds or is equal to the predetermined threshold, computing resources can be allocated to the aggregation node C3 to execute node work through the aggregation node C3.

[0044] Figure 4 FIG. is a schematic diagram showing the process of workflow 400 according to an embodiment of the present disclosure. For example, workflow 400 is used to process tasks for trajectory optimization. In some embodiments, the workflow 400 includes four nodes, namely node 402, node 404, node 406, and node 408.

[0045] Node 402, the data processing device performs the task of single-packet optimization.

[0046] In some embodiments, the data processing device first reads the definition of node 402. For example, how much computing resources are required for this node 402. Then, the data processing device will apply for the computing resources required by node 402. After waiting for sufficient resources, it starts to perform the task of single-packet optimization. After the task of single-packet optimization is completed, the data processing device will parse the next node of node 402 in the trajectory optimization workflow, for example, the data processing device will parse the downstream node of node 402.

[0047] Node 404, the data processing device performs the task of multi-packet optimization.

[0048] In some embodiments, the data processing device first reads the definition of node 404. For example, how much computing resources are required for node 404 to perform the task of multi-packet optimization. At this time, the data processing device will apply for the computing resources required to perform the task of multi-packet optimization. After waiting for sufficient resources, it starts to perform the task of multi-packet optimization. After the task of multi-packet optimization is completed, the data processing device will parse the next node of node 404 in the trajectory optimization workflow, that is, the data processing device will parse the downstream node of node 404.

[0049] It should be noted that the difference between performing the task of multi-packet optimization and performing the task of single-packet optimization is that when performing the task of multi-packet optimization, the data processing device can obtain the output results of the upstream node (such as node 404) for subsequent data processing.

[0050] Node 406, the data processing device performs the task of manual quality inspection.

[0051] In some embodiments, the downstream node of node 404 is node 406. At node 406, the data processing device will perform the task of manual quality inspection. The data processing device first reads the definition of node 406. The definition of node 406 includes, for example, the node type of this node 406 is a service node, and how much computing resources need to be applied for at this node 406. At this time, the data processing device will apply for the computing resources required for manual quality inspection. After waiting for sufficient resources, it will start to perform the task of manual quality inspection. After the task of manual quality inspection is completed, the data processing device will parse the next node of node 406 in the trajectory optimization workflow, that is, the data processing device will parse the downstream node of node 406.

[0052] Node 408, the data processing device performs the task of production materials warehousing.

[0053] In some embodiments, the downstream node of node 406 is node 408. The data processing device first reads the definition of node 408. For example, how much computing resources are required for this node 408. At this time, the data processing device will apply for the computing resources required by node 408. After waiting for sufficient resources, it will start to execute the task of storing production materials. After the task of storing the production materials is completed, the data processing device will parse the next node of node 408 in the trajectory optimization workflow, that is, the data processing device will parse the downstream node of node 408. Since node 408 is the last node of the workflow, after the data processing device finishes executing the task of storing production materials, the state of the workflow will be represented as running completed.

[0054] In some embodiments, if one of the states of multiple nodes in the workflow is running failed, the state of the workflow is represented as running failed. If you want to update configuration information such as the priority in the workflow, the user can input an intervention instruction through the user interface to let the data processing device pause the execution of the workflow. Pausing the execution of the multiple workflows can include two states. These two states are, for example, the pause processing state or the pause transfer state. When the workflow is in the pause processing state, the nodes that are running in the workflow stop executing their corresponding tasks (for example, manual quality inspection); when the workflow enters the pause transfer state, once the nodes that are running in the workflow complete their corresponding node work (for example, manual quality inspection), the workflow is suspended. Further, the suspension of the workflow means that the workflow will no longer continue to transfer to its next node. For example, the suspension of the workflow can be represented as after node 406 completes its corresponding task (for example, manual quality inspection), the workflow will no longer transfer to the downstream node 408 of node 406.

[0055] In some embodiments, after pausing the execution of the workflow, the updated configuration information can be received through the user interface. For example, adjust the priority information of the workflow. After the adjustment is completed, the data processing device restarts the workflow that was paused according to the updated configuration information. The restarted node can be 402, and the data processing device will start to execute the task of single-pack optimization. Among them, the user interface can be the interface in the data processing device or the interface of other computing devices independent of the data processing device.

[0056] Figure 5 It is a schematic diagram showing the process of a data processing method 500 according to an embodiment of the present disclosure. Note that although Figure 5Steps 502, 504, 506, 508, 510, 512, 514, 516, 518, 520, and 522 in

[0057] In step 502, the user selects the data information to be processed.

[0058] In some embodiments, the data processing device provides the user with the option to select the data information to be processed. If the user selects "raw data", it may mean that the user chooses to process the entire raw data. In this case, after the raw data is segmented, subsequent operations are performed according to the segmented means. If "segmented data" is selected, it may mean that the user chooses to use the data that has been segmented from the raw data. The raw data may include data obtained in one or more data collection tasks. The segmented data can be represented as a set of multiple data sets obtained by segmenting the data obtained in one or more data collection tasks.

[0059] In step 504, the user selects information such as the workflow, configures the priority, and sets parameters.

[0060] In some embodiments, the data processing device allows the user to select, through a user interface, the workflow that the data to be processed needs to run, configure the priority of the workflow, and select the configuration information of the nodes in the workflow. The configuration information may include, for example, the identifier and configuration parameters of newly added nodes in the workflow, the identifiers of the nodes deleted from the workflow, the configuration parameters after the original nodes are modified after the workflow is paused, the matching rules between multiple nodes with different naming methods, the default output value when a node cannot output a result, and the node identifier.

[0061] In step 506, the data processing device creates batch information.

[0062] In some embodiments, the data processing device allows the user to click to create batch information through a user interface. After the user clicks to create batch information, the data processing method 500 proceeds to the next step 508.

[0063] In step 508, the data processing device starts data segmentation.

[0064] In some embodiments, the data processing device segments the data information to be processed according to the configuration of the priority of the workflow in step 504 and the configuration information of the nodes selected in the workflow. In step 510, the data processing device executes tasks.

[0065] In some embodiments, after the data processing device splits the data information to be processed, multiple workflows can be obtained, and the data processing device can execute corresponding multiple tasks according to the multiple workflows.

[0066] Step 512, the data processing device runs a task.

[0067] In some embodiments, any one of the multiple workflows has multiple nodes, and the workflow and the nodes each have configuration information. At this time, the data processing device starts to execute the corresponding task of the workflow or the corresponding work of the node according to the configuration information of the workflow and the nodes respectively. When the data processing device runs the workflow, the data processing device can allow the user to input an intervention instruction through the user interface. For example, at this step, the execution of the workflow can be paused or terminated through the intervention instruction. Specifically, the intervention instruction can include a pause instruction or a termination instruction, which are used to pause or terminate the workflow respectively. The pause instruction can further include a pause processing instruction or a pause transfer instruction, which are used to immediately pause the workflow or pause the workflow after the current node finishes the node work respectively.

[0068] Step 514, the data processing device determines whether the workflow is completed. If the states of all the nodes in the multiple workflows are all completed, it means that the corresponding multiple tasks of the multiple workflows have been executed, and then step 518 is entered. If there are still nodes in the states of not started or failed to run, it means that there are still workflows in the running or failed state, and then step 516 is entered. Step 514 will loop until all workflows are completed.

[0069] Step 516, the user makes parameter adjustments.

[0070] In some embodiments, after the data processing device pauses or terminates the execution of the workflow, it can receive updated configuration information through the user interface. The updated configuration information includes: the priority information of the multiple workflows, the identifiers and configuration parameters of the newly added nodes in the workflow, the identifiers of the nodes deleted from the workflow, the configuration parameters after the original nodes are modified after the workflow is paused, the matching rules between multiple nodes with different naming methods, and the default output value when the node cannot output a result. Then, the data processing device will restart the paused workflow according to the updated configuration information and enter step 512.

[0071] Step 518, the data processing device records the running information.

[0072] In some embodiments, the data processing device records the running information of the workflow. For example, the data processing device records the configuration information of the workflow and whether there is updated configuration information.

[0073] Step 520, the data processing device notifies the user.

[0074] In this step, the data processing device notifies the user of the progress or status of the workflow through the user interface.

[0075] Step 522, the task is completed.

[0076] In this step, for example, the task to be completed by the workflow has been executed.

[0077] In some embodiments, the workflow of the present disclosure includes a map construction workflow. Important nodes in the map construction workflow include a road segmentation node. After obtaining computing resources, the road segmentation node can also identify inflection points on the road, and then distinguish straight sections and turning sections on the road for map construction respectively. Specifically, according to the azimuth information of multiple trajectory points relative to a reference direction, one or more inflection points are identified from the multiple trajectory points; according to the one or more inflection points, one or more inflection point ranges are established; and according to the spatial position information of the multiple trajectory points and the one or more inflection point ranges, a first set of trajectory points and a second set of trajectory points are extracted from the multiple trajectory points. Among them, the spatial positions of the first set of trajectory points are within the one or more inflection point ranges, and the spatial positions of the second set of trajectory points are outside the one or more inflection point ranges. The first set of trajectory points can be considered as turning section points, and the second set of trajectory points can be considered as straight section points. The inflection point range takes the inflection point as a reference point and determines the geometric range according to at least one length parameter as the inflection point range corresponding to the inflection point. At this time, the initial configuration parameters and new configuration parameters of the node can include the relevant parameters of the inflection point range (such as the length parameter), that is, during the map construction process of the workflow, the relevant parameters of the inflection point range are modified, thereby changing the size of the inflection point range.

[0078] Furthermore, the road segmentation node establishes an analysis interval within the multiple trajectory points, and the analysis interval covers at least a part of the multiple trajectory points; and in response to the azimuth difference between two specific trajectory points within the analysis interval (the azimuth of a trajectory point is the angle between the line connecting the trajectory point and the adjacent trajectory point and the reference direction, and the reference direction is, for example, due north) being greater than or equal to the first threshold, the analysis interval is determined as a potential inflection point interval where an inflection point may exist. Then, the multiple trajectory points within the potential inflection point interval are gradually checked, and the azimuth of each checked trajectory point is recorded; and during the process of the gradual check, when the azimuth of a trajectory point is such that the difference between the maximum azimuth and the minimum azimuth within the potential inflection point interval is greater than or equal to the second threshold, the trajectory point is determined as an inflection point. Optionally, the road segmentation node can sequentially slide the analysis interval along the time sequence direction of the multiple trajectory points to respectively determine whether there is a potential inflection point interval within each analysis interval. Each of the multiple trajectory points has a coordinate value and a timestamp, and the two specific trajectory points may include the trajectory point with the earliest time sequence and the trajectory point with the latest time sequence within the analysis interval. The analysis interval has a fixed geographical length, such as a fixed 300 meters, although it is not limited to this.

[0079] In addition, the road segmentation node can also remove the inflection points of U-turn roads from the identified inflection points, and determine whether the azimuth difference between two trajectory points at specific distance positions before and after each inflection point is greater than or equal to a specific angular value. If so, the inflection point is filtered out as a special inflection point. Then, for each remaining inflection point after filtering out all special inflection points, a corresponding inflection point range is established. Moreover, if the inflection point range of the current inflection point overlaps with the inflection point ranges of other inflection points, the inflection point range of the current inflection point and the inflection point ranges of other inflection points are merged. Therefore, the road segmentation parameters according to the embodiments of the present disclosure may further include at least one of the following: the length of the analysis interval during inflection point identification, the magnitudes of the first threshold and the second threshold, and the size of the inflection point range.

[0080] In still other embodiments, during the process of generating the initial configuration information, the data processing device can also select a target trajectory and a reference trajectory. The target trajectory includes multiple target trajectory points, and the reference trajectory includes multiple reference trajectory points; for example, the target trajectory and the reference trajectory can be selected according to the collected start and end point coordinates or start and end point timestamps (the start and end points include the starting point and the ending point). Then, after starting the workflow, when the road segmentation node obtains computing resources, it first identifies which trajectory points of the target trajectory are the associated trajectory points of the reference trajectory and which trajectory points are the different trajectory points of the reference trajectory. The associated trajectory points can be considered as similar trajectory points. If a map has been constructed based on the reference trajectory, the associated trajectory points of the reference trajectory can be ignored, and the map is constructed only based on the different trajectory points of the reference trajectory.

[0081] Specifically, for any reference trajectory point in the reference trajectory: select target trajectory points from multiple target trajectory points that are located within the target area where the reference trajectory point is located; determine whether there is an approximate relationship between the selected target trajectory points and the reference trajectory point according to the spatial position relationship between the selected target trajectory points and the reference trajectory point; and in response to the selected target trajectory point having an approximate relationship with the reference trajectory point, determine the selected target trajectory point as an associated trajectory point of the reference trajectory. Among them, the spatial position relationship includes at least one of a distance relationship and an azimuth difference. The azimuth of each target trajectory point includes the angle between the connection direction between the target trajectory point and its adjacent target trajectory point and the reference direction; the azimuth of each reference trajectory point includes the angle between the connection direction between the reference trajectory point and its adjacent reference trajectory point and the reference direction. Here, the geographical space can be divided into multiple grids according to geographical coding methods such as geohash, and the trajectory points can exist in different grids. Selecting target trajectory points from multiple target trajectory points that are located within the target area where the reference trajectory point is located may include: selecting target trajectory points from the multiple target trajectory points that are located within the target area and its adjacent areas and have not been determined as associated trajectory points.

[0082] In some embodiments, in response to the distance between the selected target trajectory point and the reference trajectory point being less than or equal to a third threshold and the azimuth offset being less than or equal to a fourth threshold, it is determined that there is an approximate relationship between the selected target trajectory point and the reference trajectory point. In response to the distance between the selected target trajectory point and the reference trajectory point being greater than the third threshold or the azimuth offset being greater than the fourth threshold, it is determined that there is no approximate relationship between the selected target trajectory point and the reference trajectory point. The first to fourth thresholds can be set as needed, and the present disclosure does not limit this. Then the initial or updated road segmentation parameters may further include the magnitudes of the third and fourth thresholds to adjust the difference degree between the target trajectory and the reference trajectory.

[0083] Figure 6 is a schematic diagram showing the flow of a data processing method 600 according to an embodiment of the present disclosure. Note that although Figure 6 the steps 602, 604, 606, 608, 610, 612, 614, and 616 in are depicted as independent blocks and seemingly separate steps, these steps should not be understood as necessarily being executed in the order shown. The method 600 may include steps 602, 604, 606, 608, 610, 612, 614, and 616.

[0084] Step 602, the data processing device monitors tasks.

[0085] For example, after the data processing device receives a request that a task needs to be executed, the data processing device can perform corresponding operations. Further, one or more tasks corresponding to the one or more workflows need to be executed through the data processing device.

[0086] In some embodiments, the data processing device can split all the raw data in a data collection task. A data collection task can obtain data of multiple roads, including but not limited to road data and environmental data. The road data includes the coordinates and collection timestamps of the collection points, and the environmental data includes data such as images and point clouds collected at each collection point. Through road splitting, a road can be split into multiple road segments, so that maps can be generated respectively according to the data of each road segment. The splitting method can be carried out according to road splitting parameters, analysis path parameters, and output result storage parameters. The raw data after splitting can form multiple split data, and the multiple split data can be used for multiple workflows, and the multiple workflows can respectively correspond to multiple tasks. In some embodiments, the data processing device can determine that the obtained multiple split data needs to perform tasks such as data processing, map recognition, trajectory optimization, or manual editing to process the data.

[0087] Step 604, the data processing device parses the workflow.

[0088] For example, the data processing device can first parse the definition of the workflow. For example, the definition of the workflow includes a workflow flowchart, inputs, and outputs. In some embodiments, the workflow flowchart presents the sequence and interrelationships between the various nodes in the workflow or the required computing resources. In some embodiments, the inputs present the data or information required for each node in the workflow. These input data can be files, database records, upstream node data, or output results of other platforms, etc. In some embodiments, the outputs present the results generated after each node in the workflow executes the task. For example, the road splitting node is used to split road data packets, the point cloud generation node is used to extract the point cloud from the split data packets, the point cloud post-processing node is used to perform point cloud registration on the extracted multiple frames of point clouds, the post-processing file push node is used to perform format conversion on the post-processed files and push them to the cloud, the speed limit information extraction node is used to extract and display the speed limit information of each lane, the lane line extraction node includes extracting the lane line information in the data packet, such as solid lane lines and dashed lane lines, and the multi-pack map optimization node includes jointly optimizing the lane line information of multiple data packets to obtain jointly optimized lane line information for a section. The output of the upstream node can be used as the input of the downstream node. For example, the output of the lane line extraction node can be used as the input of the multi-pack map optimization node.

[0089] Step 606, the data processing device determines the downstream nodes.

[0090] In this step, the data processing device determines the target downstream node based on the dependency relationships between the nodes in the workflow flowchart obtained after parsing the workflow, and according to the output data of the current node and the input requirements of the downstream nodes.

[0091] Step 608, the data processing device determines the node status and configuration information.

[0092] For example, the data processing device determines the node status and the configuration information of the workflow to initialize the node information. Initializing the node information mainly includes node runtime information (the same node configuration can be different during different task executions), such as initial parameters of the node algorithm, node resource configuration (CPU, etc.), unique identification information of the node, and node runtime status record table (current status, start time of operation, task creator, processed data information). Then, the node starts to execute by obtaining the computing resources of the node. After starting to execute the node, the node status includes at least one of the following, such as not started, queuing, skipped, running, run completed, run failed. In some embodiments, when the current node status is run completed, it will re-enter step 606 to continue to determine the downstream node of the current node.

[0093] Step 610, the data processing device maps the workflow status.

[0094] For example, the data processing device has stored the mapping relationship between the node status and the workflow status. The data processing device can determine the status of the workflow according to the status of the current node. The workflow status includes at least one of the following, such as not started, running, run completed, run failed, paused for transfer, paused for processing, terminated. In some embodiments, the mapping relationship includes that when the node status is run failed, the workflow status will also be represented as run failed.

[0095] Step 612, the data processing device determines the final state of the workflow.

[0096] For example, the data processing device determines the workflow status of the workflow according to the pre-set mapping relationship between the node status and the workflow status and the actual status of each node in the workflow. Then, a user interface is provided to display the workflow status of the workflow and the actual status of each node. In some embodiments, the workflow status, such as run completed, terminated, can represent the final state of the workflow.

[0097] Step 614, the data processing device determines the end of the workflow task.

[0098] For example, the data processing device determines whether the tasks corresponding to the workflow have been completed or are no longer to be executed based on the workflow status result confirmed in step 612. When in the final state of the workflow, it indicates that the tasks corresponding to the workflow have been completed or the tasks corresponding to the workflow are no longer to be executed.

[0099] Step 616, the data processing device determines the intermediate state of the workflow.

[0100] For example, the data processing device determines the workflow status of the workflow according to the pre-set mapping relationship between the node status and the workflow status and the actual status of each node in the workflow. Then, a user interface is provided to display the workflow status of the workflow and the actual status of each node. In some embodiments, when the workflow status is represented as not started, in progress, failed to run, paused from flowing, paused from processing, it represents the intermediate state of the workflow. In this step, the configuration information of the workflow can also be further updated by the data processing device. At this time, the workflow after updating the configuration information will re-enter step 604 to re-parse the workflow.

[0101] Figure 7 is a schematic diagram showing the flow of the data processing method 700 according to an embodiment of the present disclosure. Note that although Figure 7 Steps 702, 704, 706, 708, 710, 712, and 714 in are depicted as independent blocks and seemingly separate steps, but these steps should not be construed as having to be executed in the order shown. Method 700 may include steps 702, 704, 706, 708, 710, 712, and 714.

[0102] Step 702, the data processing device determines the node status and configuration information.

[0103] For example, the data processing device determines the node status and the configuration information of the workflow to initialize the node information. Among them, the node status includes at least one of the following, for example, not started, queuing, skipped, in progress, completed, failed. In some embodiments, when the data processing device determines that the node status is failed to run, the configuration information of the workflow can be readjusted. At this time, the data processing device updates the node status and configuration information according to the new workflow configuration information to initialize the updated node execution information.

[0104] Step 704, the data processing device processes the priority arrangement of the workflow.

[0105] In this step, the data processing device can arrange the priorities of multiple nodes in the workflow according to the priority status of the workflow.

[0106] Step 706, the data processing device determines the node type.

[0107] In this step, the data processing device determines the node type, etc. For example, the node types include service nodes, docker nodes, and judgment nodes. Among them, the service node is a resident task, and the workflow system interacts with it through the restful url method. The docker node is a container node, and the workflow maintains all configuration information of the docker node, including code and runtime environment dependencies (such as a certain algorithm library), enabling the workflow system to start a virtual environment in the cluster for operation. The judgment node belongs to the internal functional node of the workflow system and is used to judge the flow direction of the workflow branch.

[0108] Step 708, the data processing device allocates computing resources.

[0109] In this step, the data processing device will consider the priority of the node and the time when each node applies for computing resources. Among them, the priority of the node is the same as the priority of the workflow to which the node belongs. The data processing device allocates resources to each node based on the nodes with high priority and the time when each node applies for computing resources.

[0110] Step 710, the data processing device triggers the node program.

[0111] In this step, once the resource allocation for the node is completed, the data processing device will start to execute the operation of the node. For each node, the data processing device will start the corresponding processing program. At the same time, the data processing device will also continuously monitor the running status of the node to ensure the normal execution of the workflow.

[0112] Step 712, the data processing device feeds back the node status.

[0113] In this step, the data processing device will judge the status of the currently running node according to the running condition of the node. The actual status of each node is displayed through the user interface. The node status includes at least one of the following, such as not started, queuing, skipped, running, running completed, running failed.

[0114] Step 714, the data processing device judges the node status transition.

[0115] In this step, the state transition of the node is determined according to the process of the data processing device from the start of executing the corresponding task of the node until the task is completed. The state transition of the node, for example, starts from the not-yet-started state, and then sequentially enters the queued, running, and completed states. Or after the workflow is paused, the node may enter the runtime failure state. After the user inputs updated configuration information through the user interface, if the restarted node is not the current node when the previous workflow was paused, the node may enter the skipped state.

[0116] Figure 8 It is a schematic diagram showing the node state transition of the data processing method according to an embodiment of the present disclosure.

[0117] In some embodiments, taking Figure 2 as an example. In workflow A, node A3 is the target observation node in the process of node state transition. When node A1 in workflow A is running, the node state of node A2 is queued and waiting for the allocation of computing resources. As for the node state of node A3, it is not yet started 802. Not yet started means the node has not applied for computing resources. At this time, when node A2 obtains computing resources, node A2 enters the running state. As for the state of node A1, it is represented as completed. The state of node A3 transitions to queued 804 and waits for the allocation of computing resources. Then, when node A2 finishes running, the state of node A2 is represented as completed. At this time, node A3 obtains computing resources, so that the state of node A3 is represented as running 806. If node A3 runs successfully, the state of node A3 is represented as completed 810. Next, the target node in workflow A will transition to the downstream node of node A3 and start the process of state transition of the downstream node.

[0118] In some embodiments, taking Figure 2 as an example. When the state of node A3 is queued 804 or the state of node A3 is running 806, the data processing device can pause the execution of workflow A according to the intervention instruction input by the user through the user interface. In some embodiments, when the state of workflow A is represented as paused transition, it means that after node A3 in workflow A completes its corresponding task, workflow A is suspended and does not continue to transition to the next node. At this time, the state of node A3 will transition to runtime failure 808. When the state of workflow A is represented as paused processing, it means that node A3 in workflow A stops executing its corresponding task. At this time, the state of node A3 will transition to runtime failure 808.

[0119] In some embodiments, taking Figure 2For example, when the node status is a running failure, the user can further input updated configuration information through the user interface, and the data processing device can restart the paused workflow A according to the updated configuration information. The restarted node can be node A3. At this time, the next task execution will start from node A3, and the status of the restarted node A3 is represented as queuing 804. In some embodiments, the workflow A can be restarted (for example, the task corresponding to the workflow A is restarted). The restarted node can be node A1. For example, the first node of the workflow A. At this time, the restarted task will start the next task execution from node A1, and the status of the restarted node A1 is represented as not started 802. Specifically, first pause the task processing, that is, pause the node work of the current node currently being executed. Then, the parameters can be reconfigured, and the node to be restarted can be selected (only the current node or the upstream node of the current node can be selected). Restart the workflow from the selected node, and the previous task data will be cleared when restarting.

[0120] After the node starts to apply for computing resources, it enters the queuing state. After the node is allocated computing resources and starts to execute the node work, it enters the running state.

[0121] In some embodiments, take Figure 2 as an example. When the status of node A2 is queuing 804 or the status of node A1 is running 806, the data processing device can pause the execution of workflow A according to the intervention instruction input by the user through the user interface. At this time, the user inputs updated configuration information through the user interface. The data processing device can restart the paused workflow A according to the updated configuration information. The restarted node can be node A3. At this time, the next task execution will start from node A3. Further, at this time, the status of node A2 is represented as skipped. The status of the above nodes can be notified to the user through the user interface.

[0122] In some embodiments, the data processing device can further notify the user of the workflow status of the workflow through the user interface.

[0123] Figure 9 is a schematic diagram showing the workflow status transition of the data processing method according to an embodiment of the present disclosure.

[0124] In some embodiments, the workflow status includes at least one of the following, for example, not started, running, running completed, running failed, paused for transfer, paused for processing, terminated.

[0125] In some embodiments, take Figure 2For example, after the data processing device creates Workflow A, Workflow A mainly performs some data preparation. The data preparation work includes: obtaining the execution sequence of each node in the workflow; obtaining configuration parameters such as the execution parameters required when each node inputs (such as the segmentation parameters mentioned above). For example, judging the input and output of Workflow A. At this time, the status of Workflow A is represented as not started 902. After the data preparation of Workflow A is completed, the status of Workflow A transitions to running 904. The running Workflow A includes parsing the flowchart of Workflow A. The flowchart of Workflow A presents the sequence and mutual relationship between each node in the workflow or the required computing resources. Then, the data processing device will allocate and run the computing resources of each node in the flowchart of Workflow A according to the flowchart of Workflow A. At this time, when one of the multiple nodes in Workflow A (for example, Node A2) fails to run, the status of Workflow A is represented as running failed 914, and the user can input an intervention instruction through the user interface to terminate the execution of Workflow A. When the status of Workflow A is represented as terminated 916, for example, Workflow A will no longer be restarted to execute tasks. In some embodiments, after the status of Workflow A is running failed 914, Workflow A can be restarted, and the status of Workflow A transitions to running 904. The restarted node can be one of the multiple nodes in Workflow A.

[0126] In some embodiments, take Figure 2 as an example. When the status of Workflow A flows to running 904, the user can input an intervention instruction through the user interface to pause the execution of Workflow A. Among them, pausing the execution of Workflow A can be that the status of Workflow A transitions to paused transition 910 or paused processing 906. When the status of Workflow A is represented as paused transition 910, it can be that one of the multiple nodes in Workflow A (for example, Node A2) suspends Workflow A after completing its corresponding task, for example, does not continue to transition to the next node (for example, Node A3). In some embodiments, when one of the multiple nodes in Workflow A (for example, Node A2) is running, the status of Workflow A will first transition from running 904 to in paused transition 908, and wait for Node A2 to complete its corresponding task before the status of Workflow A transitions to paused transition 910. In addition, after the status of Workflow A is paused transition 910, Workflow A can be restarted, and the status of Workflow A transitions to running 904. The restarted node can be one of the multiple nodes in Workflow A. In some embodiments, when the status of Workflow A is paused transition 910 or in paused transition 908, the user can further input an intervention instruction through the user interface to terminate the execution of Workflow A, and the status of the workflow will transition to terminated 916. For example, Workflow A will no longer be restarted to execute tasks.

[0127] In some embodiments, take Figure 2 as an example. When the status of Workflow A is represented as Pause Processing 906, it means that one of the multiple nodes in Workflow A (for example, Node A1) stops executing its corresponding task. Additionally, after the status of the workflow is Pause Processing, the workflow can be restarted, and the status of Workflow A can transition to Running 904. The restarted node can be one of the multiple nodes in Workflow A. In some embodiments, after the status of Workflow A is Pause Processing 906, the user can further input an intervention instruction through the user interface to terminate the execution of Workflow A, and the status of the workflow will transition to Terminated 916. For example, Workflow A will no longer be restarted to execute tasks.

[0128] In some embodiments, take Figure 2 as an example. When the status flow of Workflow A is Running 904, the user can input an intervention instruction through the user interface to terminate the execution of Workflow A. When the status of Workflow A is represented as Terminated 916, for example, Workflow A will no longer be restarted to execute tasks.

[0129] Figure 10 is a block diagram showing a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0130] As Figure 10 shown, the data processing apparatus 1000 may include an execution module 1002, a user operation module 1004, and a configuration information module 1006. The user operation module 1004 is used to provide a user interface for the user to input an intervention instruction or adjust the configuration information of the workflow. The configuration information module 1006 is used to store the configuration information of the workflow. The execution module 1002 is used to execute multiple workflows according to the configuration information of the workflow, and in response to the intervention instruction, pause or terminate the execution of the multiple workflows.

[0131] Regarding the specific functions and effects achieved by the data processing apparatus, reference can be made to other embodiments of the present disclosure for comparison and explanation, which will not be elaborated here. Each module in the data processing apparatus can be implemented in whole or in part by software, hardware, and their combinations. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0132] The embodiments of the present disclosure also provide a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by the computer, the computer executes the data processing method in any of the above embodiments.

[0133] Figure 11 FIG. Figure 11 is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment of the present disclosure. The electronic device 1100 mainly includes a processor 1102 and a memory 1104. The memory 1104 stores at least one computer program instruction, and when the processor 1102 executes the at least one computer program instruction, the methods in any one or more of the embodiments described in the present disclosure can be implemented. In some embodiments, the electronic device 1100 can also be used to implement the data processing device mentioned in the present disclosure.

[0134] The processor 1102 includes, for example, a system-on-chip (SoC), a general-purpose processing core, a graphics core, and / or other optional processing logics. The processor 1102 can interact or communicate with other components (such as the memory 1104) through a bus 1106.

[0135] The memory 1104 can represent a machine-readable medium (or a computer-readable storage medium) on which one or more instruction sets, software, firmware, or other processing logics for implementing any one or more of the methods or functions described and / or claimed herein are stored. Although the machine-readable medium (or computer-readable storage medium) of the exemplary embodiment can be a single medium, the term "machine-readable medium" (or computer-readable storage medium) should be understood to include a single non-transitory medium or multiple non-transitory media (such as a centralized or distributed database and / or associated caches and computing systems) storing one or more instruction sets. The term "machine-readable medium" (or computer-readable storage medium) can also be understood to include any non-transitory medium capable of storing, encoding, or carrying an instruction set for a machine to execute and cause the machine to execute any one or more of the methods of various embodiments or capable of storing, encoding, or carrying a data structure utilized by or associated with such an instruction set. The term "machine-readable medium" (or computer-readable storage medium) can thus be understood to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0136] In some embodiments, the electronic device 1100 may further include various input / output (I / O) devices and / or interfaces 1110. These devices or interfaces can support wired and / or wireless data transmission, thereby enabling information or data interaction between the electronic device 1100 and other devices.

[0137] The various embodiments described in this disclosure can also be implemented as one or more computer program products, that is, as one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a substance composition affecting a machine-readable propagated signal, or a combination of one or more of them.

[0138] The term "data processing apparatus" in this disclosure encompasses all apparatuses, devices, and machines for processing data, such as including programmable processors, computers, or multiple processors or computers. In addition to the hardware, the data processing apparatus can also include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or a combination of one or more of them.

[0139] Some embodiments mentioned in this disclosure can be implemented as devices or modules using hardware circuits, software, or a combination thereof. For example, a hardware circuit implementation can include discrete analog and / or digital components, which can be integrated, for example, as part of a printed circuit board. Alternatively or additionally, the disclosed components or modules can be implemented as application-specific integrated circuits (ASICs) and / or field-programmable gate array (FPGA) devices. Additionally or alternatively, some embodiments can include digital signal processors (DSPs). Similarly, the various components or sub-components within each module can be implemented in software, hardware, or firmware. Any connection method and medium known in the art can be used to provide the connection between modules and / or components within the module.

[0140] Although this document includes many details, these details should not be construed as limiting the scope of the claimed invention, but rather as descriptions of features specific to particular embodiments. Certain features described herein in the context of different embodiments can also be combined in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be separated or implemented in multiple embodiments in any suitable sub-combination. Additionally, although features may be described as acting in certain combinations and even initially claimed as such, one or more features from the claimed combination can in some cases be removed from the combination, and the claimed combination can be directed to a sub-combination or a variation of the sub-combination. Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result.

[0141] Only a few embodiments and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Execute multiple workflows, each of which includes multiple nodes. After the node applies for computing resources, it starts to execute node work; During the execution of the plurality of workflows, receiving an intervention instruction input by a user; In response to the intervention instruction, pausing or terminating execution of the plurality of work flows; receiving updated configuration information of the plurality of workflows input by a user; Restarting the plurality of workflows that were suspended according to the updated configuration information, wherein the updated configuration information includes priority information of the plurality of workflows; and The computing resources are allocated to each node in the multiple workflows according to the priority information and the time when each node in the multiple workflows applies for computing resources.

2. The data processing method according to claim 1, characterized in that: For any workflow among the multiple workflows, the update configuration information includes a node identifier, and the data processing method further includes: Determine a restart node from the nodes of the workflow according to the node identifier; Restarting the suspended workflow at the restart node; and After restarting the workflow, the workflow is executed based on the updated configuration information.

3. The data processing method according to claim 2, characterized in that: Restarting the suspended workflow at the restart node includes: Determine the current node that is running when the workflow is suspended, and the intermediate nodes between the restart node and the current node; and The execution results generated by the restart node, the current node, and the intermediate node before the workflow is suspended are cleared.

4. The data processing method according to claim 1, characterized in that: The data processing method further comprises: in response to the intervention instruction, causing the workflow to enter a suspended processing state, When the workflow enters the suspended processing state, the currently running node in the workflow stops executing its corresponding task.

5. The data processing method according to claim 1, characterized in that: The data processing method further comprises: in response to the intervention instruction, causing the workflow to enter a suspended flow state, Wherein, when the workflow enters the suspended flow state, once the current node running in the workflow completes the task corresponding to the current node, the workflow is suspended.

6. The data processing method according to claim 1, characterized in that: The data processing method further comprises: in response to the intervention instruction, causing the workflow to enter a termination state, When the workflow enters the termination state, the workflow is not allowed to be restarted.

7. The data processing method according to claim 1, characterized in that: Allocating computing resources to each node in the multiple workflows according to the priority information and the time when each node in the multiple workflows applies for computing resources includes: Generate node serial numbers for each node in each workflow in sequence; and The computing resources are allocated to each node in the multiple workflows according to the priorities of the multiple workflows, the time when each node applies for the computing resources, and the node sequence number of each node applying for the computing resources.

8. The data processing method according to claim 1, characterized in that: The plurality of nodes of the workflow include a first node and a second node that are adjacent to each other, and the first node is an upstream node of the second node, and the data processing method further includes: Executing the first node work through the first node to generate a first execution result; In response to the first node generating the first execution result, the first computing resource of the first node is released, and the second computing resource is allocated to the second node so as to execute the second node work through the second node.

9. The data processing method according to claim 1, characterized in that: The multiple nodes of the workflow include an aggregation node and multiple inflow nodes of the aggregation node, and the data processing method further includes: Determining whether multiple inflow nodes of the aggregation node in the workflow meet specific conditions; In response to the plurality of inflow nodes satisfying the specific condition, computing resources are allocated to the aggregation node so as to execute node work through the aggregation node.

10. The data processing method according to claim 1, characterized in that: The specific conditions include: All of the multiple inflow nodes have generated execution results; or The ratio of the multiple inflow nodes generating execution results exceeds or equals a predetermined threshold.

11. The data processing method according to claim 1, characterized in that: Executing the multiple workflows includes: The multiple workflows are executed according to initial configuration information, where the initial configuration information includes initial priority information of the multiple workflows.

12. The data processing method according to claim 1, characterized in that: Receiving the update configuration information of the plurality of workflows input by the user comprises: After pausing or terminating the execution of the workflow, the updated configuration information is received through a user interface.

13. The data processing method according to claim 1, characterized in that: The data processing method further includes: Determining the workflow state of the workflow according to a preset mapping relationship between node states and workflow states and the actual state of each node in the workflow; A user interface is provided for displaying the workflow status of the workflow and the actual status of each node.

14. The data processing method according to claim 13, characterized in that: The node status includes at least one of the following: not started, queued, skipped, running, running completed, and running failed; as well as The workflow status includes at least one of the following: not started, running, completed, failed, suspended, suspended, and terminated.

15. The data processing method according to claim 1, characterized in that: The update configuration information includes at least one of the following: Identification and configuration parameters of the newly added node in the workflow; an identification of a node to be deleted from the workflow; Configuration parameters after the original node is modified after the workflow is paused; Matching rules between multiple nodes with different naming methods; and The default output value when the node cannot output a result.

16. The data processing method according to claim 1, characterized in that: The multiple nodes include a road segmentation node, which is used to generate an extended area of ​​the target road according to the buffer parameters, and determine the sections of other roads located in the extended area as associated sections of the target road. The updated configuration parameters also include the updated buffer parameters.

17. The data processing method according to claim 1, characterized in that: The multiple nodes include a point cloud extraction node, which is used to extract and display point cloud points located between a preset first height and a second height according to the first height and the second height, and the updated configuration parameters also include the updated first height and the second height.

18. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by the processor to implement the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program instruction, and when the at least one computer program instruction is executed by a processor, it can implement the data processing method according to any one of claims 1 to 17.

Images