A data processing method, apparatus and device, and a storage medium

Abstract

The application discloses a data processing method, device and equipment and a storage medium. The method comprises the following steps: obtaining target map data; determining a plurality of to-be-executed operators based on the target map data; performing directed acyclic graph construction on the plurality of to-be-executed operators based on the operator dependency relationship of the plurality of to-be-executed operators, to obtain a target directed acyclic graph corresponding to the plurality of to-be-executed operators; each to-be-executed operator corresponds to a node in the target directed acyclic graph; determining the execution sequence of each to-be-executed operator based on the in-degree information of each node in the target directed acyclic graph; and processing the target map data based on the execution sequence of each to-be-executed operator, to obtain a target processing result. The application completes the construction of the target directed acyclic graph based on the plurality of to-be-executed operators, and determines the execution sequence of each to-be-executed operator corresponding to each node in the directed acyclic graph, thereby improving the efficiency of data processing.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving of vehicles, specifically to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] High-precision maps, also known as autonomous driving maps for vehicles, are currently mostly produced and updated using a crowdsourcing model. In the crowdsourcing model, the vehicle identifies scene map elements, the cloud performs vector data preprocessing and optimization, and then the map is generated through a combination of semi-automatic and manual assistance.

[0003] In the cloud-based preprocessing stage, different operators are used to process the data based on its source, format, and type. This process requires continuous parameter updates, and the return results are used to determine whether other algorithms need to be introduced to optimize the data. When multiple operators are used to complete the data processing, each operator needs to be traversed multiple times, and the execution order of the multiple operators is not determined. The parameter settings and different combinations of operators have a significant impact on the results and efficiency of data preprocessing. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to improve the efficiency of data processing.

[0005] To address at least one of the aforementioned technical problems, this invention discloses a data processing method, apparatus, device, and storage medium.

[0006] According to one aspect of this disclosure, a data processing method is provided, comprising:

[0007] Obtain target map data;

[0008] Based on the target map data, multiple operators to be executed are determined; the operators to be executed indicate various data processing operations corresponding to the target map data;

[0009] Based on the operator dependencies of the multiple operators to be executed, a directed acyclic graph is constructed for the multiple operators to be executed to obtain a target directed acyclic graph corresponding to the multiple operators to be executed; each operator to be executed corresponds to a node in the target directed acyclic graph;

[0010] Based on the in-degree information of each node in the target directed acyclic graph, the execution order of each of the multiple operators to be executed is determined.

[0011] Based on the execution order of the multiple operators to be executed, the target map data is processed to obtain the target processing result.

[0012] In some possible embodiments, determining multiple operators to be executed based on the target map data includes:

[0013] The target map data is subjected to data format detection to obtain the format detection result;

[0014] If the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, a data conversion operator corresponding to the data format of the target map data is determined.

[0015] The target map data is processed and analyzed to obtain the data processing and analysis results;

[0016] Based on the data processing and analysis results, determine the data processing operator corresponding to the target map data;

[0017] Based on the data format of the data conversion operator, determine the data output operator corresponding to the data format of the data conversion operator;

[0018] Based on the data conversion operator, the data processing operator, and the data output operator, the plurality of operators to be executed are determined.

[0019] In some possible embodiments, before constructing a directed acyclic graph (DAG) for the plurality of operators to be executed based on their operator dependencies to obtain a target DAG corresponding to the plurality of operators to be executed, the method further includes:

[0020] Obtain the operator type parameters corresponding to each of the plurality of operators to be executed;

[0021] Based on the operator type parameters corresponding to each of the multiple operators to be executed, the dependencies between the nodes are determined.

[0022] In some possible embodiments, the step of constructing a directed acyclic graph (DAG) for the plurality of operators to be executed based on the operator dependencies, to obtain a target DAG corresponding to the plurality of operators to be executed, includes:

[0023] Based on the dependencies between the nodes, node verification is performed on any two nodes to obtain node verification results; the node verification results include dependency verification results and graph verification results.

[0024] Based on the node verification results, the connection relationship between any two nodes is verified to obtain the target directed acyclic graph.

[0025] In some possible embodiments, after constructing a directed acyclic graph (DAG) for the plurality of operators to be executed based on their operator dependencies to obtain a target DAG corresponding to the plurality of operators to be executed, the method includes:

[0026] Obtain the data operation information corresponding to each node in the target directed acyclic graph;

[0027] Based on the data operation information, information filling operations are performed on each node of the target directed acyclic graph to obtain the updated target directed acyclic graph.

[0028] In some possible embodiments, determining the execution order of the plurality of operators to be executed based on the in-degree information of each node in the target directed acyclic graph includes:

[0029] The current node is determined based on the updated target directed acyclic graph; the current node is the node with an in-degree of 0 in the updated target directed acyclic graph.

[0030] Add the current node to the node queue;

[0031] Delete the current node and each out-degree edge of the current node from the current directed acyclic graph to obtain the updated directed acyclic graph;

[0032] In the updated directed acyclic graph, the node with an in-degree of 0 is identified as the current node.

[0033] Repeated execution: Add the current node to the node queue until it is determined that the node with an in-degree value of 0 in the updated directed acyclic graph is the current node, until the current node is the last node in the updated directed acyclic graph with an in-degree value of 0 and an out-degree value of 0.

[0034] Based on the node queue, the execution order of each of the multiple operators to be executed is determined.

[0035] In some possible embodiments, processing the target map data based on the execution order of the plurality of operators to be executed to obtain the target processing result includes:

[0036] Based on the execution order of the multiple operators to be executed, each node is traversed.

[0037] Based on the currently traversed node, perform data processing operations on the target map data corresponding to the currently traversed node to obtain the data processing result corresponding to the currently traversed node;

[0038] The target processing result is generated based on the data processing results corresponding to each node.

[0039] According to a second aspect of this disclosure, a data processing apparatus is provided, comprising:

[0040] The data acquisition module is used to acquire target map data;

[0041] An operator determination module is used to determine multiple operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

[0042] The graph construction module is used to construct a directed acyclic graph for the multiple operators to be executed based on the operator dependency relationship, so as to obtain a target directed acyclic graph corresponding to the multiple operators to be executed; each operator to be executed corresponds to a node in the target directed acyclic graph;

[0043] The sequence determination module is used to determine the execution order of the multiple operators to be executed based on the in-degree information of each node in the target directed acyclic graph.

[0044] The data processing module is used to process the target map data based on the execution order of the multiple operators to be executed, and obtain the target processing result.

[0045] According to a third aspect of this disclosure, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the data processing method as described above.

[0046] According to a fourth aspect of this disclosure, a computer storage medium is provided, wherein at least one instruction and at least one program are stored therein, the at least one instruction and the at least one program being loaded and executed by a processor to implement the data processing method described above.

[0047] Implementing this invention has the following beneficial effects:

[0048] This invention determines multiple operators to be executed based on target map data and constructs a target directed acyclic graph (DAG) corresponding to these operators to determine the in-degree information of each node in the DAG. Furthermore, based on the in-degree information of each node in the DAG, the execution order of the multiple operators to be executed is obtained. Data processing of the target map data is performed according to this execution order, avoiding multiple traversals of the operators to be executed during the processing of the target map data, thereby improving the efficiency of determining the operators to be executed. Furthermore, by reasonably determining the execution order of the multiple operators based on the DAG, the accuracy of target map data processing can be improved. Attached Figure Description

[0049] To more clearly illustrate the technical solution of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the process of determining the corresponding operator provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram illustrating the process of determining node dependencies provided in this embodiment of the invention;

[0053] Figure 4 This is a schematic diagram illustrating the process of constructing a target directed acyclic graph provided in an embodiment of the present invention.

[0054] Figure 5 A flowchart illustrating the information filling process provided in this embodiment of the invention;

[0055] Figure 6 A flowchart illustrating the execution order determination provided in this embodiment of the invention;

[0056] Figure 7 This is a schematic diagram of the process corresponding to the target map data processing provided in the embodiments of the present invention;

[0057] Figure 8 This is a schematic diagram of the data processing toolbox structure corresponding to one embodiment of the present invention;

[0058] Figure 9 This is a schematic diagram illustrating the execution sequence of a single data cleaning pass according to one embodiment of the present invention;

[0059] Figure 10 This is a schematic diagram illustrating the execution sequence of multiple passes of combined fusion according to one embodiment of the present invention;

[0060] Figure 11 This is a visualization image corresponding to map data provided in one embodiment of the present invention;

[0061] Figure 12 This is a visualization image corresponding to the denoised map data provided in one embodiment of the present invention;

[0062] Figure 13 This is a visualization image corresponding to the fitted map data provided in one embodiment of the present invention;

[0063] Figure 14 This is a visualization image corresponding to the fused map data provided in one embodiment of the present invention;

[0064] Figure 15 This is a visualization image corresponding to the data processing result provided in one embodiment of the present invention;

[0065] Figure 16 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0068] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0069] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0070] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0071] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0072] Figure 1 The diagram shows a flowchart of the data processing method provided in this embodiment of the invention; the executing entity can be any terminal capable of running the data processing method; specifically, any terminal can be a vehicle body controller, a server terminal, etc.

[0073] like Figure 1 As shown, a data processing method includes:

[0074] Step S101: Obtain target map data;

[0075] In this embodiment of the invention, the target map data may be map data obtained through a crowdsourcing collection mode and used for the production or updating of high-precision maps.

[0076] Step S102: Determine multiple operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

[0077] In this embodiment of the invention, an operator is an algorithm used to process target map data. Different operators correspond to different data processing methods. Multiple operators to be executed are determined according to the data format and processing requirements of the target map data. That is, multiple operators to be executed that are suitable for the current target map data can be selected from the operators.

[0078] Step S103: Based on the operator dependency relationship of the plurality of operators to be executed, construct a directed acyclic graph for the plurality of operators to be executed to obtain a target directed acyclic graph corresponding to the plurality of operators to be executed; each operator to be executed corresponds to a node in the target directed acyclic graph;

[0079] In this embodiment of the invention, the operator dependency relationship refers to the order in which operators are executed. Specifically, if operator A is used to process data, operator B is used to acquire data, and operator C is used to output data, then the execution order of operator A, operator B, and operator C is operator B - operator A - operator C.

[0080] In a specific embodiment, each operator to be executed corresponds to a node in the target directed acyclic graph (DAG). When constructing a DAG for multiple operators to be executed, in addition to considering operator dependencies, it is also necessary to consider whether a DAG can be obtained after associating multiple operators. Specifically, for example, based on the obtained target map data, operators 1, 2, 3, and 4 are determined to be multiple operators to be executed, and the operator dependencies indicate that operator 1 can connect to operator 2, operator 2 can connect to operator 3, and operator 3 can connect to both operator 2 and operator 4. In the DAG generated based on the operator dependencies, operators 2 and 3 form a closed loop, which does not meet the requirements of a DAG. Therefore, when constructing the DAG, operator 2 can connect to operator 3, but operator 3 cannot connect to operator 2.

[0081] Step S104: Based on the in-degree information of each node in the target directed acyclic graph, determine the execution order of each of the multiple operators to be executed;

[0082] In this embodiment of the invention, after obtaining the target directed acyclic graph, it is necessary to determine the execution order of each node in the target directed acyclic graph. The execution order is calculated based on the in-degree information of each node.

[0083] Specifically, in-degree information refers to the in-degree value of each node. For each node in the target directed acyclic graph, its in-degree value is the number of edges ending at the current node.

[0084] Step S105: Based on the execution order of the multiple operators to be executed, process the target map data to obtain the target processing result.

[0085] In this embodiment of the invention, the execution order of each node is the execution order of each operator to be executed, and the target map data is processed according to the execution order of each operator to be executed.

[0086] Specifically, the corresponding operators to be executed are executed according to the first-in, first-out order of each node in the node queue. The current operator to be executed is reflected and parsed, and then combined with data operation information to achieve task scheduling. Reflection parsing is used to dynamically obtain or call data information of each node, such as operator type parameters and data operation information. After each operator to be executed completes its task, memory message listening is used to trigger the next operator to be executed to perform the corresponding data processing operation. The data processing result of the previous stage is used as the input of the next stage, and finally the target processing result is obtained.

[0087] In this embodiment of the invention, multiple operators to be executed are determined based on the target map data, and a target directed acyclic graph (DAG) corresponding to the multiple operators to be executed is constructed to determine the in-degree information of each node in the target DAG. Furthermore, based on the in-degree information of each node in the target DAG, the execution order of the multiple operators to be executed can be obtained. Data processing of the target map data according to the execution order of the multiple operators to be executed can avoid traversing multiple operators to be executed multiple times during the processing of the target map data, thereby improving the efficiency of determining the operators to be executed. Furthermore, based on the reasonable determination of the execution order of multiple operators based on the DAG, the accuracy of the target map data processing can be improved.

[0088] Figure 2 This diagram illustrates the flowchart for determining the corresponding operator provided in an embodiment of the present invention, such as... Figure 2 As shown, the step of determining multiple operators to be executed based on the target map data includes:

[0089] Step S201: Perform data format detection on the target map data to obtain the format detection result;

[0090] In this embodiment of the invention, the data processing toolbox includes a data source operator set, a data processing operator set, and a data output operator set. Each of the data input, data processing, and data output operator sets corresponds to an operator type, and each operator type includes multiple operators. Therefore, the determination of the operator to be executed begins with the data output and ends with the data output.

[0091] Specifically, after obtaining the target map data, the first step is to perform data format detection on the target map data to determine whether the data format of the target map data is compatible with the data format provided by the data source subset.

[0092] Step S202: If the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, determine the data conversion operator corresponding to the data format of the target map data;

[0093] In this embodiment of the invention, the format detection result determines whether the target map data needs to be converted. If the data format of the target map data is inconsistent with the preset data format, a data conversion operator that can convert the data format of the target map data is determined. If the data format of the target map data is consistent with the preset data format, it is not necessary to determine the operator from the data source operator set.

[0094] In another specific embodiment, the target map data format can be directly converted to a format compatible with the data processing toolbox. That is, for target map data of any format, the data format of the target map data is converted into GeoJSON format based on the data source operator subset in the data processing toolbox.

[0095] In another specific embodiment, if there is no data conversion operator in the data source operator set that corresponds to the data format of the target map data, a data conversion operator corresponding to the data format of the target map data can be developed and saved to the data source operator set.

[0096] Step S203: Perform data processing and analysis on the target map data to obtain the data processing and analysis results;

[0097] In this embodiment of the invention, by analyzing the target map data, data processing analysis results are obtained to determine the data processing operations that need to be performed on the target map data.

[0098] Step S204: Based on the data processing and analysis results, determine the data processing operator corresponding to the target map data;

[0099] In this embodiment of the invention, based on the data processing and analysis results, an operator to be executed corresponding to the data processing operation required for the target map data is determined from the data processing operator set. Depending on the target map data, the operator to be executed determined from the data processing operator set may include one or more.

[0100] Step S205: Based on the data format of the data conversion operator, determine the data output operator corresponding to the data format of the data conversion operator;

[0101] In this embodiment of the invention, the data format specified by the OGC (Open Geospatial Consortium) standard is used as the unified vector feature data transmission format in the data processing toolbox to shield the differences in the connection of various operators. Therefore, depending on the actual application of the target map data, data format conversion is required when outputting the processing results. The data output format can be consistent with the data format corresponding to the data conversion operator, that is, the data format corresponding to the data output operator is consistent with the data format corresponding to the data conversion operator.

[0102] In another embodiment, the data output format can also be determined according to the data format corresponding to the storage layer where the output data is stored, that is, the data format corresponding to the data output operator is consistent with the data format corresponding to the storage layer.

[0103] Step S206: Based on the data conversion operator, the data processing operator, and the data output operator, determine the plurality of operators to be executed.

[0104] In this embodiment of the invention, the data conversion operator, data processing operator, and data output operator determined based on the target map data are multiple operators to be executed.

[0105] In this embodiment of the invention, by unifying the data format of target map data in the data processing toolbox, the differences in the connection of various operators can be shielded; by converting the data format, it is possible to be compatible with target map data of different data formats, thereby improving the versatility of the data processing toolbox; by determining multiple corresponding operators to be executed based on the target map data, the accuracy of target map data processing can be improved.

[0106] Figure 3 This diagram illustrates the process for determining node dependencies according to an embodiment of the present invention. Figure 3 As shown, before constructing a directed acyclic graph (DAG) for the multiple operators to be executed based on their operator dependencies to obtain the target DAG corresponding to the multiple operators to be executed, the method further includes:

[0107] Step S301: Obtain the operator type parameters corresponding to each of the plurality of operators to be executed;

[0108] In this embodiment of the invention, a graph editing engine is used to construct a directed acyclic graph for multiple operators to be executed. Before constructing the target directed acyclic graph, it is necessary to determine each node in the target directed acyclic graph.

[0109] Specifically, the operator type parameter includes the name of the operator to be executed and the type of the operator to be executed; for example, operator m is used to perform a multi-pass fitting operation on the target map data, and the operator type parameter corresponding to operator m is (data processing, multi-pass fitting); furthermore, an identifier can be set to represent the name of the operator to be executed and the type of the operator to be executed, for example, 01 represents data processing and fitting represents multi-pass fitting, then the operator type parameter corresponding to operator m is (01, fitting).

[0110] In this embodiment of the invention, each node in the target directed acyclic graph corresponds to an operator to be executed. The number of nodes in the target directed acyclic graph is determined based on the number of operators to be executed. After determining the number of nodes, the operator type parameters of the operators to be executed need to be mapped to each node to determine the operator to be executed represented by each node in the target directed acyclic graph.

[0111] Step S302: Determine the dependency relationship between the nodes based on the operator type parameters corresponding to each of the multiple operators to be executed.

[0112] In this embodiment of the invention, the dependency relationship between nodes is the operator dependency relationship, that is, the order in which operators are executed. Specifically, the order in which operators are executed is determined based on the operator parameter type, thereby determining the operator dependency relationship. For example, if operator A is used to process data, operator B is used to acquire data, and operator C is used to output data, then the execution order of operator A, operator B, and operator C is operator B - operator A - operator C.

[0113] In another specific embodiment, operator A1 is data processing operator 1, operator A2 is data processing operator 2, operator B1 is data transformation operator, and operator C1 is data output operator. The execution order among operators A1, A2, B1, and C1 can be operator B1-operator A1-operator A2-operator C1 or operator B1-operator A2-operator A1-operator C1. Which operator dependency relationship is applicable needs to be determined based on the specific target map data, that is, based on the specific target map data and the construction requirements of the directed acyclic graph, the corresponding operator dependency relationship is determined.

[0114] In another specific embodiment, the operator dependency can be preset by the user based on the data information contained in the specific target map data.

[0115] In this embodiment of the invention, generating corresponding operator dependencies based on different target map data can improve the diversity of connections between operators.

[0116] Figure 4 This diagram illustrates the process of constructing a target directed acyclic graph according to an embodiment of the present invention, as shown below. Figure 4As shown, the step of constructing a directed acyclic graph (DAG) for the multiple operators to be executed based on their operator dependencies, to obtain a target DAG corresponding to the multiple operators to be executed, includes:

[0117] Step S401: Based on the dependency relationships between the nodes, perform node verification on any two nodes to obtain node verification results; the node verification results include dependency verification results and graph verification results.

[0118] In this embodiment of the invention, during the construction of the target directed acyclic graph using a graph editing engine, the nodes are randomly connected. Besides considering operator dependencies, it is also necessary to consider whether a directed acyclic graph can be obtained after associating multiple operators to be executed. After the nodes are connected, node verification is required for any two nodes to determine whether the connection is correct and reasonable.

[0119] Furthermore, node verification includes dependency verification and graph verification, and the node verification results include dependency verification results and graph verification results. Dependency verification refers to verifying the connection relationship between every two nodes based on the dependencies between each node, generating a dependency verification result; graph verification refers to verifying whether the initial acyclic graph meets the requirements of a directed acyclic graph, based on the fact that the connection relationship between two nodes conforms to the dependencies between each node, generating a graph verification result.

[0120] Step S402: Verify the connection relationship between any two nodes based on the node verification result to obtain the target directed acyclic graph.

[0121] In this embodiment of the invention, based on the node verification result, if the dependency verification result indicates that the connection relationship between two nodes conforms to the dependency relationship between each node, a graph verification is performed to determine whether the connection between nodes conforms to a directed acyclic graph; if the graph verification result indicates that the connection relationship between two nodes conforms to a directed acyclic graph, then the node connection is successful.

[0122] In another embodiment, if the dependency verification result indicates that the connection relationship between two nodes does not conform to the dependency relationship between the nodes, a new node is selected for connection, and node verification is performed on it.

[0123] In another embodiment, if the dependency verification result indicates that the connection relationship between two nodes conforms to the dependency relationship between each node, but its graph verification result does not meet the requirements of a directed acyclic graph, the connection relationship is corrected according to the specific target map data, and the node verification is performed again.

[0124] In a specific embodiment, specifically, based on the acquired target map data, operators 1, 2, 3, and 4 are determined to be multiple operators to be executed, and the operator dependency relationship indicates that operator 1 can connect to operator 2, operator 2 can connect to operator 3, and operator 3 can connect to both operator 2 and operator 4. In the directed acyclic graph generated based on the operator dependency relationship, operator 2 and operator 3 form a closed loop, which does not meet the requirements of a directed acyclic graph. When constructing the directed acyclic graph, operator 2 can connect to operator 3, but operator 3 cannot connect to operator 2.

[0125] In this embodiment of the invention, node verification can ensure the accuracy of the construction of the target directed acyclic graph, thereby improving the efficiency of the process of determining the execution order of operators based on the directed acyclic graph to process the target map data.

[0126] Figure 5 A flowchart illustrating the information filling process provided in the embodiments of the present invention, such as... Figure 5 As shown, after constructing a directed acyclic graph (DAG) for the multiple operators to be executed based on their operator dependencies, and obtaining the target DAG corresponding to the multiple operators to be executed, the method includes:

[0127] Step S501: Obtain the data operation information corresponding to each node in the target directed acyclic graph;

[0128] In this embodiment of the invention, the data operation information includes the entity attributes of the operator to be executed, which are determined based on the parameters of the data processing operation corresponding to the operator. For example, if the data processing operation corresponding to the operator to be executed is distance thinning, the entity attributes of the node corresponding to the operator to be executed include a set length, for example, setting distance thinning to be performed according to a length of one meter, which means that the parameter corresponding to the node for distance thinning is 1.

[0129] Step S502: Based on the data operation information, perform information filling operation on each node of the target directed acyclic graph to obtain the updated target directed acyclic graph.

[0130] In this embodiment of the invention, in the target directed acyclic graph, each node includes an operator type parameter of an operator to be executed, but does not include the entity attribute of the operator to be executed. Therefore, it is necessary to fill the node with the entity attribute corresponding to the operator to be executed.

[0131] In this embodiment of the invention, data operation information is filled into the target directed acyclic graph to obtain an updated target directed acyclic graph. The data processing operation corresponding to the operator to be executed can be directly executed based on the data operation information filled in the node, thereby improving the convenience of data processing.

[0132] Figure 6 This diagram illustrates the execution order determination process provided in an embodiment of the present invention. Figure 6 As shown, determining the execution order of the plurality of operators to be executed based on the in-degree information of each node in the target directed acyclic graph includes:

[0133] Step S601: Determine the current node based on the updated target directed acyclic graph; the current node is the node with an in-degree of 0 in the updated target directed acyclic graph;

[0134] In this embodiment of the invention, the order in which nodes are added to the node queue is determined based on their in-degree values. A node with an in-degree of 0, representing the starting point of a directed acyclic graph (DAG), is the first step in the entire process. Therefore, in this embodiment, after obtaining the updated target DAG, the node with an in-degree of 0 in the updated target DAG is first identified as the current node, and then added to the node queue as the starting point for processing the target map data.

[0135] Step S602: Add the current node to the node queue;

[0136] In this embodiment of the invention, each node with an in-degree value of 0 is added to the node queue. When there are multiple nodes with an in-degree value of 0, the nodes with an in-degree value of 0 are randomly added to the node queue.

[0137] Step S603: Delete the current node and each out-degree edge of the current node from the current directed acyclic graph to obtain the updated directed acyclic graph;

[0138] Step S604: In the updated directed acyclic graph, the node with an in-degree value of 0 is the current node;

[0139] In this embodiment of the invention, based on the updated directed acyclic graph, the in-degree value of each node in the graph is recalculated, and nodes with an in-degree value of 0 are added to the node queue.

[0140] Step S605: Determine whether the current node is the last node in the updated directed acyclic graph with an in-degree value of 0 and an out-degree value of 0.

[0141] Step S606: If yes, add the current node to the node queue to obtain the updated node queue; determine the updated node queue as the execution order corresponding to the multiple operators to be executed.

[0142] Step S607: If not, repeat steps S602-S605.

[0143] In this embodiment of the invention, steps S602-S605 are repeated until the last node in the updated directed acyclic graph has an in-degree value of 0 and an out-degree value of 0, in order to generate a node queue including all nodes in the updated directed acyclic graph. The execution order of multiple operators to be executed is then determined based on the first-in-first-out order of nodes in the node queue. Therefore, it is necessary to determine whether the current node is the last node in the updated directed acyclic graph with an in-degree value of 0 and an out-degree value of 0, in order to determine the next execution step.

[0144] In this embodiment of the invention, in the updated directed acyclic graph, the order in which each node is added to the node queue is determined based on the in-degree value of each node. The execution order of multiple operators to be executed is determined according to the first-in-first-out order when each node is added to the queue. The corresponding execution order is generated according to different target map data, thereby improving the efficiency of determining the operators to be executed and improving the flexibility of data processing.

[0145] Figure 7 This diagram illustrates the flowchart of target map data processing provided in an embodiment of the present invention, such as... Figure 7 As shown, the process of processing the target map data based on the execution order of the plurality of operators to be executed, to obtain the target processing result, includes:

[0146] Step S701: Based on the execution order of the multiple operators to be executed, traverse each node;

[0147] In this embodiment of the invention, each node is traversed according to its first-in-first-out order in the node queue to obtain the current operator to be executed.

[0148] Step S702: Based on the currently traversed node, perform data processing operations on the target map data corresponding to the currently traversed node to obtain the data processing result corresponding to the currently traversed node;

[0149] In this embodiment of the invention, based on the obtained current operator to be executed, the current operator to be executed is reflected and parsed, and the data processing operation corresponding to the current operator to be executed is performed on the target map data according to the filled data operation information to obtain the data processing result.

[0150] Step S703: Generate the target processing result based on the data processing results corresponding to each node.

[0151] In this embodiment of the invention, after each operator to be executed completes its task, it uses memory message listening to trigger the next operator to be executed to perform the corresponding data processing operation. The data processing result obtained in the previous stage is used as the input for the next stage, and finally the target processing result is obtained.

[0152] In this embodiment of the invention, the target map data is processed according to the execution order of multiple operators to be executed to obtain the target processing result. This eliminates the need to traverse multiple operators to be executed when performing each data processing operation, thereby improving data processing efficiency.

[0153] In one specific embodiment, the data processing toolbox, such as Figure 8 As shown, the data toolbox includes a set of data source operators, a set of data processing operators, and a set of data output operators. Each set of operators includes multiple operators of the corresponding type. After obtaining the map data to be processed, multiple operators to be executed are determined from the data processing toolbox. For example, if denoising, coordinate transformation, thinning sampling, distance clustering, piecewise line fitting, and data output operations need to be performed on the map data to be processed, a target directed acyclic graph is constructed based on the multiple operators to be executed to determine the execution order of the multiple operators. Data processing of the map data to be processed can be performed through real-time triggering or timed triggering. Real-time triggering uses Kafka (Apache Kafka, a distributed publish-subscribe messaging system) to listen for processing task topics, and triggers the task when a message is received. Timed task triggering uses a timer to set execution time rules.

[0154] Figure 9 The execution order of multiple operators to be executed for a single pass of data cleaning of the map data to be processed; such as Figure 9 As shown, the execution order of the multiple operators to be executed is as follows: data input (PG format), coordinate transformation, distance clustering, thinning sampling, denoising, distance clustering, piecewise fitting of straight lines, and data output (PG format). Data processing is performed on the map data to be processed based on the execution order.

[0155] In another specific embodiment, Figure 10 The execution order of multiple operators to be executed for the combination and fusion of data to be processed; such as Figure 10 As shown, after the data input (PG format), the results of distance clustering, fitting, and aggregation of the map data to be processed are subjected to intersection extraction with the data after R-tree indexing of the data to be processed. The extracted map data is then subjected to smoothing curves, topology verification, and data output (PG format). The data processing of the map data to be processed is performed based on the execution order.

[0156] Figure 11 This is a visualization image corresponding to map data provided in one embodiment of the present invention; Figure 12 This is a visualization image corresponding to the denoised map data provided in one embodiment of the present invention; Figure 13 This is a visualization image corresponding to the fitted map data provided in one embodiment of the present invention; Figure 14 This is a visualization image corresponding to the fused map data provided in one embodiment of the present invention; Figure 15 This is a visualization image corresponding to the data processing result provided in one embodiment of the present invention.

[0157] In another specific embodiment, such as Figure 11 As shown, after inputting the collected map data into the data processing toolbox, noise reduction processing is performed on the data to be processed, such as... Figure 12 As shown, the denoised map data is obtained; data fitting processing is then performed on the denoised map data, as follows: Figure 13 As shown, the fitted map data is obtained; the fitted map data is then fused, as follows: Figure 14 As shown, the merged map data is obtained, and the merged data is the final data processing result. The final data processing result is stored in the results database table, and the final data processing result is output, as shown below. Figure 15 As shown, a high-speed base map corresponding to the map data is obtained.

[0158] This invention also provides a data processing apparatus, such as... Figure 16 As shown, the device includes:

[0159] Data acquisition module 1610 is used to acquire target map data;

[0160] The operator determination module 1620 is used to determine multiple operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data.

[0161] The graph construction module 1630 is used to construct a directed acyclic graph for the multiple operators to be executed based on the operator dependency relationship, so as to obtain a target directed acyclic graph corresponding to the multiple operators to be executed; each operator to be executed corresponds to a node in the target directed acyclic graph;

[0162] The sequence determination module 1640 is used to determine the execution order of each of the multiple operators to be executed based on the in-degree information of each node in the target directed acyclic graph.

[0163] The data processing module 1650 is used to process the target map data based on the execution order of the multiple operators to be executed, and obtain the target processing result.

[0164] In other embodiments, the data acquisition module 1610 further includes:

[0165] The format detection module is used to perform data format detection on the target map data and obtain the format detection result;

[0166] The first operator determination module is used to determine a data conversion operator corresponding to the data format of the target map data when the format detection result indicates that the data format of the target map data is inconsistent with the preset data format;

[0167] The data analysis module is used to process and analyze the target map data to obtain the data processing and analysis results;

[0168] The second operator determination module is used to determine the data processing operator corresponding to the target map data based on the data processing and analysis results.

[0169] The third operator determination module is used to determine the data output operator corresponding to the data format of the data conversion operator based on the data format of the data conversion operator;

[0170] The overall operator determination module is used to determine the plurality of operators to be executed based on the data transformation operator, the data processing operator, and the data output operator.

[0171] In other embodiments, the device further includes:

[0172] The parameter determination module is used to obtain the operator type parameters corresponding to each of the plurality of operators to be executed;

[0173] The dependency determination module is used to determine the dependency relationships between the nodes based on the operator type parameters corresponding to each of the multiple operators to be executed.

[0174] In other embodiments, the graphics building module 1630 further includes:

[0175] The node verification module is used to perform node verification on any two nodes based on the dependency relationship between the nodes, and obtain the node verification result; the node verification result includes dependency verification result and graph verification result;

[0176] The graph generation module is used to verify the connection relationship between any two nodes based on the node verification result, and obtain the target directed acyclic graph.

[0177] In other embodiments, the device further includes:

[0178] The information acquisition module is used to acquire data operation information corresponding to each node in the target directed acyclic graph;

[0179] The information filling module is used to perform information filling operations on each node of the target directed acyclic graph based on the data operation information, so as to obtain the updated target directed acyclic graph.

[0180] In other embodiments, the sequence determination module 1640 further includes:

[0181] First current node determination module: used to determine the current node based on the updated target directed acyclic graph; the current node is the node with an in-degree value of 0 in the updated target directed acyclic graph;

[0182] The node queue determination module is used to add the current node to the node queue;

[0183] The graph update module is used to delete the current node and each out-degree edge of the current node from the current directed acyclic graph to obtain the updated directed acyclic graph.

[0184] Second current node determination module: used to determine the node with an in-degree value of 0 in the updated directed acyclic graph as the current node;

[0185] The repeated execution module is used to repeatedly execute: adding the current node to the node queue until it is determined that the node with an in-degree value of 0 in the updated directed acyclic graph is the current node, until the current node is the last node in the updated directed acyclic graph with an in-degree value of 0 and an out-degree value of 0.

[0186] The execution order generation module is used to determine the execution order of the multiple operators to be executed based on the node queue.

[0187] In other embodiments, the data processing module 1160 further includes:

[0188] The node traversal module is used to traverse each node based on the execution order of the multiple operators to be executed.

[0189] The data processing execution module is used to perform data processing operations on the target map data based on the currently traversed node, and obtain the data processing result corresponding to the currently traversed node.

[0190] The target result generation module is used to generate the target processing result based on the data processing results corresponding to each node.

[0191] The apparatus and method embodiments described above are based on the same inventive concept and are used to implement the above-described data processing method.

[0192] This invention also provides a data processing device, the device comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the data processing method as described in the method embodiment.

[0193] Embodiments of the present invention also provide a storage medium, which may be disposed in a server to store at least one instruction, at least one program, code set, or instruction set for implementing a data management method in the method embodiments, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data processing method as described in any of the method embodiments.

[0194] Optionally, in embodiments of the present invention, the storage medium may be located at at least one of a plurality of network servers in a computer network. Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0195] As can be seen from the embodiments provided by the present invention above, the present invention determines multiple operators to be executed based on the target map data and constructs a target directed acyclic graph corresponding to the multiple operators to be executed, so as to determine the in-degree information of each node in the target directed acyclic graph; furthermore, based on the in-degree information of each node in the target directed acyclic graph, the execution order of the multiple operators to be executed is obtained, and the target map data is processed according to the execution order of the multiple operators to be executed, which can avoid traversing multiple operators to be executed multiple times in the process of processing the target map data, thereby improving the efficiency of determining the operators to be executed; and furthermore, based on the reasonable determination of the execution order of multiple operators based on the directed acyclic graph, the accuracy of processing the target map data can be improved.

[0196] It should be noted that the embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, characterized in that, Applied to the vehicle end, the method includes: Obtain target map data; Based on the target map data and the vehicle's processing requirements, multiple operators to be executed are determined; the operators to be executed indicate various data processing operations corresponding to the target map data; Based on the operator dependency relationship of the multiple operators to be executed, node verification is performed on any two nodes to obtain node verification results; the node verification results include dependency verification results and graph verification results. The connection relationship between any two nodes is verified based on the dependency verification result and the graph verification result to obtain the target directed acyclic graph; each operator to be executed corresponds to a node in the target directed acyclic graph; Based on the in-degree information of each node in the target directed acyclic graph, the execution order of each of the multiple operators to be executed is determined. Based on the execution order of the multiple operators to be executed, the target map data is processed to obtain the target processing result displayed on the vehicle.

2. The data processing method according to claim 1, characterized in that, The determination of multiple operators to be executed based on the target map data includes: The target map data is subjected to data format detection to obtain the format detection result; If the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, a data conversion operator corresponding to the data format of the target map data is determined. The target map data is processed and analyzed to obtain the data processing and analysis results; Based on the data processing and analysis results, determine the data processing operator corresponding to the target map data; Based on the data format of the data conversion operator, determine the data output operator corresponding to the data format of the data conversion operator; Based on the data conversion operator, the data processing operator, and the data output operator, the plurality of operators to be executed are determined.

3. The data processing method according to claim 1, characterized in that, The method further includes: Obtain the operator type parameters corresponding to each of the plurality of operators to be executed; Based on the operator type parameters corresponding to each of the multiple operators to be executed, the operator dependencies of the multiple operators to be executed are determined.

4. The data processing method according to claim 1, characterized in that, The method includes: Obtain the data operation information corresponding to each node in the target directed acyclic graph; Based on the data operation information, information filling operations are performed on each node of the target directed acyclic graph to obtain the updated target directed acyclic graph.

5. The data processing method according to claim 4, characterized in that, The step of determining the execution order of the plurality of operators to be executed based on the in-degree information of each node in the target directed acyclic graph includes: The current node is determined based on the updated target directed acyclic graph; the current node is the node with an in-degree of 0 in the updated target directed acyclic graph. Add the current node to the node queue; Delete the current node and each out-degree edge of the current node from the current directed acyclic graph to obtain the updated directed acyclic graph; In the updated directed acyclic graph, the node with an in-degree of 0 is identified as the current node. Repeated execution: Add the current node to the node queue until it is determined that the node with an in-degree value of 0 in the updated directed acyclic graph is the current node, until the current node is the last node in the updated directed acyclic graph with an in-degree value of 0 and an out-degree value of 0. Based on the node queue, the execution order of each of the multiple operators to be executed is determined.

6. The data processing method according to claim 1, characterized in that, The process of processing the target map data based on the execution order of the plurality of operators to be executed, to obtain the target processing result, includes: Based on the execution order of the multiple operators to be executed, each node is traversed. Based on the currently traversed node, perform data processing operations on the target map data corresponding to the currently traversed node to obtain the data processing result corresponding to the currently traversed node; The target processing result is generated based on the data processing results corresponding to each node.

7. A data processing apparatus, characterized in that, Applied to the vehicle end, the device includes: The data acquisition module is used to acquire target map data; The operator determination module is used to determine multiple operators to be executed based on the target map data and the processing requirements of the vehicle; the operators to be executed indicate various data processing operations corresponding to the target map data; A graph construction module is used to perform node verification on any two nodes based on the operator dependency relationship of the multiple operators to be executed, and obtain a node verification result; the node verification result includes a dependency verification result and a graph verification result; the connection relationship between any two nodes is verified based on the dependency verification result and the graph verification result to obtain a target directed acyclic graph; each operator to be executed corresponds to a node in the target directed acyclic graph; The sequence determination module is used to determine the execution order of the multiple operators to be executed based on the in-degree information of each node in the target directed acyclic graph. The data processing module is used to process the target map data based on the execution order of the multiple operators to be executed, and to obtain the target processing result displayed on the vehicle.

8. An electronic device comprising a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the data processing method as claimed in any one of claims 1-6.

9. A computer storage medium storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by a processor to implement the data processing method as claimed in any one of claims 1-6.

Images