Method and device for processing map data acquisition task

Abstract

The embodiment of the invention discloses a map data acquisition task processing method and device. According to the technical scheme, the method comprises the following steps: acquiring demand data of a map data acquisition task, wherein the demand data comprises an acquisition demand and a hardware performance constraint condition of a to-be-deployed node; inputting the demand data into a demand analysis model, and generating a target calculation graph meeting the acquisition demand and the hardware performance constraint condition; the target calculation graph comprises a plurality of target calculation nodes and directed edges, each target calculation node corresponds to a processing unit in a map data acquisition algorithm, and the directed edges indicate the data flow direction between the processing units corresponding to the two target calculation nodes connected with the directed edges; and packaging the target computational graph into an executable software product, wherein the executable software product is used for being deployed into the to-be-deployed node. According to the method, the flexibility and adaptation capability of map data acquisition algorithm development can be improved, meanwhile, the maintenance cost is reduced, and the development efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of map data, and particularly to a method and device for processing map data collection tasks. Background Art

[0002] With the development of technology and the popularization of mobile Internet, the map industry is undergoing a major transformation from static maps to dynamic maps. The geographical information provided by traditional static maps is fixed and has a long update cycle, unable to reflect the immediate road conditions and traffic events in real time, and it is difficult to meet the user's demand for real-time travel information. In modern travel scenarios, users hope to understand road dynamic events in real time, such as traffic accidents, road construction, congestion situations, and weather impacts, etc., to optimize travel routes. This demand has prompted the map industry to transform towards dynamic maps, and real-time collection and update of road dynamic data have become the key to industry competition.

[0003] However, there are many problems in the traditional development method of map data collection algorithms, mainly manifested in poor flexibility, high maintenance cost, and low development efficiency. Specifically, its deployment process is fixed, and steps such as pre-processing, model inference, and post-processing are hard-coded in the program. Each model has an independent and fixed processing process, resulting in the need to manually modify the code, recompile, and deploy when the collection requirements change or the model is updated; in addition, dividing the development process by model leads to code redundancy, increasing the maintenance and extension costs. At the same time, the fixed process is difficult to adapt to different hardware platforms and resource constraints, restricting the adaptation efficiency of dynamic collection algorithms. Summary of the Invention

[0004] The present application provides a method and device for processing map data collection tasks to improve the flexibility and adaptation ability of map data collection algorithm development, while reducing the maintenance cost and improving the development efficiency.

[0005] The present application provides the following solutions:

[0006] According to a first aspect, a method for processing map data collection tasks is provided, and the method includes:

[0007] Obtain the requirement data of the map data collection task, where the requirement data includes collection requirements and hardware performance constraint conditions of the nodes to be deployed;

[0008] Input the requirement data into a requirement analysis model to generate a target computation graph that meets the collection requirements and the hardware performance constraint conditions; wherein, the target computation graph includes a plurality of target computation nodes and directed edges, each target computation node corresponds to a processing unit in the map data collection algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computation nodes it connects;

[0009] Encapsulate the target computation graph into an executable software product for deployment to the node to be deployed.

[0010] According to a second aspect, there is provided a processing apparatus for a map data collection task, the apparatus comprising:

[0011] A requirement data acquisition unit configured to acquire requirement data for a map data collection task, the requirement data including collection requirements and hardware performance constraint conditions of the node to be deployed;

[0012] A target computation graph generation unit configured to input the requirement data into a requirement analysis model to generate a target computation graph that meets the collection requirements and the hardware performance constraint conditions; wherein, the target computation graph includes a plurality of target computation nodes and directed edges, each of the target computation nodes corresponds to a processing unit in a map data collection algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computation nodes connected by it;

[0013] An encapsulation unit configured to encapsulate the target computation graph into an executable software product for deployment to the node to be deployed.

[0014] According to a third aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of the first aspects above.

[0015] According to the specific embodiments provided in the present application, the present application discloses the following technical effects:

[0016] In the embodiments of the present application, by acquiring requirement data including collection requirements and hardware performance constraint conditions, inputting the requirement data into a requirement analysis model to generate a target computation graph, and encapsulating the target computation graph into an executable software product for deployment to the node to be deployed. Compared with the prior art, this method can generate a target computation graph adapted to the node to be deployed according to different collection requirements and hardware performance constraint conditions, avoiding the problems of hard coding, re - development and deployment of code due to changes in hardware resources or updates of collection requirements in the traditional method. Through the dynamic generation and encapsulation of the target computation graph, the flexibility and adaptability of map data collection algorithm development are significantly improved, while the maintenance cost is reduced and the development efficiency is enhanced.

[0017] Of course, it is not necessary for any product implementing the present application to achieve all of the above - mentioned advantages simultaneously. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0019] Figure 1 It is a system structure diagram of a traditional map data collection algorithm;

[0020] Figure 2 It is a flowchart of a method for processing a map data collection task provided by an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of initial calculation graph generation provided by an embodiment of the present application;

[0022] Figure 4 It is a system architecture diagram of an intelligent map data collection platform provided by an embodiment of the present application;

[0023] Figure 5 It is a flowchart of the operation of an intelligent map data collection platform provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic block diagram of a device for processing a map data collection task provided by an embodiment of the present application;

[0025] Figure 7 It is a schematic block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0027] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms of "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0028] It should be understood that the term " / and / " used herein is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0029] Depending on the context, as used herein, the word "if" can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (stated condition or event)" or "in response to detecting (stated condition or event)".

[0030] As Figure 1 shown, it is a system structure diagram of a traditional map data acquisition algorithm. Among them, independent codes are developed for seven acquisition tasks (a to g) respectively. When acquisition task a needs to switch from running on the CPU (Central Processing Unit) to running on the NPU (Neural Network Processor), since the code of acquisition task a cannot be directly applied to the NPU, it is necessary to re-develop the code adapted to the NPU and conduct tests.

[0031] It can be seen from the development method of the above traditional map data acquisition algorithm that its deployment process is fixed, and steps such as pre-processing, model inference, and post-processing are hard-coded in the program. Each model has an independent and fixed processing process. Once the acquisition requirements change or the model is updated, it is necessary to manually modify the code, recompile, and deploy. This method has poor flexibility. The development process is divided according to the model. When the acquisition requirements change and cause changes in the model functions, a large number of different models and processes need to be maintained, and the common computing units cannot be efficiently reused, resulting in code redundancy and increasing the maintenance and expansion costs. At the same time, the maintenance cost is high. Every time a new model or an existing model is updated, the code needs to be modified, increasing the development and maintenance workload and reducing the development efficiency. In addition, the efficiency is low. The fixed process is difficult to adapt to different hardware platforms and resource constraints, and cannot be flexibly adjusted to cope with the differences in device performance and computing power, restricting the adaptation efficiency of the dynamic acquisition algorithm.

[0032] In view of this, the present application provides a new idea and provides a method for processing map data acquisition tasks. Figure 2 It is a flowchart of the method for processing map data acquisition tasks provided by an embodiment of the present application. The execution subject of this method can be a processing device for map data acquisition tasks. This device can be set in any computer device with data storage and processing capabilities, can be set on the server side, or can be set in a computer terminal with strong data processing capabilities. This method may include the following steps:

[0033] Step 201: Obtain the requirement data of the map data acquisition task, where the requirement data includes acquisition requirements and hardware performance constraint conditions of the nodes to be deployed.

[0034] Step 202: Input the requirement data into the requirement analysis model to generate a target computation graph that meets the acquisition requirements and hardware performance constraints; wherein, the target computation graph includes multiple target computation nodes and a directed graph, each target computation node corresponds to a processing unit in the map data acquisition algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computation nodes it connects.

[0035] Step 203: Package the target computation graph into an executable software product, and the executable software product is used to be deployed to the nodes to be deployed.

[0036] It can be seen that in the embodiment of the present application, by obtaining the requirement data including the acquisition requirements and hardware performance constraints, inputting it into the requirement analysis model to generate a target computation graph, and packaging the target computation graph into an executable software product for deployment to the nodes to be deployed. Compared with the prior art, this method can generate a target computation graph adapted to the nodes to be deployed according to different acquisition requirements and hardware performance constraints, avoiding the problems of hard coding, re - development and deployment of code due to changes in hardware resources or updates of acquisition requirements in the traditional method. Through the dynamic generation and packaging of the target computation graph, the present application can significantly improve the flexibility and adaptation ability of the map data acquisition algorithm development, while reducing the maintenance cost and improving the development efficiency.

[0037] The following describes in detail the above - mentioned step 201, that is, "obtain the requirement data of the map data acquisition task" in combination with the embodiments.

[0038] In the embodiment of the present application, the requirement data of the map data acquisition task includes acquisition requirements and hardware performance constraints of the nodes to be deployed.

[0039] Among them, the acquisition requirements refer to the specific requirements put forward for the map data acquisition task according to the specific application scenarios and task objectives, and these specific requirements include but are not limited to:

[0040] Data type: The type of map data to be acquired, such as road information, traffic flow, road construction status, weather impact, etc.

[0041] Acquisition range: Specify the geographical area, such as a specific city, road section or area range.

[0042] Acquisition frequency: The time interval of data acquisition, such as real - time acquisition, hourly acquisition or daily acquisition.

[0043] Data accuracy: The accuracy requirement for the acquired data, such as image resolution, positioning accuracy, etc.

[0044] The hardware performance constraints refer to the hardware resource limitations of the nodes to be deployed, and these constraints directly affect the execution efficiency and feasibility of the map data acquisition task.

[0045] Among them, the node to be deployed refers to the device that will deploy the map data collection algorithm and execute the map data collection task. It can be a terminal device, such as a smart phone, a tablet computer, an in-vehicle computer, etc.; it can also be a server-side device, such as a cloud server, an edge server, etc.; or other devices with data processing capabilities, such as drones, robots, etc. These devices can be used for map data collection tasks in specific scenarios.

[0046] The hardware performance constraint conditions can include but are not limited to:

[0047] The memory requirement of the node to be deployed: It refers to the available memory capacity of the node to be deployed. In the map data collection task, the memory is generally used to store the collected data and the temporary data during the operation of the map data collection algorithm. If the memory is insufficient, it may cause data loss or the interruption of the algorithm operation.

[0048] The CPU requirement of the node to be deployed: It refers to the computing power of the CPU of the node to be deployed, which can usually be represented by parameters such as the number of cores and the main frequency. For complex map data processing algorithms, usually higher CPU computing power is required to ensure real-time performance. The more cores there are, the more tasks can be processed simultaneously, and the higher the main frequency, the faster the processing speed of each core.

[0049] The GPU (Graphics Processing Unit) requirement of the node to be deployed: It refers to the graphics processing ability of the GPU of the node to be deployed, which can usually be represented by parameters such as the video memory capacity and the number of CUDA (Compute Unified Device Architecture) cores.

[0050] The NPU requirement of the node to be deployed: It refers to the inference ability of the NPU of the node to be deployed and the neural network framework it supports. Among them, the inference ability can usually be measured by FPS (Frames Per Second) or TOPS (Tera Operations Per Second).

[0051] Through the above-mentioned hardware performance constraint conditions, the generation and adjustment of the target computation graph can be carried out more precisely according to the hardware resource situation of the node to be deployed. Such refined constraint conditions can better adapt to the characteristics of different nodes to be deployed and improve the adaptability and performance of the algorithm. For example, for devices with different GPUs or NPUs, a target computation graph adapted to their specific hardware resources can be generated to make full use of the hardware acceleration ability and improve the operation efficiency and real-time performance of the algorithm.

[0052] In addition to the above-mentioned hardware performance constraints, it may also include the network bandwidth requirements, network latency requirements, network stability requirements, etc. of the nodes to be deployed. This application does not limit this.

[0053] The following describes in detail the above step 202, that is, "input the requirement data into the requirement analysis model to generate a target computation graph that meets the acquisition requirements and hardware performance constraints", in combination with embodiments.

[0054] In the embodiments of this application, the target computation graph includes multiple target computation nodes and directed edges. Each target computation node corresponds to a processing unit in the map data acquisition algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computation nodes it connects. That is to say, in the embodiments of this application, the implementation process of the map data acquisition algorithm is abstracted into multiple independent target computation nodes, and the data flow direction between the processing units corresponding to two target computation nodes (that is, the data input and output direction between the processing units corresponding to two target computation nodes) is abstracted into a directed edge. Based on these target computation nodes and directed edges, a target computation graph is constructed.

[0055] As a feasible way, the target computation nodes can communicate through shared memory, so as to achieve efficient data transmission and sharing.

[0056] Among them, the types of target computation nodes can include but are not limited to:

[0057] Pre-processing node: Responsible for pre-processing the input data, such as data format conversion, data cleaning, data augmentation, etc. For example, in the map data acquisition task, the pre-processing node can perform operations such as scaling, cropping, and normalization on the acquired images.

[0058] Model inference node: Responsible for running the models in the map data acquisition algorithm, such as convolutional neural network (CNN), recurrent neural network (RNN), etc. These models are used to extract features and make predictions from the input data.

[0059] Post-processing node: Responsible for post-processing the results of model inference, such as result filtering, result fusion, result visualization, etc.

[0060] This clear type division of target computation nodes can help improve the readability and maintainability of the target computation graph.

[0061] In addition, in the embodiments of this application, before inputting the requirement data into the requirement analysis model, the acquired requirement data can be pre-processed first to convert it into a format that the requirement analysis model can process.

[0062] Among them, the pre-processing operations can include but are not limited to:

[0063] Data standardization: Convert the acquisition requirements and hardware performance constraints into a unified data format, such as JSON or XML format. For example, the acquisition requirements may include fields such as data type, acquisition range, acquisition frequency, etc., while the hardware performance constraints may include fields such as memory requirements, CPU requirements, GPU requirements, etc.

[0064] Data verification: Verify the requirement data to ensure the integrity and consistency of the data. For example, check whether the data types in the acquisition requirements are legal and whether the hardware performance constraints meet the basic requirements.

[0065] Data input: Input the requirement data into the requirement analysis model through methods such as API interfaces, file transfer, or message queues. The requirement analysis model will further analyze and process the requirement data to generate a target computation graph.

[0066] Among them, the generation process of the target computation graph can be carried out in the following way:

[0067] First, input the requirement data into the requirement analysis model to generate an initial computation graph, which includes multiple initial computation nodes. Specifically, multiple basic computation graphs can be pre-constructed using prompt engineering, and each basic computation graph corresponds to a basic requirement data. These basic computation graphs and their corresponding basic requirement data can be used as basic contexts to provide a reference standard for the requirement analysis model. After inputting the requirement data into the requirement analysis model, the requirement analysis model parses the input requirement data and generates an initial computation graph by referring to the pre-constructed basic contexts.

[0068] It should be noted that the initial computation nodes in this initial computation graph also abstract the implementation process of the map data acquisition algorithm into multiple independent initial computation nodes. Each initial computation node corresponds to a processing unit in the map data acquisition algorithm, and two adjacent initial computation nodes are connected by a directed edge, which indicates the data flow direction between the processing units corresponding to the two initial computation nodes it connects.

[0069] As an example, such as Figure 3As shown in the figure, it is a schematic diagram of the generation of the initial computational graph. Among them, the demand data includes the newly collected demand E, the new memory demand E, and the new CPU demand E. The pre-constructed basic computational graphs include computational graph A, computational graph B, computational graph C, and computational graph D. Computational graph A corresponds to the collection demand A, the memory demand A, and the CPU demand A. Computational graph B corresponds to the collection demand B, the memory demand B, and the CPU demand B. Computational graph C corresponds to the collection demand C, the memory demand C, and the CPU demand C. Computational graph D corresponds to the collection demand D, the memory demand D, and the CPU demand D. When the new collection demand E, the new memory demand E, and the new CPU demand E are input into the demand analysis model, the demand analysis model parses them and generates the initial computational graph, that is, computational graph E.

[0070] Secondly, run the initial computational graph to obtain the running result of the initial computational graph.

[0071] Specifically, after obtaining the initial computational graph, first load the initial computational graph, for example, load the initial computational graph into the memory, then parse the dependency relationships of the initial computational nodes in the initial computational graph to determine the execution order of each initial computational node. After that, allocate computing resources to each initial node, and finally, according to the execution order of each initial node, schedule the corresponding computing resources to execute the initial computational nodes in sequence, so as to obtain the running result.

[0072] This process can ensure the correct operation of the initial computational graph and the reasonable allocation of computing resources. Through reasonable scheduling and resource allocation, the operation efficiency and stability of the initial computational graph are improved.

[0073] It should be noted that one or more operators are configured in the initial computational nodes. After the initial computational nodes are allocated computing resources, they will call the underlying operators to perform actual operations, so as to obtain the operation results. The operation results will be used as the input of the next initial computational node. When all the initial computational nodes have been executed, the finally obtained operation result is the running result.

[0074] Among them, the running result of the initial computational graph can include but is not limited to the actual task output data and the hardware performance data. Here, the actual task output data refers to the data related to the collection demand generated during the operation of the initial computational graph, and the hardware performance data refers to the usage of hardware resources during the operation of the initial computational graph. Still taking Figure 3 as an example, the hardware performance data includes the CPU usage rate and the memory usage rate.

[0075] Next, determine whether the running result of the initial computational graph meets the above-mentioned acquisition requirements and hardware performance constraint conditions. Specifically, a task analysis policy library and a performance analysis policy library are pre-constructed. Based on the task analysis policy library, determine whether the actual task output data meets the acquisition requirements. When the actual task output data does not meet the acquisition requirements, it is determined that the running result does not meet the acquisition requirements; and based on the performance analysis policy library, determine whether the hardware performance data meets the hardware constraint conditions. When the hardware performance data does not meet the hardware performance constraint conditions, it is determined that the running result does not meet the hardware performance constraint conditions.

[0076] Finally, when the running result does not meet the acquisition requirements and / or does not meet the hardware performance constraint conditions, adjust the initial computational graph until the running result of the adjusted computational graph meets the acquisition requirements and the hardware performance constraint conditions. At this time, the adjusted computational graph is determined as the target computational graph.

[0077] In the embodiment of the present application, after adjusting the initial computational graph to obtain a new computational graph, run the new computational graph, and then evaluate the running result of the new computational graph. If the evaluation still fails, repeat the steps of "adjust - run - evaluate". That is to say, when the running result does not meet the acquisition requirements and / or the hardware performance constraint conditions, the process of adjusting the initial computational graph is an iterative optimization process, and the finally obtained computational graph needs to meet the acquisition requirements and the hardware performance constraint conditions.

[0078] Through this iterative optimization method, the embodiment of the present application can ensure that the finally generated target computational graph not only meets the acquisition requirements but also conforms to the hardware performance constraint conditions. This process avoids the repeated development and testing that may be caused by the computational graph generated at one time not meeting the requirements, improves the development efficiency and accuracy of the map data acquisition algorithm, and reduces the development cost and time waste caused by demand mismatch.

[0079] Furthermore, in the embodiment of the present application, based on the pre-constructed task analysis policy library and performance analysis policy library, determine whether the actual task output data and the hardware performance data meet the acquisition requirements and the hardware performance constraint conditions respectively. This method utilizes the pre-constructed policy library and can more accurately and efficiently evaluate whether the running result meets the requirements. Compared with the traditional manual evaluation method, it reduces the subjective error in the evaluation process, improves the accuracy and consistency of the evaluation. At the same time, through clear judgment criteria, the problem can be quickly located, which is convenient for targeted adjustment of the computational graph, further improving the efficiency and quality of algorithm development.

[0080] Among them, when adjusting the initial computational graph, the structure of the initial computational graph can be optimized, including but not limited to adding initial computational nodes in the initial computational graph, reducing initial computational nodes in the initial computational graph, adjusting the execution order of each initial computational node, and so on.

[0081] For example, when more complex data cleaning or enhancement operations need to be performed on data during the data preprocessing stage, corresponding preprocessing nodes can be added; if certain nodes are proven to be redundant or have little impact on the final result during actual operation, these nodes can be considered removed to simplify the computational graph and improve execution efficiency; parallelize some independent computational nodes, or rearrange the execution order of nodes according to data dependencies to reduce waiting time and improve resource utilization.

[0082] In addition to optimizing the structure of the initial computational graph, the parameters and resource allocation strategies of each initial computational node in the initial computational graph can also be optimized.

[0083] Specifically, adjust the algorithm parameters used in the initial computational nodes, such as adjusting the model parameters in the model inference node to improve the accuracy and efficiency of inference; optimize the parameters in the preprocessing nodes and postprocessing nodes, such as image scaling ratio, data normalization parameters, etc., to improve the efficiency and quality of data processing. The resource allocation strategies of each initial computational node can also be adjusted according to the computational complexity of each initial computational node and the hardware performance constraints, such as priorities or resource usage limit conditions, etc.

[0084] This flexible adjustment method can dynamically optimize the initial computational graph according to the actual operation results and requirements. These adjustment methods work together to improve the flexibility and adaptability of the algorithm and reduce the development cost caused by demand changes.

[0085] The following describes in detail step 203 above, that is, "encapsulating the target computational graph into an executable software product, and the executable software product is used to be deployed to the nodes to be deployed".

[0086] After obtaining the target computational graph, the target computational graph can be further encapsulated into an executable software product and deployed to the nodes to be deployed.

[0087] As an implementable method, the executable software product can be transmitted to the nodes to be deployed through a hot update mechanism, so that the nodes to be deployed can load the map data collection algorithm based on the target computational graph or upgrade the original map data collection algorithm.

[0088] This hot update mechanism can achieve loading and upgrading without modifying the code when adding or updating the map data collection algorithm, thereby reducing the upgrade cost and bandwidth consumption. Moreover, it can quickly respond to changes in the collection requirements, reduce the downtime and task interruption risks caused by updates, and improve the stability of the system and the user experience.

[0089] In practical applications, the method for processing map data collection tasks provided by the embodiments of this application can be used in an intelligent map data collection platform.

[0090] As an example, as Figure 4 shown, it is the system architecture diagram of the intelligent map data collection platform. The intelligent map data collection platform can be divided into two parts: a high-performance computing system and a large model analysis system. Among them, the high-performance computing system serves as the foundation, responsible for specific computing logics and providing unified computing capabilities to the upper layer. It can further include a scheduling module, a computing module, and a tool module. The scheduling module is mainly used to schedule and manage computing resources. The computing module is mainly responsible for underlying computing tasks. The tool module provides various analysis tools for tracking and analyzing algorithms and task processes. The large model analysis system serves as the upper layer and automatically generates executable software products based on the demand data of map data collection tasks, including large models. For example, the demand analysis model mentioned earlier is a type of large model.

[0091] Next, refer to Figure 4 and Figure 5 , and explain the operating principle of the intelligent map data collection platform. As Figure 5 shown, it is the operation flowchart of the intelligent map data collection platform. It specifically includes the following steps:

[0092] Step 501, the large model analysis system obtains the demand data of the map data collection task, and the demand data includes the collection requirements and the CPU requirements and memory requirements of the nodes to be deployed.

[0093] Step 502, the large model analysis system inputs the collection requirements, CPU requirements, and memory requirements into the large model of the large model analysis system. The large model parses the input demand data and generates an initial computation graph based on the pre-constructed basic computation graph.

[0094] Step 503, call the high-performance computing system to run the initial computation graph to obtain the operation result.

[0095] The operation process includes:

[0096] First, load the initial computation graph, parse the dependency relationships of the initial computation nodes in the initial computation graph, and determine the execution order of each initial computation node.

[0097] Secondly, allocate computing resources dynamically for each initial computing node.

[0098] Finally, according to the execution order of each initial computing node, schedule the corresponding computing resources to execute each initial computing node in sequence, and obtain the operation result, where the operation result includes actual task output data, CPU usage rate, and memory usage rate.

[0099] Step 504: The large model analysis system respectively determines whether the actual task output data meets the collection requirements, whether the CPU usage rate meets the CPU requirements, and whether the memory usage rate meets the memory requirements based on the pre-constructed analysis strategy library. When all the judgment results are satisfied, step 507 is executed; otherwise, step 505 is continued.

[0100] Step 505: The large model analysis system uses the large model to analyze the input requirement data and the previous operation result, generates an adjustment strategy, and adjusts the previously generated computation graph based on this adjustment strategy to obtain an adjusted computation graph.

[0101] Step 506: Call the high-performance computing system to run the adjusted computation graph to obtain the operation result. Then step 504 is executed.

[0102] The operation process is the same as that in step 503 and will not be elaborated here.

[0103] Step 507: Package the computation graph into an executable software product. The process ends.

[0104] In addition, it should be noted that when calling the high-performance computing system to execute the computation graph (here, the initial computation graph or the adjusted computation graph is collectively referred to as the computation graph), it mainly relies on the cooperation of the scheduling module, computing module, and tool module in the high-performance computing system.

[0105] The scheduling module is responsible for loading the computation graph into the memory, parsing the dependency relationships of each computing node in the computation graph, determining the execution order of each computing node, then dynamically allocating computing resources (such as CPU, GPU, NPU, etc.) for each initial computing node according to the structure of the computation graph and the execution requirements of the nodes, and finally scheduling the corresponding computing resources to execute each initial computing node in the determined execution order.

[0106] The computing module is mainly responsible for executing the actual computing tasks of each computing node, such as data processing, model inference, etc., and generating operation results such as actual task output data, CPU usage rate, and memory usage rate during the execution process.

[0107] The tool model can provide tools to monitor the execution process of the computation graph, record the running results (such as actual task output data, CPU usage rate, and memory usage rate), and perform performance analysis on the running results to provide data support for subsequent optimization.

[0108] Among them, the computing module, as the actual execution unit of the computation, mainly includes two units, namely PUFF_INFERENCE and PUFF_COMPUTE. Each upper-layer abstract computation expression will be mapped to specific instructions in the computing module. The main functional division of the two sub-modules is as follows:

[0109] PUFF_INFERENCE inference unit: Manage the running of different inference frameworks, including MNN, NNIE, etc.

[0110] PUFF_COMPUTE computing unit: Implement operators related to pre-processing and post-processing processes, and accelerate specific operators using assembly instruction sets (including NEON, SSE, AVX).

[0111] The PUFF_INFERENCE inference unit can be further composed of the following parts:

[0112] Unified inference Interface: Provide a unified inference method for the cpu and npu to achieve flexible switching of inference.

[0113] Inference configuration management module: Distribute and configure the inference configuration information parsed from the computation graph.

[0114] Inference resource management module: Unified management of cpu and npu memory.

[0115] Inference process management module: Inference initialization, inference process startup, and inference release.

[0116] Adapter: Adapters for each inference engine, including inferences such as mnn and Hisilicon NNIE.

[0117] The PUFF_COMPUTE computing unit, also as the execution layer of the computation graph, provides the required operators in the pre-processing and post-processing processes of the entire computation graph. It is composed of the following parts:

[0118] Operator parallel accelerator: Provide unified thread pool resource management for operator parallelism.

[0119] Memory allocator: Manage memory resources of different back-end cpus and gpus for operators to apply for and release.

[0120] Operator registrar: Manage underlying registered operators using an automated registration mechanism.

[0121] Operators: Optimize and accelerate CV, pre - processing operators, and post - processing operators using the arm / sse / avx instruction sets to provide high - performance operator support.

[0122] The tool module may further include, but is not limited to:

[0123] Business analysis tool: Quickly process algorithm and business logs, providing fast filtering and analysis functions in three dimensions: time, module, and content.

[0124] Policy library: Establish an analysis policy library for business analysis and performance analysis to achieve automated and intelligent analysis.

[0125] Batch analysis tool: Automatically analyze batch policies for batch operations.

[0126] Computational graph management tool: Manage the computational graph, visually edit and save the computational graph to improve the efficiency of building business computational graphs. For example, a computational graph editor can be used to load the computational graph, visualize the grid structure of the computational graph, edit the attribute information of specified computational nodes, and save and update it with one click.

[0127] Performance tracking and analysis tool: Automatically track and analyze the performance of terminal intelligent devices.

[0128] PuffCopilot: Based on large - model and local RAG solutions, conduct Q&A and analysis for the proprietary operations in the road dynamic data collection scenario.

[0129] In addition, the computational graph generated in the embodiments of this application abstractly represents concepts such as overall computational nodes, edges, and graphs based on Python, flexibly constructs and saves the computational graph through Python interfaces, and finally can achieve the visualization of the computational graph.

[0130] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] According to an embodiment of another aspect, a processing device for map data collection tasks is provided. Figure 6 The schematic block diagram of a processing device for map data collection tasks according to an embodiment is shown, as Figure 6 shown, the device 600 includes:

[0132] A requirement data acquisition unit 601, configured to acquire requirement data for a map data collection task, where the requirement data includes collection requirements and hardware performance constraint conditions of nodes to be deployed;

[0133] A target computation graph generation unit 602, configured to input the requirement data into a requirement analysis model to generate a target computation graph that meets the collection requirements and the hardware performance constraint conditions; wherein, the target computation graph includes a plurality of target computation nodes and directed edges, each target computation node corresponds to a processing unit in a map data collection algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computation nodes it connects;

[0134] An encapsulation unit 603, configured to encapsulate the target computation graph into an executable software product, and the executable software product is used to be deployed to the nodes to be deployed.

[0135] Optionally, when the target computation graph generation unit 602 executes inputting the requirement data into the requirement analysis model to generate a target computation graph, it is configured to:

[0136] Input the requirement data into the requirement analysis model to generate an initial computation graph, where the initial computation graph includes a plurality of initial computation nodes;

[0137] Run the initial computation graph to obtain the running result of the initial computation graph;

[0138] Determine whether the running result meets the collection requirements and the hardware performance constraint conditions;

[0139] When the running result does not meet the collection requirements and / or does not meet the hardware performance constraint conditions, adjust the initial computation graph until the running result of the adjusted computation graph meets the collection requirements and the hardware performance constraint conditions;

[0140] Determine the adjusted computation graph as the target computation graph.

[0141] Further, the running result includes actual task output data and hardware performance data;

[0142] When the target computation graph generation unit 602 executes determining whether the running result meets the collection requirements and the hardware performance constraint conditions, it is configured to:

[0143] Based on a pre-constructed task analysis strategy library, determine whether the actual task output data meets the collection requirements;

[0144] Based on a pre-constructed performance analysis strategy library, determine whether the hardware performance data meets the hardware performance constraint conditions;

[0145] When the actual task output data does not meet the acquisition requirements, it is determined that the operation result does not meet the acquisition requirements, and when the hardware performance data does not meet the hardware performance constraint conditions, it is determined that the operation result does not meet the hardware performance constraint conditions.

[0146] Further, when the target computation graph generation unit 602 executes the initial computation graph to obtain the operation result of the initial computation graph, it is configured to:

[0147] Load the initial computation graph;

[0148] Analyze the dependency relationships of the initial computation nodes in the initial computation graph to determine the execution order of each initial computation node;

[0149] Allocate computing resources to each initial computation node;

[0150] Schedule the corresponding computing resources to execute the initial computation nodes in sequence according to the execution order of each initial computation node to obtain the operation result.

[0151] Further, the adjustment of the initial computation graph includes at least one of the following:

[0152] Add initial computation nodes to the initial computation graph;

[0153] Reduce the initial computation nodes in the initial computation graph;

[0154] Adjust the execution order of each initial computation node;

[0155] Adjust the parameters of the initial computation nodes;

[0156] Adjust the resource allocation strategy of the initial computation nodes.

[0157] Further, the hardware performance constraint conditions include at least one of the following:

[0158] The memory requirements of the node to be deployed;

[0159] The central processing unit (CPU) requirements of the node to be deployed;

[0160] The graphics processing unit (GPU) requirements of the node to be deployed;

[0161] The neural network processing unit (NPU) requirements of the node to be deployed.

[0162] Further, the types of the target computation nodes include at least one of the following: pre-processing nodes, model inference nodes, and post-processing nodes.

[0163] Further, the device 600 may further include:

[0164] A transmission unit 604, configured to transmit the executable software product to the node to be deployed through a hot update mechanism.

[0165] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for a system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for the user to select to authorize or reject.

[0167] In addition, an embodiment of this application also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the method described in any one of the foregoing method embodiments.

[0168] And an electronic device, including:

[0169] One or more processors; and

[0170] A memory associated with the one or more processors, where the memory is used to store program instructions. When the program instructions are read and executed by the one or more processors, they execute the steps of the method described in any one of the foregoing method embodiments.

[0171] This application also provides a computer program product, including a computer program, which implements the steps of the method described in any one of the foregoing method embodiments when executed by a processor.

[0172] Wherein, Figure 7Exemplarily shows the architecture of an electronic device, which may specifically include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The above-mentioned processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and the memory 720 can be communicatively connected through a communication bus 730.

[0173] Among them, the processor 710 can be implemented in ways such as a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in this application.

[0174] The memory 720 can be implemented in forms such as ROM (Read Only Memory), RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input / output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. In addition, it can also store a web browser 723, a data storage management system 724, a processing device 600 for map data collection tasks, and so on. The above-mentioned processing 600 of map data collection tasks can be the application program that specifically implements the operations of the foregoing steps in the embodiments of this application. In short, when implementing the technical solutions provided in this application through software or firmware, the relevant program codes are stored in the memory 720 and are called and executed by the processor 710.

[0175] The input / output interface 713 is used to connect to an input / output module to achieve information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0176] The network interface 714 is used to connect to a communication module (not shown in the figure) to achieve communication interaction between this device and other devices. Among them, the communication module can achieve communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.).

[0177] The bus 730 includes a path for transmitting information between various components of the device, such as the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720.

[0178] It should be noted that although the above device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the present application, and do not necessarily include all the components shown in the figure.

[0179] From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0180] The above has introduced the technical solution provided by the present application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for processing a map data collection task, characterized in that, The method includes: Obtaining requirement data for a map data collection task, where the requirement data includes collection requirements and hardware performance constraint conditions of nodes to be deployed; Inputting the requirement data into a requirement analysis model to generate a target computation graph that meets the collection requirements and the hardware performance constraint conditions; wherein, the target computation graph includes multiple target computation nodes and directed edges, each of the target computation nodes corresponds to a processing unit in a map data collection algorithm, and the directed edges indicate the data flow direction between the processing units corresponding to the two target computation nodes connected by them; Encapsulating the target computation graph into an executable software product, and the executable software product is used to be deployed into the nodes to be deployed.

2. The method according to claim 1, characterized in that The inputting the requirement data into the requirement analysis model to generate a target computation graph includes: Inputting the requirement data into the requirement analysis model to generate an initial computation graph, and the initial computation graph includes multiple initial computation nodes; Running the initial computation graph to obtain the running result of the initial computation graph; Judging whether the running result meets the collection requirements and the hardware performance constraint conditions; When the running result does not meet the collection requirements and / or does not meet the hardware performance constraint conditions, adjusting the initial computation graph until the running result of the adjusted computation graph meets the collection requirements and the hardware performance constraint conditions; Determining the adjusted computation graph as the target computation graph.

3. The method according to claim 2, wherein The running result includes actual task output data and hardware performance data; the judging whether the running result meets the collection requirements and the hardware performance constraint conditions includes: Based on a pre-constructed task analysis strategy library, judging whether the actual task output data meets the collection requirements; Based on a pre-constructed performance analysis strategy library, judging whether the hardware performance data meets the hardware performance constraint conditions; When the actual task output data does not meet the collection requirements, determining that the running result does not meet the collection requirements, and when the hardware performance data does not meet the hardware performance constraint conditions, determining that the running result does not meet the hardware performance constraint conditions.

4. The method according to claim 2, wherein The running the initial computation graph to obtain the running result of the initial computation graph includes: Loading the initial computation graph; Analyzing the dependency relationships of the initial computation nodes in the initial computation graph to determine the execution order of each of the initial computation nodes; Allocating computing resources to each of the initial computation nodes; According to the execution order of each of the initial computation nodes, scheduling the corresponding computing resources to sequentially execute the initial computation nodes to obtain the running result.

5. The method according to claim 2, characterized in that, The adjusting the initial computation graph includes at least one of the following: Adding initial computation nodes in the initial computation graph; Reducing initial computation nodes in the initial computation graph; Adjusting the execution order of each of the initial computation nodes; Adjusting the parameters of the initial computation nodes; Adjusting the resource allocation strategy of the initial computation nodes.

6. The method according to claim 1, wherein The hardware performance constraint conditions include at least one of the following: The memory requirement of the node to be deployed; The central processing unit (CPU) requirement of the node to be deployed; The graphics processing unit (GPU) requirements of the node to be deployed; The neural network processing unit (NPU) requirements of the node to be deployed.

7. The method according to claim 1, characterized in that, The type of the target computing node includes at least one of the following: a pre-processing node, a model inference node, and a post-processing node.

8. The method according to any one of claims 1 to 7, characterized in that The method further includes: Transmitting the executable software product to the node to be deployed through a hot update mechanism.

9. A processing device for a map data collection task, characterized in that, The device includes: A requirement data acquisition unit, configured to acquire requirement data for a map data acquisition task, where the requirement data includes acquisition requirements and hardware performance constraint conditions of the node to be deployed; A target computing graph generation unit, configured to input the requirement data into a requirement analysis model to generate a target computing graph that meets the acquisition requirements and the hardware performance constraint conditions; where the target computing graph includes a plurality of target computing nodes and directed edges, each target computing node corresponds to a processing unit in the map data acquisition algorithm, and the directed edge indicates the data flow direction between the processing units corresponding to the two target computing nodes connected by it; An encapsulation unit, configured to encapsulate the target computing graph into an executable software product for deployment to the node to be deployed.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images