Map data compiling method and device, computer device and storage medium

Abstract

The application discloses a map data compiling method and device, computer equipment and a storage medium, and belongs to the technical field of maps. According to the compiling task of the map service request, the system is dynamically and timely allocated with a computing device according to a system load request, and the computing device is triggered to start the compiling task through a remote execution command. Since the computing device is mounted in a cloud disk, the map data in the cloud disk can be read and written to realize remote compiling. Since the computing device bound to each compiling task can read and write the map data in the cloud disk by being mounted in the cloud disk, data copying between the physical machines of the upstream and downstream tasks is avoided in the case that the compiling task is bound to the physical machine, the compiling process of the map data is simplified, and the compiling efficiency of the map data is greatly improved.

Description

Technical Field

[0001] This application relates to the field of map technology, and in particular to a method, apparatus, computer device, and storage medium for compiling map data. Background Technology

[0002] With the development of map technology and the advancement of vehicle networking technology, more and more products and services need to request the compilation results of map data. Therefore, the compilation efficiency of map data is an important foundation for the ability of various application map services to provide services to users. Map data has the characteristics of large data volume, high data change frequency, multiple and complex data processing processes, large consumption of storage and computing resources, high dependence on software environment, difficulty in migration, and high requirements for compilation efficiency.

[0003] Currently, in the process of compiling map data, each compilation task is usually assigned to a fixed physical machine for execution, and the software environment required for the compilation task is pre-configured on the physical machine. However, since there are often upstream and downstream dependencies between different compilation tasks, such as compilation task B depending on the data generated by the previous compilation task A, this requires data to be transferred between the physical machine bound to compilation task A and the physical machine bound to compilation task B, resulting in a cumbersome and lengthy compilation process and low compilation efficiency for map data. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and storage medium for compiling map data, which can simplify the map data compilation process and improve the compilation efficiency. The technical solution is as follows:

[0005] On the one hand, a method for compiling map data is provided, which includes:

[0006] In response to a map service request, obtain at least one compilation task associated with the map service request;

[0007] For any one of the at least one compilation task, a computing device is allocated to the compilation task for processing the compilation task;

[0008] Send a remote execution command for the compilation task to the computing device, the remote execution command being used to instruct the computing device to execute the compilation task using map data stored in the cloud disk;

[0009] Return the compilation result obtained by at least one computing device based on the remote execution command.

[0010] On the one hand, a map data compilation apparatus is provided, the apparatus comprising:

[0011] The acquisition module is used to acquire at least one compilation task associated with the map service request in response to the map service request;

[0012] An allocation module is configured to allocate a computing device for processing the compilation task to any one of the at least one compilation tasks.

[0013] The sending module is used to send a remote execution command for the compilation task to the computing device, wherein the remote execution command is used to instruct the computing device to execute the compilation task using map data stored in the cloud disk;

[0014] The return module is used to return the compilation result obtained by at least one computing device based on the remote execution command.

[0015] In some embodiments, the allocation module includes:

[0016] The determining unit is used to determine the computing resources required to process the compilation task based on the compilation information table, wherein the compilation information table is used to record the computing resources required by multiple compilation tasks.

[0017] The determining unit is also used to determine the idle resources of each of the multiple computing devices in the computing cluster based on the device information table;

[0018] An allocation unit is used to allocate computing devices with more free resources than computing resources to the compilation task.

[0019] In some embodiments, where the compilation task is associated with a specific device, the allocation unit is used to:

[0020] If the idle resources of the specific device are greater than the computing resources, the compilation task will be assigned to the specific device.

[0021] If the idle resources of the specific device are less than or equal to the computing resources, the compilation task is put into a waiting state.

[0022] In some embodiments, where the compilation task is not associated with any specific device, the allocation unit includes:

[0023] The first allocation subunit is used to allocate the non-specific device that meets the first resource condition to the compilation task;

[0024] The second allocation subunit is used to allocate a specific device that meets the second resource condition to the compilation task when there is no non-specific device that meets the first resource condition in the computing cluster.

[0025] The waiting subunit is used to cause the compilation task to enter a waiting state if no specific device meeting the second resource condition exists in the computing cluster.

[0026] In some embodiments, the first allocation subunit is used to:

[0027] If there are idle non-specific devices in the computing cluster, the idle non-specific devices shall be allocated to the compilation task;

[0028] If no idle non-specific devices exist in the computing cluster, the compilation task is allocated a non-specific device with idle resources greater than the computing resources.

[0029] In some embodiments, the second allocation subunit is used for:

[0030] Allocate a specific device that is idle and associated with a downstream task of the compilation task to the compilation task, wherein the downstream task is a task that can only be executed after the compilation task has been completed;

[0031] If there is no idle specific device in the computing cluster that is associated with a downstream task of the compilation task, allocate a specific device to the compilation task that has more idle resources than the computing resources and is associated with a downstream task of the compilation task.

[0032] In some embodiments, the allocation module further includes:

[0033] The routing unit is used to route the compilation task to a computing cluster that matches the service type of the map service request.

[0034] In some embodiments, the allocation module further includes:

[0035] The routing unit is used to route the compilation task to a computing cluster that supports splitting the compilation task into multiple subtasks, provided that the compilation task does not depend on global map data.

[0036] The routing unit is further configured to route the compilation task to a computing cluster whose computing resources meet the third resource condition when the compilation task depends on global map data.

[0037] In some embodiments, the allocation module further includes:

[0038] The routing unit is used to determine the calling interface service associated with the compilation task and to route the compilation task to the computing cluster associated with the calling interface service.

[0039] In some embodiments, the apparatus further includes:

[0040] A blocking module is used to block a target compilation task, which is a compilation task assigned to the failed computing device, in the event of a failure of any computing device.

[0041] The marking module is used to mark the malfunctioning computing device as unavailable in the device information table;

[0042] The allocation module is also used to reallocate target computing devices that are in an available state for the target compilation task.

[0043] In some embodiments, the apparatus further includes:

[0044] The module adds a first computing device to the device information table when a first computing device is added to the computing cluster.

[0045] A mounting module is used to mount the cloud disk on the first computing device;

[0046] The configuration module is used to set the environment variables that the compilation task depends on for the first computing device.

[0047] In some embodiments, the computing devices used to process any map service request compilation task are all mounted to the cloud disk, which refers to a storage cluster composed of multiple storage devices, and the storage cluster is used to store the map data required for the compilation task.

[0048] On one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the method for compiling map data as described above.

[0049] On the one hand, a storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to implement the method for compiling map data as described above.

[0050] On one hand, a computer program product is provided, comprising at least one computer program stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the at least one computer program from the computer-readable storage medium, and the one or more processors execute the at least one computer program, enabling the computer device to perform the above-described method for compiling map data.

[0051] The beneficial effects of the technical solutions provided in this application include at least the following:

[0052] By compiling map service requests, computing devices are dynamically and in real-time allocated according to the system's load requests. Compilation tasks are triggered by remotely executing commands. Since the computing devices are mounted to the cloud disk, they can read and write map data in the cloud disk to achieve remote compilation. Because each compilation task is bound to a computing device that can read and write map data in the cloud disk by being mounted to the cloud disk, data copying between the physical machines of upstream and downstream tasks is avoided when compilation tasks are bound to physical machines. This simplifies the map data compilation process and greatly improves the efficiency of map data compilation. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram illustrating the implementation environment of a map data compilation method provided in an embodiment of this application;

[0055] Figure 2 This is a flowchart of a map data compilation method provided in an embodiment of this application;

[0056] Figure 3 This is a flowchart of a map data compilation method provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram illustrating the computing device selection principle of a scheduling decision service provided in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram illustrating the principle of a scheduling decision-making strategy provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of a cloud disk data storage soft link provided in an embodiment of this application;

[0060] Figure 7 This is a schematic flowchart illustrating a method for compiling map data provided in an embodiment of this application.

[0061] Figure 8 This is a schematic diagram of the structure of a map data compilation device provided in an embodiment of this application;

[0062] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0064] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0065] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple first positions means two or more first positions.

[0066] The term "including at least one of A or B" in this application refers to the following situations: including only A, including only B, and including both A and B.

[0067] The user-related information (including but not limited to user device information, personal information, behavioral information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, when applied to specific products or technologies using the methods of the embodiments of this application, are all obtained with the user's permission, consent, authorization, or full authorization from all parties, and the collection, use, and processing of related information, data, and signals must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the map service-related data involved in this application were all obtained under full authorization.

[0068] Before introducing the embodiments of this application, it is necessary to introduce some basic concepts in the field of cloud technology.

[0069] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It encompasses network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. These technologies can form resource pools, allowing for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support in the cloud technology field. Backend services of technical network systems require substantial computing and storage resources, such as those for video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring data to be transmitted to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, all of which can be achieved through cloud computing.

[0070] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file systems to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0071] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0072] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0073] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0074] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called IaaS, or Infrastructure as a Service) platform is established. Various types of virtual resources are deployed within the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0075] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0076] Cloud-native: Cloud-native is a distributed cloud based on distributed deployment and unified operation and management. It is a cloud technology product system built on technologies such as containers, microservices, and DevOps (a combination of Development and Operations, a collective term for processes, methods, and systems). After using cloud-native technologies, developers do not need to consider the underlying technical implementation and can fully leverage the elasticity and distributed advantages of the cloud platform to achieve rapid deployment, on-demand scaling, and uninterrupted delivery. Cloud-native technologies provide reliable underlying technical guarantees for the separation of storage and compute.

[0077] The terminology used in the embodiments of this application will be explained below.

[0078] Intelligent Traffic System (ITS): Also known as Intelligent Transportation System, it effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing. It strengthens the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy. The map data compilation task involved in the embodiments of this application can be applied to the aforementioned ITS system.

[0079] Storage-compute separation: This refers to the separation of storage and computation. In business processing systems, two processes are involved: storage and computation. The storage process refers to storing business-related data that needs to be persisted, while the computation process refers to using resources such as CPU (Central Processing Unit) and memory to participate in business computation. Currently, storage and computation resources are often combined into a single machine to provide services. For example, a single physical machine provides both computation and storage resources. When expanding resources, expansion is done on a machine-by-machine basis, which leads to resource waste. Storage-compute separation technology allows storage and computation resources to be independent of each other, enabling independent management, elastic scaling, and on-demand allocation.

[0080] Map data compilation refers to the process of integrating data from various sources, processing it through multiple stages, and producing the data format required for online map services. This includes compiling data for sub-services such as base maps, navigation, public transportation, and traffic conditions. The map data compilation process is a crucial foundation for map services to provide services to users, and it has several key characteristics: extremely large data volume, high frequency of data changes, often complex and multifaceted data processing, significant consumption of storage and computing resources, high dependence on the software environment, difficulty in migration, and high requirements for compilation efficiency. These characteristics are precisely the biggest differences between map services and other services. Because the map data compilation process directly affects the choice of technical solutions and the quality of service capabilities in each stage of map services, map services can be considered a typical data- and computationally-intensive business. The data compilation process involves loading, outputting, handling network throughput, and migrating large amounts of data.

[0081] Workflow scheduling: From a computer technology perspective, a workflow is a series of steps where multiple business objects collaborate to complete a task. Arranging these steps in a specific order and according to certain rules to accomplish the task is workflow scheduling. In the process of compiling map data, workflow scheduling requires organizing complex compilation steps according to rules such as dependencies, resource consumption, software environment requirements, and execution efficiency, to automate, quickly, and efficiently compile the resulting data.

[0082] Shared storage systems: These are persistent data storage systems that can be mounted on different machines to achieve shared storage. They can provide ample storage space. Common products include Ceph (a distributed file system), TopoLVM (a new container storage interface plugin for local storage), and CBS (Cloud Backup Service).

[0083] Computing cluster: A system consisting of a group of computing nodes (i.e. computing devices) that can be scheduled together to complete computing tasks.

[0084] With the development of map technology and the advancement of vehicle networking technology, more and more products and services need to request the compilation results of map data. Therefore, the compilation efficiency of map data is an important foundation for the ability of various application map services to provide services to users. Map data has the characteristics of large data volume, high data change frequency, multiple and complex data processing processes, large consumption of storage and computing resources, high dependence on software environment, difficulty in migration, and high requirements for compilation efficiency.

[0085] Currently, the compilation process for map data typically employs a architecture combining physical machines with large cores and storage, along with a workflow engine. The main characteristic of this approach is that each compilation task is assigned to a fixed physical machine for execution, and the software environment required for the compilation tasks is pre-configured on the physical machine, minimizing changes to the physical machine's server environment. For example, the various compilation tasks (equivalent to compilation steps) for map data compilation are pre-allocated to designated physical machines based on resource consumption and software dependencies. The software environments required by each compilation task are also pre-installed on different physical machines. When the compilation process begins, the workflow engine, based on the pre-bound IP address (Internet Protocol) and path of each compilation task, remotely executes commands to initiate the execution of the compilation tasks on the physical machines.

[0086] Since the resource consumption and software dependencies of each compilation task are different, it is necessary to prepare physical machines with suitable environment and sufficient resources in advance. In this case, each physical machine has independent computing and storage resources, and there is no resource sharing between different physical machines. The overall compilation process is cumbersome and lengthy, and the compilation efficiency is low.

[0087] Furthermore, due to the frequent upstream and downstream dependencies between different compilation tasks, data needs to be copied between their respective bound physical machines. For example, compilation task B depends on data generated by the previous compilation task A. This requires data transfer between the physical machines bound to compilation task A and the physical machines bound to compilation task B. Since map services already involve a large amount of data, the copying time will naturally be long, significantly impacting the overall compilation process. In addition, the already complex compilation process becomes even more complicated and difficult to maintain due to the added data copying steps.

[0088] Furthermore, because many compilation tasks require significant resources (such as CPU or memory) or depend on specific software environments, such as GCC (GNU Compiler Collection) versions or TensorFlow, compilation tasks are bound to physical machines and can only be executed on the physical machines they are bound to. Once the bound physical machine fails (such as network outages or machine crashes), the compilation tasks bound to the failed physical machine will be suspended, and the entire compilation process will be blocked. Whether replacing the physical machine or waiting for the physical machine to recover, it takes a long time, resulting in low availability and poor stability.

[0089] Furthermore, when compilation tasks have upstream and downstream dependencies, different machines need to transfer data to each other. If the compilation process needs adjustment, such as adding or deleting a compilation task or changing dependencies, manual verification of dependencies and adjustments to data transfer steps are required. Errors in this process can lead to problems in the compilation workflow. In addition, if a machine becomes unavailable and needs to be replaced, the workflow engine configuration needs to be manually modified, and the dependent software environment needs to be redeployed on the replacement machine. After deployment, overall testing is required. These manual verification, modification, deployment, and testing tasks consume significant time and manpower, resulting in high maintenance costs.

[0090] Furthermore, when environment migration is required, such as when certain projects need to compile the same map data, it is necessary to replicate a compilation environment or set up an identical testing process. This requires investing in new machine resources, reinstalling dependent environments, and reallocating machines based on the number of machines and dependencies. Environment migration has high deployment costs, complicated and cumbersome processes, and high machine and human resource costs.

[0091] Furthermore, when adding new machines or computing resources, it is necessary to adjust the process scheduling and allocation, increase the data transmission process of upstream and downstream tasks, and install and configure the software environment. This makes the addition of machines or computing resources very inflexible. Any operation to add or remove machines or computing resources requires cumbersome work such as adjusting the process scheduling and allocation, installing the software environment, and adjusting the data copying steps, resulting in poor scalability.

[0092] Furthermore, due to the cumbersome data transmission and fixed machine allocation mechanisms, many compilation tasks that could originally be executed in parallel can only be executed serially, failing to fully utilize idle computing resources. Moreover, since physical machines are bound together with computing and storage resources, expanding computing resources will inevitably increase storage resources as well. However, computing resources are usually insufficient while storage resources are sufficient, which leads to a waste of storage resources and low resource utilization.

[0093] In view of this, this application provides a solution for building a map data compilation process or platform using cloud-native storage-compute separation technology, which can improve the execution efficiency of the map data compilation process, improve the resource utilization within the system, and enhance the scalability of the system. The details of the solution will be described in detail in the subsequent embodiments.

[0094] The storage-compute separation architecture of the embodiments of this application will be described below.

[0095] Figure 1 This is a schematic diagram illustrating the implementation environment of a map data compilation method provided in an embodiment of this application. See also... Figure 1 This illustrates a cloud-native compilation infrastructure architecture involving multiple compute clusters 110 and a shared storage cluster 120.

[0096] Each computing cluster 110 consists of a group of computing devices 111 (i.e., computing nodes), where computing devices 111 are also called cloud virtual machines (CVMs). The computing devices 111 in the computing cluster 111 can be scheduled to work together to complete computing tasks, such as various computing tasks for map services. The computing devices 111 in the computing cluster 111 do not need to all have high-configuration computing resources. Different computing resources with varying configurations can be combined on different computing devices 111 to adapt to compilation tasks with different CPU and memory requirements.

[0097] In the map data compilation process, a virtual host with appropriate CPU and memory configuration is selected based on the computing resources (such as memory resources) required by the compilation task. This is equivalent to dynamically allocating a suitable computing device 111 for each compilation task.

[0098] In some embodiments, when dividing the computing cluster 110, the computing devices 111 can be grouped into different computing clusters 110 according to different applications or projects (i.e. business types). For example, for the compilation task of map data, according to the business type to which the request belongs, it can be divided into formal compilation cluster, test compilation cluster and other project clusters.

[0099] The formal compilation cluster is mainly used to produce compilation data that can be directly deployed online. That is, the compilation results generated by the formal compilation cluster will be directly deployed to the formal map product or applied to map-based service products. These compilation results can be deployed to the product in full or incrementally.

[0100] The test compilation cluster is mainly used to test and verify new data, new tools, or new processes. However, the test results are not uploaded to the product. Instead, they are transferred to the official compilation cluster after the test is passed and then fully deployed. Since the test process does not have high requirements for compilation efficiency, the test compilation cluster can run multiple test tasks in parallel. The process engine schedules and controls the resource usage of each computing device in the test compilation cluster.

[0101] Among them, other project clusters are used to provide independent compilation environments for other applications that need to compile map data. Optionally, different projects or applications can share the same other project cluster, or different other project clusters can be configured for different projects or different applications. This application embodiment does not specifically limit this.

[0102] It should be noted that since there are no shared hosts between different computing clusters 110, the different computing clusters 110 are isolated from each other and will not cause mutual interference due to shared hosts.

[0103] In some embodiments, the deployment region of the computing cluster 110 is selected according to the geographical distribution characteristics of the servers commonly used to request map services. For example, if the business servers of map applications and navigation applications are deployed in cities A, B, and C, in order to transmit the compilation results to the business servers as quickly as possible, the computing cluster 110 used to compile map data can be deployed in a city in the middle of cities A, B, and C, such as city E or F. This allows the computing cluster 110 deployed in city E or F to transmit the compiled map data results to cities A, B, and C as quickly as possible, maximizing the transmission efficiency of the compilation results.

[0104] In some embodiments, each computing device 111 in each computing cluster 110 is mounted to a shared storage cluster 120, which is equivalent to mounting each CVM to the same CBS (Cloud Backup Service), enabling different computing devices 111 in different computing clusters 110 to mount to the shared cloud storage directory in the storage cluster 120.

[0105] Since each computing device 111 is mounted with the cloud storage directory of the storage cluster 120, data sharing in the storage cluster 120 is achieved. That is, all computing devices 111 inside the computing cluster 110 read and write to the same cloud disk. During the compilation process, there is no need to frequently transfer or copy data, and the data, tools or dependent software environment only need to be prepared once in the storage cluster 120.

[0106] The shared storage cluster 120 refers to a persistent data storage system that can be mounted on different computing devices 111 to achieve shared storage. It can provide sufficient storage space; in other words, the storage cluster 120 provides the functionality of a shared storage system. The storage cluster 120 has the software environment required for compilation tasks installed and deployed. By mounting cloud disks and setting environment variables, all computing devices 111 can remotely call and execute compilation, greatly reducing the difficulty of environment deployment and migration. Furthermore, the deployment operation is fully automated, providing a foundation for automatic expansion. This deployment operation can be considered as the initialization operation of the computing devices. When any new computing device is added to the computing cluster, it can be initialized through an automated script, enabling the rapid addition of new computing devices to the cluster for automatic expansion. The newly added computing devices can then be quickly deployed and participate in compilation tasks.

[0107] Because storage cluster 120 has a shared cloud disk with ample space, it can store multiple versions of data and tools, and can easily realize fast update or rollback operations for compilation tasks. The shared feature of the cloud disk can not only reduce the transfer of data between machines, but also take advantage of the same disk to reduce local copying of data by using soft links, so as to further improve the execution efficiency of compilation tasks and optimize the resource utilization of storage space.

[0108] The aforementioned computing cluster 110 or storage cluster 120 can be provided externally as computing servers or storage servers. The servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0109] In the storage-compute separation system architecture, computing cluster 110 has sufficient computing resources, while storage cluster 120 (also known as cloud storage or cloud disk) is dedicated to storing massive amounts of map data and / or compiled data. Computing cluster 110 and storage cluster 120 are deployed and managed separately, thus achieving storage-compute separation (i.e., storage-compute separation). This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0110] Since all computing devices 111 in all computing clusters 110 are mounted on the shared storage cluster 120, data and software environments only need to be pre-configured in the storage cluster 120 to achieve sharing between different computing devices 111. For example, by storing all input data sources and result data during the compilation process in the shared storage cluster 120, all computing devices 111 do not need to transfer or copy data to each other when executing compilation tasks, greatly saving the time cost of the compilation process. Furthermore, by deploying the software environment and applications (such as executable programs) that the compilation depends on to the shared storage cluster 120, simply mounting them to a cloud disk and setting environment variables can achieve the deployment and migration of the compilation environment, which is both convenient and automated.

[0111] Because of the abundant computing resources in the cloud, the existing computing cluster 110 can be quickly expanded (e.g., by adding computing devices within the cluster), or a new computing cluster 110 can be added, enabling the clustering of computing resources for data compilation. Based on this, through reasonable process scheduling, on the one hand, the computing resources within the cluster can be utilized to the maximum extent, increasing the parallelism of compilation tasks and thus improving the compilation efficiency of map data. On the other hand, computing devices executing compilation tasks can be dynamically allocated, avoiding the blockage of the entire process due to the failure of a single computing device. Furthermore, the computing devices can be automatically scaled up and down without stopping the compilation process or requiring manual intervention, thus achieving a flexible and intelligent automated scheduling mechanism.

[0112] It should be noted that, before the computing cluster 110 and the storage cluster 120, a process scheduling device can also be configured in the system. The process scheduling device is used to receive map service requests from the terminal (i.e., the client side) and dynamically allocate computing devices to a series of compilation tasks obtained based on the parsing of the map service request. The dynamic allocation and process scheduling process will be described in detail in subsequent embodiments.

[0113] In some embodiments, the terminal initiating the map service request may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited thereto. The terminal and the process scheduling device can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this.

[0114] Figure 2 This is a flowchart illustrating a map data compilation method provided in an embodiment of this application. See also... Figure 2 This embodiment is executed by a computer device. Taking the computer device as an example, which is a process scheduling device in the above implementation environment, the embodiment includes the following steps:

[0115] 201. The process scheduling device responds to a map service request and obtains at least one compilation task associated with the map service request.

[0116] A process scheduling device is a computer device located before a storage-compute separated map compilation system, used to process map service requests sent by clients on the terminal side. The process scheduling device can be a standalone device, integrated on the same physical machine with any computing device in any computing cluster, or integrated on the same physical machine with a gateway or proxy device; this application embodiment does not specifically limit this.

[0117] In some embodiments, map service requests are sent by a client on a terminal to a process scheduling device. The client can be any product in the map industry that can provide basic service capabilities, such as mobile maps, in-vehicle navigation systems, ride-hailing platforms, logistics systems, etc. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0118] In some embodiments, map service requests are sent by the client on the terminal to the service server, which then forwards them to the process scheduling device. These map service requests can be location query requests, navigation requests, logistics query requests, route preview requests, etc. All of these types of requests require at least the compilation results of the map data.

[0119] In some embodiments, after receiving any request, the process scheduling device parses the header field of the request. When the header field indicates that the request is a map service request (e.g., the header field carries a map service identifier), it parses the data field of the map service request to obtain the service request statement of the map service request.

[0120] In some embodiments, the process scheduling device is configured with three modules: a process engine, a remote execution tool, and a scheduling decision service. Optionally, the parsed business request statement is sent to the process engine, which analyzes the business request statement to obtain at least one compilation task.

[0121] 202. The process scheduling device allocates a computing device to any one of the at least one compilation tasks for processing the compilation task.

[0122] In some embodiments, for compilation tasks that do not have upstream or downstream dependencies, these compilation tasks can be executed in parallel to maximize compilation efficiency. Therefore, the process scheduling device can wake up all these compilation tasks that can be executed in parallel at once, and then send a scheduling allocation request to the scheduling decision service to request the scheduling decision service to dynamically allocate a suitable computing device for each of the above compilation tasks in real time.

[0123] In some embodiments, since there may be upstream and downstream dependencies between compilation tasks, these compilation tasks cannot be executed in parallel. The process scheduling device executes these compilation tasks with upstream and downstream dependencies in sequence. For any compilation task, after the process engine determines that the upstream task of the compilation task has been completed, it wakes up the execution process of the compilation task and then sends a scheduling allocation request to the scheduling decision service. The scheduling allocation request is used to request the scheduling decision service to dynamically allocate a suitable computing device for the currently woken-up compilation task in real time.

[0124] In some embodiments, the scheduling decision service prioritizes allocating an idle computing device to the compilation task. An idle computing device is one that is not currently occupied by any task. All computing resources on an idle computing device are available computing resources, which enables the computing cluster to be as load-balanced as possible.

[0125] In some embodiments, the scheduling decision service prioritizes allocating a computing device with more idle resources than the computing resources required to process the compilation task. That is, it only needs to ensure that the allocated computing device has sufficient idle resources to process the compilation task, so that the compilation task will not be terminated due to insufficient computing resources.

[0126] In some embodiments, if a compilation task requires a lot of computing resources, and only a few computing devices within the computing cluster can provide these computing resources, it is equivalent to the compilation task being bound to a specific device and only able to be executed on that specific device. In this case, an idle specific device can be allocated to the compilation task first, or an idle specific device with more computing resources than the computing resources required to process the compilation task can be allocated to the compilation task first. The scheduling and allocation strategies under different circumstances will be described in detail in the next embodiment, and will not be repeated here.

[0127] Because in a storage-compute separation system architecture, each computing device is mounted on a shared storage cluster (i.e., cloud disk), and the computing resources in the cloud are sufficient, there is no constraint that a compilation task can only be bound to a certain physical machine when allocating computing devices for compilation tasks. The process scheduling device can allocate the most suitable computing device for the compilation task in real time according to the load requests of each computing device in the computing cluster, ensuring that there are enough idle resources on the computing device to handle the compilation task, and can achieve dynamic load balancing of the computing cluster, thereby improving the resource utilization within the computing cluster.

[0128] 203. The process scheduling device sends a remote execution command for the compilation task to the computing device. The remote execution command is used to instruct the computing device to execute the compilation task using map data stored in the cloud disk.

[0129] In some embodiments, after the process scheduling device allocates a computing device for the compilation task, the process scheduling device sends a remote execution command to the allocated computing device through a remote execution tool. Upon receiving the remote execution command, the computing device determines the map data involved in the operation of the compilation task in the cloud storage directory of the mounted cloud disk, and performs remote compilation on the determined map data to execute the compilation task. After the compilation task is completed, the computing device returns the execution status of the compilation task to the process scheduling device. The execution status is used to indicate whether the compilation task was executed successfully or failed.

[0130] In some embodiments, the process scheduling device performs steps 202-203 as described above for each compilation task that is woken up in sequence. It should be noted that for some compilation tasks that do not have upstream and downstream dependencies, the process scheduling device can wake up these compilation tasks in parallel, that is, simultaneously allocate multiple computing devices to multiple compilation tasks that do not have upstream and downstream dependencies, and simultaneously send remote execution commands to multiple computing devices. This application embodiment does not specifically limit this.

[0131] 204. The process scheduling device returns the compilation result obtained by at least one computing device based on the remote execution command.

[0132] In some embodiments, after sending a remote execution command to the computing device allocated to each compilation task, the process scheduling device obtains the execution status of the computing device after executing the compilation task based on the remote execution command. If the execution status is successful, the downstream tasks of the compilation task are woken up. If the execution status is failed, a new computing device needs to be allocated to the compilation task so that the compilation task can be executed again on the new computing device. The above operation is repeated until the execution status of all compilation tasks indicates successful execution. At this time, it means that all compilation tasks have been successfully executed. A compilation result after at least one compilation task has been executed sequentially will be generated on the cloud disk. The process scheduling device can read the compilation result from the cloud disk and return the compilation result to the device that sent the map service request. For example, if the map service request is sent directly to the process scheduling device by the client on the terminal, the process scheduling device will return the compilation result to the client. Or, if the map service request is forwarded to the process scheduling device by the service server, the process scheduling device will return the compilation result to the service server. The service server will then process or integrate other data into the compilation result before returning it to the client. This application embodiment does not specifically limit this.

[0133] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0134] The method provided in this application dynamically allocates computing devices in real time according to the load request within the system by compiling map service requests, and triggers the computing devices to start compilation tasks by remotely executing commands. Since the computing devices are mounted to the cloud disk, they can read and write map data in the cloud disk to achieve remote compilation. Since the computing devices bound to each compilation task can read and write map data in the cloud disk by being mounted to the cloud disk, it avoids the need to copy data between the physical machines of upstream and downstream tasks when the compilation task is bound to a physical machine, which simplifies the map data compilation process and greatly improves the compilation efficiency of map data.

[0135] Figure 3 This is a flowchart illustrating a map data compilation method provided in an embodiment of this application. See also... Figure 3 This embodiment is executed by a computer device. Taking the computer device as an example, which is a process scheduling device in the above implementation environment, the embodiment includes the following steps:

[0136] 301. The process scheduling device responds to a map service request and obtains at least one compilation task associated with the map service request.

[0137] Step 301 above is similar to step 201 above, and will not be described in detail here.

[0138] In this embodiment, an example is taken where a process scheduling device is configured with three modules: a process engine, a remote execution tool, and a scheduling decision service. These three modules work together to form a scheduling system for compilation tasks. A map service request is associated with an overall compilation process, which includes at least one compilation task, and each compilation task is equivalent to a compilation step in the compilation process.

[0139] The process engine is used to parse and split a given map business request into at least one compilation task. The execution of the compilation process depends on the scheduling of the process engine. For compilation tasks that can be executed in parallel, the process engine can allocate computing devices for them and start execution at the same time. For compilation tasks that cannot be executed in parallel, there are usually upstream and downstream dependencies. The process engine will execute them sequentially according to the order of compilation tasks (i.e., dependencies).

[0140] Each compilation task can be started by a tool that can trigger remote execution commands: the compiler operator. After the process engine wakes up a compilation task, it sends the compilation task to the compiler operator. The compiler operator requests the allocation of a computing device from the scheduling decision service and sends the remote execution command of the compilation task to that computing device.

[0141] The scheduling decision service dynamically returns the IP addresses of the computing devices allocated to each compilation task to the compiler operator of the process engine, achieving unified scheduling and allocation of computing resources. Typically, based on the status and usage of the computing cluster, the scheduling decision service dynamically allocates suitable computing devices to each compilation task during execution. Simultaneously, it dynamically identifies the availability of computing devices, promptly detecting faulty devices so that compilation tasks originally assigned to faulty devices can be reassigned to new, healthy computing devices for re-execution, ensuring high availability of the entire computing cluster.

[0142] 302. For any one of the at least one compilation tasks, the process scheduling device determines the computing resources required to process the compilation task based on the compilation information table, which is used to record the computing resources required by multiple compilation tasks.

[0143] In some embodiments, the process scheduling device determines from the at least one compilation task that there is no upstream or downstream dependency between the compilation tasks. These compilation tasks are compilation tasks that can be executed in parallel. The device can wake up each compilation task at once and allocate computing devices for processing the compilation task in parallel through steps 302-305.

[0144] In some embodiments, the process scheduling device determines from the at least one compilation task that has upstream and downstream dependencies. Since the upstream task among these compilation tasks can be woken up along with the compilation tasks that can be executed in parallel, the remaining compilation tasks need to wait for their upstream tasks to finish executing before they can be woken up.

[0145] In some embodiments, for any awakened compilation task, the process scheduling device queries a pre-configured compilation information table, which records the computing resources required by each of the multiple compilation tasks. For example, the task identifier of the compilation task and the computing resources required are associated and stored in the compilation information table. In this case, by using the task identifier of the compilation task as an index, querying the index content associated with the index, the computing resources required by the currently awakened compilation task can be obtained.

[0146] Schematic illustration: In the Module_Info table, a data row is created for each compilation task. This data row contains the following fields: Task Identification (ID), Task Name (Chinese), Task Name (English), CPU Required, Memory Required, Timeout Parameter, Fixed IP, Upstream Task ID, and Notifier. The Timeout Parameter specifies the maximum waiting time before the compilation task is considered timed out. The Fixed IP records whether the compilation task is configured to execute on a specific device; if the Fixed IP is empty, it means the compilation task is not specified for execution on any particular device. The Upstream Task ID refers to the upstream task of this compilation task when there are dependencies between upstream and downstream tasks. The Notifier refers to the management account of the technical personnel to be notified when a compilation task encounters an exception. Table 1 illustrates one possible structure of the Module_Info table.

[0147] Table 1

[0148]

[0149]

[0150] As can be seen, the road network compilation task has a task ID of 1, and its upstream task is the compilation task with ID 0 (not shown in the table). The road network compilation task requires at least 4GB of CPU and at least 30GB of memory. It is considered to have timed out if it waits for more than 30 minutes, and no specific device is specified. The induced compilation task is the downstream task of the road network compilation task. The induced compilation task has a task ID of 2, requires at least 5GB of CPU and at least 40GB of memory, and is considered to have timed out if it waits for more than 1 hour. It is specified to be executed on the specific device GUIDE_IP.

[0151] 303. The process scheduling device routes the compilation task to a computing cluster that matches the service type of the map service request.

[0152] The business type refers to whether the map service request belongs to a formal business, a test business, or another project business.

[0153] In some embodiments, under a storage-compute separation system architecture, a formal compilation cluster, a test compilation cluster, and other project clusters are configured. The formal compilation cluster is primarily used to produce compilation data that can be directly deployed online; that is, the compilation results generated by the formal compilation cluster will be directly deployed to the official map product or applied to map-based service products. These compilation results can be deployed to the product in full or incrementally. The test compilation cluster is mainly used to test and verify new data, new tools, or new processes. However, the test results are not deployed to the product; instead, they are transferred to the formal compilation cluster after the tests are passed for final deployment. Since the testing process does not have high requirements for compilation efficiency, the test compilation cluster can run multiple test tasks in parallel, with the process engine scheduling and controlling the resource usage of each computing device in the test compilation cluster. The other project clusters are used to provide independent compilation environments for other applications that need to perform map data compilation. Optionally, different projects or applications may share the same other project cluster, or different other project clusters may be configured for different projects or applications. This embodiment does not specifically limit this.

[0154] In some embodiments, the process scheduling device parses the map service request and, based on the service identifier carried in the map service request, determines the service type to which the service identifier belongs. Then, it routes the map service request to a computing cluster that matches the service type. For example, if the service identifier carried in the map service request is a test service identifier, the map service request is routed to a test compilation cluster that matches the test service identifier. Or, if the service identifier carried in the map service request is a production service identifier, the map service request is routed to a production compilation cluster that matches the production service identifier.

[0155] In step 303 above, only the case where multiple computing clusters are divided is shown. How to correctly route map service requests to the appropriate computing cluster according to the service type in order to optimize the resource configuration of the computing cluster. In some embodiments, it is assumed that there are no multiple computing clusters in the system architecture, that is, only one computing cluster is provided. In this case, there is no need to execute step 303 above.

[0156] In other embodiments, in addition to dividing computing clusters according to business type, computing clusters can also be divided according to the amount of computing resources provided. For example, computing clusters can be divided into computing clusters with higher computing resource configurations and computing clusters with lower computing resource configurations.

[0157] For computing clusters with limited computing resources, this approach can be used to process subtasks derived from compilation tasks that do not rely on global map data. In other words, because these compilation tasks do not depend on global map data, the global map data can be broken down into multiple map tiles. One or more map tiles can then be packaged into a subtask. This allows a time-consuming and resource-intensive compilation task to be broken down into multiple subtasks that can be executed in parallel on multiple computing devices, thereby improving the compilation efficiency for such tasks. Specifically, for any compilation task that does not depend on global map data, the task is routed to a computing cluster that supports splitting the task into multiple subtasks (i.e., a computing cluster with limited computing resources).

[0158] For computing clusters with high computing resource configurations, they can be used to handle compilation tasks that depend on global map data. These compilation tasks, such as navigation services like selecting navigation paths, can be directly routed to computing clusters with high computing resource configurations. These clusters typically contain computing devices with large amounts of memory, enabling them to process compilation tasks that depend on global map data quickly and efficiently. That is, for any compilation task that depends on global map data, the compilation task is routed to a computing cluster whose computing resources meet a third resource condition. This third resource condition means that the computing resources are greater than a target threshold, or that the computing resources provided by the lowest-configured computing device in the computing cluster exceed the computing resources required by the compilation task. This application does not specifically limit the third resource condition.

[0159] By splitting tasks according to data range, different compilation tasks can be routed to computing clusters with different computing resource configurations. This enables dynamic scheduling of computing resources, allocating each compilation task to a computing cluster with appropriate computing resources, and further improving resource utilization in a storage-compute separation system.

[0160] In other embodiments, besides dividing computing clusters according to business type or the amount of computing resources provided, computing clusters can also be divided according to compilation tasks. In other words, for each compilation task, the operations to be performed by the compilation task are encapsulated into an asynchronous API service, such as an API (Application Programming Interface). Each API service is associated with a computing cluster containing one or more computing devices. After receiving a map business request, the process scheduling device parses at least one compilation task. For each compilation task, it can determine the API service associated with that compilation task and then route the compilation task to the computing cluster associated with that API service. In the above process, after parsing at least one compilation task, each compilation task is routed to a dedicated small computing cluster through the API service. This eliminates the need to distinguish whether the business type is formal business, test business, or other project business. Compilation tasks under all business types will asynchronously call the API, thereby being routed to the associated computing cluster for execution. This makes the granularity of computing cluster division more refined.

[0161] 304. The process scheduling device determines the idle resources of each of the multiple computing devices in the computing cluster based on the device information table.

[0162] In some embodiments, after determining which computing cluster to route the compilation task to through step 303 above, the process scheduling device can query the device information table, which records whether each computing device in each computing cluster is available, or the device information table also records how many resources are available in each computing device in each computing cluster. This application embodiment does not specifically limit this.

[0163] In some embodiments, the process scheduling device maintains a separate device information table for each computing cluster. This device information table records relevant information about each computing device in the cluster, such as device IP, device login information, CPU, memory, belonging computing cluster, CBS mount directory, unique identifier, availability, and remaining idle resources. In other embodiments, the process scheduling device maintains all computing devices in all computing clusters in a single master device information table. This application does not specifically limit whether a separate device information table is maintained for each computing cluster.

[0164] In some embodiments, for each computing device in each computing cluster, the idle resources of that computing device can be directly queried from the device information table. That is, a column is set in the device information table to record the idle resources of each computing device, so that the idle resources of each computing device can be quickly queried, thereby improving the efficiency of obtaining idle resources.

[0165] In other embodiments, for each computing device in each computing cluster, the maximum computing resources that the computing device can provide and the task identifiers of each compilation task currently allocated to the computing device are retrieved from the device information table. Then, the computing resources required by each compilation task are retrieved from the compilation information table. The computing resources required by each compilation task allocated to the computing device are added together to obtain the occupied resources of the computing device. The difference between the maximum computing resources that the computing device can provide and the occupied resources is determined as the idle resources of the computing device. In this way, there is no need to add an idle resource field to the device information table, which can save the storage overhead occupied by the device information table.

[0166] Indicatively, in the device information table (Node_Info), a data row is created for each compilation task. This data row is configured with the following fields: Device IP, Device Login Information, CPU, Memory, Compute Cluster, CBS Mount Directory, Dedicated Identifier, Availability, Current Load, and Remaining Idle Resources. Specifically, Device IP refers to the IP address of the computing device; Login Information refers to the account information currently logged into the computing device; CPU and Memory represent the maximum computing resources the computing device can provide; Compute Cluster indicates which computing cluster the computing device currently belongs to; CBS Mount Directory indicates which cloud storage directory the computing device is currently mounted to; Dedicated Identifier indicates whether the computing device is dedicated to processing a specific compilation task (a non-empty identifier indicates the device is dedicated to a specific task, an empty identifier indicates it is not); Availability indicates whether the computing device is currently available (equivalent to indicating whether the device has malfunctioned); Current Load represents the user load currently handled by the computing device, which can be considered the number of compilation tasks being processed and not yet processed; and Remaining Idle Resources represents the amount of idle resources currently remaining on the computing device. Table 2 shows one possible way to construct a device information table.

[0167] Table 2

[0168] 0.0.0.1 User:XXXX 64 256 Formal / data ROAD 1 … … … … … … … …

[0169] As can be seen, Table 2 represents the data columns that the device information table can have during the initialization phase. During initialization, since no compilation task has been assigned to each computing device, the idle resources are equal to the maximum computing resources that the CPU and memory fields can provide. As the compilation process progresses, assuming that at some point a compilation task with ID 1 is assigned to the computing device with IP 0.0.0.1 in Table 2, a field can be added to the far right to record which compilation task is currently using that computing device. For example, a field called "Active Load" could be added, recording the compilation task ID "1" in the "Active Load" field, representing the current status of the computing device. Currently being used by the compilation task with ID 1, assuming that the compilation task with ID 1 requires 4GB of CPU and 30GB of memory, then obviously, subtracting the CPU used by the compilation task from the CPU of the computing device, and subtracting the memory used by the compilation task from the memory of the computing device, yields the remaining free resources, including 60GB of CPU and 226GB of memory. The free resources can be added as a separate column to the device information table. The free resources can be the value obtained by subtracting the computing resources used by the allocated compilation task from the computing resources that the computing device can provide based on the device information table. This application embodiment does not specifically limit the source of the free resources.

[0170] 305. The process scheduling device allocates a computing device with more idle resources than the computing resources to the compilation task.

[0171] In some embodiments, certain compilation tasks may require significant memory usage or specific operating system version requirements, necessitating execution on specific devices. In such cases, these compilation tasks are associated with these specific devices and will only be assigned to these associated devices, not dynamically allocated to other non-specific devices. For example, the "Fixed IP" field of a compilation task in the compilation information table records the IP address of the associated specific device. Therefore, for any currently awakened compilation task, querying the compilation information table can determine whether the task has an associated specific device. For instance, querying the "Fixed IP" field of the data row containing the compilation task in the compilation information table reveals whether the task is associated with any specific device. If the "Fixed IP" field is empty, the compilation task is not associated with any specific device; if it is not empty, the task is associated with the specific device indicated by the IP address recorded in the field. The following discussion will address both cases of association with and non-association with specific devices.

[0172] In some embodiments, when the compilation task is associated with a specific device, since the specific device is also an example of a computing device, the idle resources of the specific device can be obtained through step 304 above. Optionally, the idle resources of the specific device are compared with the computing resources required by the compilation task. If the idle resources of the specific device are greater than the computing resources required by the compilation task, it means that the specific device has sufficient resources to process the compilation task, and the compilation task can be assigned to the specific device. If the idle resources of the specific device are less than or equal to the computing resources required by the compilation task, it means that the specific device does not have sufficient resources to process the compilation task, and the compilation task can be put into a waiting state. The waiting state means that there is currently no suitable specific device in the computing cluster that can process the compilation task, and it can only wait for the specific device to finish processing the currently executing task and release the computing resources originally occupied before processing the compilation task. It should be noted that the compilation task will not be in an endless wait. After the waiting time exceeds the timeout parameter in the compilation information table, the process scheduling device directly determines that the compilation task has failed.

[0173] In some embodiments, if the compilation task is not associated with any specific device, then a non-specific device with sufficient resources can be prioritized for allocation to the compilation task; that is, the non-specific device meeting the first resource condition can be allocated to the compilation task. If no non-specific device meeting the first resource condition exists in the computing cluster, then a specific device with sufficient resources can be considered for allocation to the compilation task; that is, a specific device meeting the second resource condition can be allocated to the compilation task. If no specific device meeting the second resource condition exists in the computing cluster, it means that neither the specific device nor the non-specific device currently has sufficient resources to process the compilation task, and the compilation task can be put into a waiting state. The waiting state is similar to the previous situation and will not be elaborated here. Here, a non-specific device refers to a computing device that is not associated with any compilation device; in other words, a non-specific device refers to a computing device whose "dedicated identifier" field is empty in the device information table.

[0174] In the above process, for compilation tasks that are not associated with any specific device, by prioritizing the allocation of non-specific devices with sufficient resources to the compilation task, we can avoid occupying the computing resources of specific devices as much as possible. Secondly, we consider allocating specific devices with sufficient resources to the compilation task, which can ensure that the compilation task can be executed smoothly and avoid getting stuck in a waiting state. Finally, if neither of these conditions is met, we enter a waiting state. In this way, we can realize the dynamic allocation of computing devices according to a certain priority order and improve the resource utilization of the entire scheduling system.

[0175] In some embodiments, when prioritizing the allocation of compilation tasks to non-specific devices with sufficient resources, the following operations are performed: If idle non-specific devices exist in the computing cluster, the compilation task is allocated to these idle non-specific devices. Idle non-specific devices refer to computing devices that are not currently assigned any compilation tasks. For example, all non-specific devices with an empty "Dedicated Identifier" field are retrieved from the device information table. Then, among all non-specific devices, further filtering is performed to identify devices with an empty "Current Load" field or an empty "Assigned Task Identifier" field. These filtered devices are the idle non-specific devices. When only one idle non-specific device is found, the compilation task is directly assigned to that single selected non-specific device. When multiple idle non-specific devices are found, [further steps may be taken]. The compilation task is randomly assigned to any of the selected non-specific devices; this embodiment does not specifically limit this. Furthermore, if no idle non-specific devices exist in the computing cluster, the idle resources of each non-specific device can be determined from the device information table. The idle resources of each non-specific device are compared with the computing resources required by the compilation task. Then, a non-specific device with more idle resources than the required computing resources is assigned to the compilation task. This ensures that the compilation task is assigned to a non-specific device with sufficient resources. Optionally, assuming there are multiple non-specific devices with more idle resources than the required computing resources, the non-specific device with the most idle resources can be prioritized for the compilation task. This avoids resource congestion and reduces competition for computing resources within the computing cluster.

[0176] In some embodiments, if no non-specific device meeting the first resource condition exists in the computing cluster, then since none of the non-specific devices within the computing cluster have sufficient resources to execute the compilation task, it is advisable to allocate a specific device with sufficient resources to the compilation task. That is, when allocating a specific device meeting the second resource condition to the compilation task, the following operations are performed: Allocate an idle specific device that is associated with a downstream task of the compilation task. This downstream task is a task that can only be executed after the compilation task has completed. In other words, query the device information table to obtain all specific devices whose "dedicated identifier" field is not empty. Then, for each specific device, determine whether the associated compilation task of that specific device is a downstream task of the compilation task, and then... Among the specific devices associated with the downstream tasks of the compilation task, priority is given to allocating idle specific devices that are associated with the downstream tasks of the compilation task. Furthermore, if there are no idle specific devices associated with the downstream tasks of the compilation task in the computing cluster, the idle resources of each specific device associated with the downstream tasks of the compilation task can be retrieved from the device information table. Then, the compilation task is allocated a specific device whose idle resources are greater than the computing resources required by the compilation task and which is associated with the downstream tasks of the compilation task. Additionally, assuming there are no specific devices in the computing cluster whose idle resources are greater than the computing resources required by the compilation task and which are associated with the downstream tasks of the compilation task, the compilation task is put into a waiting state.

[0177] To illustrate, the process involves querying the device information table to find all specific devices whose "dedicated identifier" field is not empty. Then, for each specific device, it is determined whether the associated compilation task of that specific device is a downstream task of the current compilation task. In other words, it is determined whether the compilation task is an upstream task of the associated compilation task of that specific device. After reading the task name recorded in the "dedicated identifier" field of the specific device, the "upstream ID" field of the associated compilation task indicated by the task name is queried in the compilation information table. If the task ID recorded in the "upstream ID" field is exactly equal to the task ID of the compilation task to be assigned, it means that the current compilation task is the upstream task of the associated compilation task of that specific device. Repeating the above operation can filter out all specific devices that are associated with the downstream task of the compilation task. Then, from the filtered specific devices, idle specific devices are preferentially allocated to the compilation task. If there are no idle specific devices, then specific devices with currently remaining idle resources greater than the computing resources required by the compilation task are allocated.

[0178] In the above process, by targeting a specific device, in addition to determining whether it is idle and has sufficient resources, it is also determined whether the compilation task is an upstream task of the associated compilation task of that specific device. This ensures that the specific device prioritizes providing computing resources to its associated compilation tasks and the upstream tasks of those associated compilation tasks. For other compilation tasks that do not have upstream dependencies, the computing resources of the specific device will not be occupied. This ensures that the computing resources of the specific device will not be arbitrarily occupied, and improves the rationality of resource allocation of the entire computing cluster.

[0179] In some embodiments, it may not be necessary to determine whether the compilation task is an upstream task of the associated compilation task of the specific device. This can ensure that the current compilation task can be allocated to a computing device as soon as possible, avoid the current compilation task from entering a waiting state, and speed up the progress of the compilation task allocation to the computing device.

[0180] Figure 4 This is a schematic diagram illustrating the computing device selection principle of a scheduling decision service provided in an embodiment of this application, such as... Figure 4 As shown, assuming the computing cluster contains computing devices 401-410, the shaded areas represent the computing resources already occupied on the devices, and the blank areas represent the idle resources (i.e., unoccupied computing resources) on the devices. At a certain moment, a compilation task is woken up, requesting the scheduling decision service to allocate a computing device. The following filtering can be performed: First, filtering reveals computing devices 405-410 within the computing cluster with more idle resources than the compilation task requires. Computing devices 405-408 are non-specific devices, while computing devices 409-410 are specific devices. Next, filtering reveals idle non-specific devices 405 and 406. Assuming no idle non-specific devices exist, then filtering reveals non-specific devices 407 and 408 with sufficient resources. Assuming no resources exist... If there are no idle devices that are associated with downstream tasks of the compilation task, then the specific device 409 is selected. For example, the specific device 409 is associated with compilation tasks with task IDs 13 and 140, and the current compilation task is an upstream task of the compilation task with task IDs 13 or 140 (such as the compilation task with ID 12 or 139). If there are no idle devices that are associated with downstream tasks of the compilation task, then the specific device 410 with the largest remaining idle resources and associated with downstream tasks of the compilation task is selected. For example, the specific device 410 is associated with the compilation task with task ID 171, and the current compilation task is an upstream task of the compilation task with task ID 171 (such as the compilation task with ID 170).

[0181] Figure 5This is a schematic diagram illustrating the principle of a scheduling decision-making strategy provided in an embodiment of this application, such as... Figure 5 As shown, assuming the request queue contains two compilation tasks with IDs 13 and 1352, and these two compilation tasks are currently being woken up in parallel, for each compilation task, in step 501, it is determined whether to execute on a specific device. For example, the "Fixed IP" field of the data row where the compilation task is located is queried in the compilation information table. If the "Fixed IP" field is empty, it means that the compilation task is not associated with any specific device, and proceeds to step 503. If the "Fixed IP" field is not empty, it means that the compilation task is associated with the specific device indicated by the IP address recorded in the field, and proceeds to step 502. In step 502, it is determined whether the remaining idle resources of the specific device are available. Sufficient means determining whether the available resources are greater than the computing resources required by the compilation task. If the available resources are greater than the computing resources required by the compilation task, it means that there are sufficient remaining available resources. In this case, the IP address of the specific device is returned and the compilation task is executed immediately. If the available resources are less than or equal to the computing resources required by the compilation task, it means that there are insufficient remaining available resources. In this case, the compiler operator is returned and the process enters a waiting state. In step 503, a computing device with sufficient remaining available resources is selected. At this time, the following is determined in sequence: (1) Select an available non-specific device. If there is an available non-specific device, the IP address of the available non-specific device is returned and the process is executed immediately. If there is no free non-specific device, proceed to (2); (2) Select a non-free but sufficiently free non-specific device. If there is a sufficiently free non-specific device, return the IP address of the sufficiently free non-specific device and execute the compilation task immediately. If there is no sufficiently free non-specific device, proceed to (3); (3) Select a free specific device. Then, determine whether the associated compilation task of the specific device is a downstream task of the current compilation task, or whether the compilation task is an upstream task of the associated compilation task of the specific device. If the associated compilation task of the specific device is a downstream task of the current compilation task, or if the associated compilation task of the specific device is an upstream task of the associated compilation task of the specific device, proceed to (3); (3) Select a free specific device. Then, determine whether the associated compilation task of the specific device is a downstream task of the current compilation task, or whether the associated compilation task of the specific device is an upstream task of the associated compilation task of the specific device. If the compilation task is an upstream task of the associated compilation task of the specific device (the two are equivalent), then return the IP address of the specific device that is idle and meets any of the above equivalent conditions, and execute the compilation task immediately. If there is no specific device that is idle and meets the above conditions, proceed to (4); (4) Select a specific device that is not idle but has sufficient idle resources. Then, determine whether it meets any of the above equivalent conditions. If it meets any of the above equivalent conditions, return the IP address of the specific device that has sufficient idle resources and meets the conditions, and execute the compilation task immediately. If there is no specific device that has sufficient idle resources and meets the conditions, return to the compiler Operator and enter the waiting state.

[0182] In steps 302-305 above, a possible implementation of the process scheduling device allocating computing devices for processing the compilation task is shown. That is, for compilation tasks that are associated with a specific device, an idle specific device is allocated first. If there is no idle specific device, an idle specific device with sufficient resources is allocated next. If neither condition is met, a waiting state is entered. For compilation tasks that are not associated with a specific device, an idle non-specific device is allocated first. If there is no idle non-specific device, an idle non-specific device with sufficient resources is allocated next. If there is no idle non-specific device with sufficient resources, an idle specific device that is associated with the downstream task of the compilation task is allocated next. If there is no idle specific device that is associated with the downstream task of the compilation task, an idle specific device that is associated with the downstream task of the compilation task is allocated next. If neither condition is met, a waiting state is entered.

[0183] In other embodiments, the following allocation strategy can also be implemented: For compilation tasks associated with a specific device, priority is given to allocating the largest and most available non-specific device. If no non-specific device with sufficient free resources is available, the task enters a waiting state. For compilation tasks not associated with a specific device, priority is given to allocating the largest and most available non-specific device. If no non-specific device with sufficient free resources is available, then the largest and most available specific device is allocated. If neither condition is met, the task enters a waiting state. That is, it is only necessary to ensure sufficient free resources. Among devices with sufficient free resources, the device with the largest free resource is allocated first. This eliminates the need to determine whether a device is idle, saving the complexity of the device allocation and scheduling phase and improving the coordination efficiency of the device allocation and scheduling phase.

[0184] 306. The process scheduling device sends a remote execution command for the compilation task to the computing device. The remote execution command is used to instruct the computing device to execute the compilation task using map data stored in the cloud disk.

[0185] Step 306 above is similar to step 203 above, and will not be described in detail here.

[0186] 307. In response to the remote execution command, the computing device executes the compilation task based on the map data stored in the cloud disk and returns the execution status of the compilation task to the process scheduling device.

[0187] In some embodiments, the computing devices used to process any map service request compilation task are all mounted to the cloud disk. In other words, all computing devices in all computing clusters are mounted on the same cloud disk, which refers to a storage cluster (equivalent to a shared storage system) composed of multiple storage devices. This storage cluster is used to store the map data required for the compilation task.

[0188] In some embodiments, since all computing devices are mounted on the same cloud disk, leveraging the advantages of shared disk, symbolic links are used during the execution of compilation tasks to reduce local data copying. Symbolic links, also known as soft links, are used to link any file or directory in different file systems, typically represented by a file containing the pathname of another file. When reading or writing to a symbolic file via a symbolic link, the system automatically translates the operation into an operation on the source file. However, when deleting a link file, the system only deletes the link file, not the source file itself. Using symbolic links for remote compilation on the cloud disk reduces local data copying, thereby further improving the execution efficiency of compilation tasks and optimizing storage space resource utilization.

[0189] Figure 6 This is a schematic diagram of the structure of a cloud disk data storage soft link provided in an embodiment of this application, such as... Figure 6 As shown, during the compilation task, the road network data "2022021203" from the symbolic link is read into the data directory "road->", and the mining data "220202163456" from the symbolic link is read into the data directory "digdata->". Similarly, the guidance toolkit "202202161114" from the symbolic link is read into the tool execution directory "guid_compile", and the weighting toolkit "20220216093323" from the symbolic link is read into the tool execution directory "cost_compile". Next, the source files of each symbolic link in the data directory and tool execution directory are compiled, and finally the compilation result directory 601 is obtained. The compilation result directory 601 records a series of compilation results of the current compilation task. Each compilation result will be provided to one or more server data directories 602. The server data directory 602 will be divided into multiple server data (ServiceData). Each server data is connected to the symbolic link of the data directory or compilation result directory in the form of a soft link, and is provided to the business server to use the server data for subsequent business processing.

[0190] In some embodiments, after each computing device completes the assigned compilation task, the computing device returns the execution status of the compilation task to the process scheduling device. The execution status is used to indicate whether the compilation task was executed successfully or failed.

[0191] 308. If at least one compilation task has been successfully executed, the process scheduling device returns the compilation result obtained by at least one computing device based on the remote execution command.

[0192] In some embodiments, the process scheduling device performs steps 302-307 above for each compilation task. It should be noted that for some compilation tasks that do not have upstream and downstream dependencies, the process scheduling device can wake up these compilation tasks in parallel, that is, simultaneously allocate multiple computing devices to multiple compilation tasks that do not have upstream and downstream dependencies, send remote execution commands to multiple computing devices at the same time, and receive the execution status returned by each computing device, or obtain an execution status after waiting for a timeout (the default is that the execution fails if the waiting timeout occurs). This application embodiment does not specifically limit this.

[0193] In some embodiments, for compilation tasks with upstream and downstream dependencies, after sending a remote execution command to the computing device allocated to each compilation task, the process scheduling device obtains the execution status of the computing device after executing the compilation task based on the remote execution command. If the execution status is successful, the downstream task of the compilation task is woken up. If the execution status is failed, a new computing device needs to be allocated to the compilation task so that the compilation task can be executed again on the new computing device. The above operation is repeated until the execution status of all compilation tasks indicates successful execution. At this time, it means that all compilation tasks have been successfully executed, and a compilation result after at least one compilation task has been executed sequentially will be generated on the cloud disk. The process scheduling device can read the compilation result from the cloud disk and return the compilation result to the device that sent the map service request. For example, if the map service request is sent directly to the process scheduling device by the client on the terminal, the process scheduling device returns the compilation result to the client. Or, if the map service request is forwarded to the process scheduling device by the service server, the process scheduling device returns the compilation result to the service server, which processes the compilation result or integrates other data before returning it to the client. This application embodiment does not specifically limit this.

[0194] Figure 7 This is a schematic flowchart illustrating a method for compiling map data according to an embodiment of this application. Figure 7As shown, assuming the process scheduling device includes three modules: a scheduling decision service 701, a process engine 702, and a remote execution tool, in the map data compilation process, after receiving a map business request, the process is started. The process engine 702 parses and obtains at least one compilation task. The scheduling decision service 701 initializes a compilation information table for this task. Then, the scheduling decision service 701 notifies the compiler operator to start a DAG (Directed Acyclic Graph), which is the core data structure of the compiler operator's backend. Next, the compiler operator obtains the execution IP of the current compilation task, i.e., requests the scheduling decision service 701 to allocate a suitable computing device for the task. Finally, the compiler operator sends the remote execution command for the compilation task to the allocated computing device via the remote execution tool, for example, ... Figure 7 As shown, the scheduling decision service 701 assigns an execution IP "0.0.0.3" to the current compilation task. Then, it calls the remote execution tool to send the remote execution command of the current compilation task to the computing device with IP address "0.0.0.3" in the CVM cluster 703 (i.e., the computing cluster to which the compilation task is routed), so that the computing device with IP address "0.0.0.3" can execute the current compilation task.

[0195] In the above map data compilation process, the basic environment for compiling map data based on cloud-native technology enables storage and computing separation. Furthermore, when the computing resources in the CVM cluster are insufficient or there are faulty computing devices, the expansion or reduction of computing devices can be achieved quickly and automatically.

[0196] To illustrate, when a CVM cluster needs to be scaled down, such as when a computing device in the CVM cluster needs to be removed, you only need to delete the data row containing the computing device in the device information table and unmount the computing device from the cloud disk. Alternatively, you can mark the computing device to be removed as unavailable in the device information table, which can also avoid assigning new compilation tasks to the computing device to be removed.

[0197] To illustrate, if any computing device in the CVM cluster fails (due to network outages, machine crashes, etc.), and if the device is not idle, the target compilation task already assigned to the failed device will stop executing in the compiler operator, i.e., the target compilation task will be blocked. Simultaneously, the compiler operator will report the failure to the scheduling decision service. The scheduling decision service will then mark the failed device as unavailable in the device information table, thus preventing the assignment of new compilation tasks to the failed device. Finally, the scheduling decision service will reassign an available target computing device from the CVM cluster for the target compilation task and restart the task through the compiler operator. This prevents the entire compilation process from being blocked due to the target compilation task being blocked; only a small portion of the target compilation task needs to be redone to continue the compilation process. This avoids the entire compilation process being rolled back or redone due to the blocking of a single compilation task, greatly improving the compilation success rate and efficiency in the event of a failure.

[0198] To illustrate, when a CVM cluster needs to be expanded—that is, when a first computing device (a newly added computing device during expansion) is added to the CVM computing cluster—the scheduling decision service can add the first computing device to the device information table. Then, it performs an automatic initialization operation on the first computing device. Optionally, this automatic initialization operation includes two steps: a) mounting the cloud disk on the first computing device; b) setting the environment variables that the compilation tasks depend on for the first computing device. After the initialization operation is complete, when the compiler operator starts a new compilation task, the scheduling decision service dynamically reads the configuration information of the newly added first computing device and determines whether to allocate the current compilation task to the first computing device through the process scheduling strategy. This is equivalent to adding the first computing device to the candidate task processing queue.

[0199] Under the aforementioned storage-compute separation system architecture, by setting up a scheduling decision service module, an automated scheduling mechanism for computing devices can be implemented, and the stability of data update and compilation efficiency can be improved. This enables rapid iteration of data and strategies, and the latest and best compilation results can be provided to relevant business servers as quickly as possible, thereby indirectly improving the user experience of application map services, projects, or products.

[0200] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0201] The method provided in this application dynamically allocates computing devices in real time according to the load request within the system by compiling map service requests, and triggers the computing devices to start compilation tasks by remotely executing commands. Since the computing devices are mounted to the cloud disk, they can read and write map data in the cloud disk to achieve remote compilation. Since the computing devices bound to each compilation task can read and write map data in the cloud disk by being mounted to the cloud disk, it avoids the need to copy data between the physical machines of upstream and downstream tasks when the compilation task is bound to a physical machine, which simplifies the map data compilation process and greatly improves the compilation efficiency of map data.

[0202] Furthermore, since both the computing cluster where the computing devices reside and the storage cluster where the cloud disks reside are built on a cloud-native environment-based storage-compute separation compilation platform, they can guarantee the provision of sufficient computing and storage resources. This improves upon the traditional construction scheme based on large cores and large storage physical machines combined with process engines, which suffers from insufficient computing resources and coupling of computing and storage resources on the same physical machine.

[0203] Furthermore, based on the storage-compute separation system architecture, since map data is uniformly stored in the cloud disk, only the software environment and compilation tools required for the compilation task need to be deployed in the cloud disk. This enables one-time deployment and sharing across all computing devices, greatly reducing the deployment difficulty of the compilation-dependent software environment and improving the situation of difficult deployment and migration of the compilation environment.

[0204] Furthermore, since computing resources such as CPU and memory and storage resources such as storage space can be planned separately, there is no need to increase the bundled storage resources just because more computing resources are needed. Appropriate computing and storage resources can be requested according to actual compilation needs, which realizes full utilization of computing and storage resources, improves resource utilization, and reduces hardware costs.

[0205] Furthermore, since only one copy of the data needs to be persistently stored in the cloud disk and the directory is organized by mounting it with a soft link, the data is generated only once and occupies only one storage space. There is no need to transfer or copy data across computing devices, and duplicate or redundant data will not be stored, thus minimizing the storage space occupied.

[0206] Furthermore, since each computing device within the computing cluster only needs to be mounted to the cloud disk and have the corresponding environment variables deployed, when there is a need for expansion, the entire process from requesting computing devices and initialization to adding computing devices to the computing cluster can be automated through script tools. This provides a foundation for the rapid and automated expansion of computing devices, reducing the manpower and time costs of expanding the computing cluster. Similarly, when there is a need for scaling down or a faulty computing device is discovered, the computing device can be automatically and quickly removed from the computing cluster or marked as unavailable. This allows for flexible scheduling of computing devices, automatic avoidance of faulty devices, and adjustments to the compilation process, automating the expansion or scaling down of the cluster. This improves the scalability and extensibility of the computing cluster and reduces the maintenance cost and difficulty of the entire system.

[0207] Furthermore, by establishing a computing cluster and providing a flexible and intelligent scheduling mechanism through the process scheduling device, on the one hand, it can make full use of the idle computing resources within the computing cluster, improve the parallelism of compilation, and enhance compilation efficiency. On the other hand, it can also effectively identify computing devices that have malfunctioned and quickly switch the affected compilation tasks to new computing devices for re-execution, so that the process will not be blocked due to the failure of a certain computing device, thus mitigating the impact of the failure of a single node within the cluster.

[0208] Furthermore, relying on the cloud-native environment ensures the stability of the compilation process, enabling data and strategy iterations to take effect stably and quickly online. For example, when computing clusters are divided according to business types, the production and testing environments can be separated, requiring any adjustments to the compilation process to undergo complete testing before being pushed to the production environment. The production environment is handled by an independent production compilation cluster and is not subject to resource competition or impact from testing or other projects. Moreover, cloud computing resources and the underlying environment are more stable and reliable than offline maintained environments, and the anomaly detection and repair mechanisms are more comprehensive.

[0209] Figure 8 This is a schematic diagram of the structure of a map data compilation device provided in an embodiment of this application. Please refer to it. Figure 8 The device includes:

[0210] The acquisition module 801 is used to acquire at least one compilation task associated with the map service request in response to the map service request;

[0211] The allocation module 802 is used to allocate a computing device for processing the compilation task to any one of the at least one compilation tasks;

[0212] The sending module 803 is used to send a remote execution command for the compilation task to the computing device. The remote execution command is used to instruct the computing device to execute the compilation task using map data stored in the cloud disk.

[0213] Return module 804 is used to return the compilation result obtained by at least one computing device based on the remote execution command.

[0214] The apparatus provided in this application dynamically and in real time allocates computing devices according to the load request within the system by compiling map service requests, and triggers the computing devices to start compilation tasks by remotely executing commands. Since the computing devices are mounted to the cloud disk, they can read and write map data in the cloud disk to achieve remote compilation. Since the computing devices bound to each compilation task can read and write map data in the cloud disk by being mounted to the cloud disk, it avoids the need to copy data between the physical machines of upstream and downstream tasks when the compilation task is bound to a physical machine, which simplifies the compilation process of map data and greatly improves the compilation efficiency of map data.

[0215] In some embodiments, based on Figure 8 The device comprises, wherein the distribution module 802 includes:

[0216] The determination unit is used to determine the computing resources required to process the compilation task based on the compilation information table, which records the computing resources required by multiple compilation tasks.

[0217] The determining unit is also used to determine the idle resources of each of the multiple computing devices in the computing cluster based on the device information table;

[0218] An allocation unit is used to allocate computing devices with more free resources than the computing resources for the compilation task.

[0219] In some embodiments, where the compilation task is associated with a specific device, the allocation unit is used for:

[0220] If the available resources of a specific device are greater than the computing resources, the compilation task will be assigned to that specific device.

[0221] If the available resources of a particular device are less than or equal to the computing resources, the compilation task will enter a waiting state.

[0222] In some embodiments, where the compilation task is not associated with any particular device, based on Figure 8 The device comprises:

[0223] The first allocation subunit is used to allocate the non-specific device that meets the first resource condition to the compilation task;

[0224] The second allocation subunit is used to allocate a specific device that meets the second resource condition to the compilation task if there is no non-specific device that meets the first resource condition in the computing cluster.

[0225] The waiting subunit is used to put the compilation task into a waiting state if the specific device that meets the second resource condition does not exist in the computing cluster.

[0226] In some embodiments, the first allocation subunit is used for:

[0227] If an idle non-specific device exists in the computing cluster, allocate that idle non-specific device to the compilation task;

[0228] If no idle non-specific device exists in the computing cluster, allocate the compilation task to a non-specific device with idle resources greater than the computing resources.

[0229] In some embodiments, the second allocation subunit is used for:

[0230] Allocate a specific device that is idle and associated with a downstream task of the compilation task, which is a task that can only be executed after the compilation task has been completed;

[0231] If there is no idle specific device in the computing cluster that is associated with the downstream task of the compilation task, allocate the specific device with idle resources greater than the computing resources and associated with the downstream task of the compilation task to the compilation task.

[0232] In some embodiments, based on Figure 8 The device comprises, and the distribution module 802 further includes:

[0233] The routing unit is used to route the compilation task to a computing cluster that matches the service type of the map service request.

[0234] In some embodiments, based on Figure 8 The device comprises, and the distribution module 802 further includes:

[0235] The routing unit is used to route the compilation task to a computing cluster that supports splitting the compilation task into multiple subtasks, provided that the compilation task does not depend on global map data.

[0236] The routing unit is also used to route the compilation task to a computing cluster whose computing resources meet the third resource condition when the compilation task depends on global map data.

[0237] In some embodiments, based on Figure 8 The device comprises, and the distribution module 802 further includes:

[0238] The routing unit is used to determine the API service associated with the compilation task and to route the compilation task to the computing cluster associated with the API service.

[0239] In some embodiments, based on Figure 8 The device comprises:

[0240] A blocking module is used to block a target compilation task, which is a compilation task assigned to the failed computing device, in the event of a failure of any computing device.

[0241] The marking module is used to mark the malfunctioning computing device as unavailable in the device information table;

[0242] The allocation module 802 is also used to reallocate the target computing device that is in an available state for the target compilation task.

[0243] In some embodiments, based on Figure 8 The device comprises:

[0244] The module adds a first computing device to the device information table when a first computing device is added to the computing cluster.

[0245] A mounting module is used to mount the cloud disk on the first computing device;

[0246] The configuration module is used to set the environment variables that the compilation task depends on for the first computing device.

[0247] In some embodiments, the computing devices used to process any map service request compilation task are all mounted to the cloud disk, which refers to a storage cluster consisting of multiple storage devices, and the storage cluster is used to store the map data required for the compilation task.

[0248] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0249] It should be noted that the map data compilation device provided in the above embodiments is only illustrated by the division of the above functional modules when controlling the compilation of map data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the map data compilation device and the map data compilation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the map data compilation method embodiments, which will not be repeated here.

[0250] Figure 9 This is a schematic diagram of the structure of a computer device 900 provided in an embodiment of this application. The computer device 900 can be provided as a process scheduling device or any computing device in a computing cluster. The computer device 900 can vary significantly due to differences in configuration or performance. The computer device 900 includes one or more Central Processing Units (CPUs) 901 and one or more memories 902. The memories 902 store at least one computer program, which is loaded and executed by the one or more processors 901 to implement the map data compilation methods provided in the various embodiments described above. Optionally, the computer device 900 also has wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 900 also includes other components for implementing device functions, which will not be elaborated here.

[0251] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which can be executed by a processor in a terminal to complete the map data compilation method in the various embodiments described above. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0252] In an exemplary embodiment, a computer program product is also provided, including at least one computer program stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the at least one computer program from the computer-readable storage medium, and the one or more processors execute the at least one computer program, enabling the computer device to perform a method for compiling map data as described in the above embodiments.

[0253] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. Optionally, the program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0254] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of compiling map data, characterized by, The method includes: In response to a map service request, obtain at least one compilation task associated with the map service request; For any one of the at least one compilation task, a computing device is allocated to the compilation task for processing the compilation task; Send a remote execution command for the compilation task to the computing device, the remote execution command being used to instruct the computing device to execute the compilation task using map data stored in the cloud disk; Return the compilation result obtained by at least one computing device based on the remote execution command; The computing devices used to process any map service request compilation task are all mounted to the cloud disk, which is a shared storage cluster composed of multiple storage devices. The shared storage cluster is used to store the map data required for the compilation task. The computing devices in the computing cluster are all mounted to the shared storage cluster. The input data source and result data in the compilation process are all stored in the shared storage cluster. The computing devices do not need to transfer or copy data to each other when executing the compilation task.

2. The method according to claim 1, characterized in that, The computing device allocated for processing the compilation task includes: Based on the compilation information table, the computing resources required to process the compilation task are determined. The compilation information table is used to record the computing resources required by multiple compilation tasks. Based on the device information table, determine the available resources of each computing device in the computing cluster; Allocate computing devices with more free resources than computing resources to the compilation task.

3. The method of claim 2, wherein, When the compilation task is associated with a specific device, the computing device for which the compilation task is allocated idle resources greater than the computing resources includes: If the idle resources of the specific device are greater than the computing resources, the compilation task will be assigned to the specific device; If the idle resources of the specific device are less than or equal to the computing resources, the compilation task is put into a waiting state.

4. The method of claim 2, wherein, In cases where the compilation task is not associated with any specific device, the computing device for allocating idle resources greater than the computing resources to the compilation task includes: Allocate a non-specific device that meets the first resource condition to the compilation task; If no non-specific device meeting the first resource condition exists in the computing cluster, a specific device meeting the second resource condition is allocated to the compilation task. If no specific device meeting the second resource condition exists in the computing cluster, the compilation task is put into a waiting state.

5. The method of claim 4, wherein, The allocation of non-specific devices that meet the first resource condition for the compilation task includes: If there are idle non-specific devices in the computing cluster, the idle non-specific devices shall be allocated to the compilation task; If no idle non-specific devices exist in the computing cluster, the compilation task is allocated a non-specific device with idle resources greater than the computing resources.

6. The method of claim 4, wherein, The specific device that meets the second resource condition for allocating the compilation task includes: Allocate a specific device that is idle and associated with a downstream task of the compilation task to the compilation task, wherein the downstream task is a task that can only be executed after the compilation task has been completed; If there is no idle specific device in the computing cluster that is associated with a downstream task of the compilation task, allocate a specific device to the compilation task that has more idle resources than the computing resources and is associated with a downstream task of the compilation task.

7. The method of claim 2, wherein, Before determining the idle resources of each computing device in the computing cluster based on the device information table, the method further includes: The compilation task is routed to a computing cluster that matches the service type of the map service request.

8. The method of claim 2, wherein, Before determining the idle resources of each computing device in the computing cluster based on the device information table, the method further includes: If the compilation task does not depend on global map data, the compilation task is routed to a computing cluster that supports splitting the compilation task into multiple subtasks; If the compilation task depends on global map data, the compilation task will be routed to a computing cluster whose computing resources meet the third resource condition.

9. The method according to claim 2, characterized in that, Before determining the idle resources of each computing device in the computing cluster based on the device information table, the method further includes: Determine the API service associated with the compilation task; The compilation task is routed to the computing cluster associated with the API call service.

10. The method according to claim 1, characterized in that, The method further includes: In the event of a failure of any computing device, the target compilation task, which is the compilation task assigned to the failed computing device, is blocked. In the device information table, the computing device that has malfunctioned is marked as unavailable; The target computing device that is in an available state is reassigned to the target compilation task.

11. The method according to claim 1, characterized in that, The method further includes: When adding a first computing device to the computing cluster, the first computing device is added to the device information table; Mount the cloud disk on the first computing device; Set the environment variables that the compilation task depends on for the first computing device.

12. A map data compilation apparatus, characterized in that, The device includes: The acquisition module is used to acquire at least one compilation task associated with the map service request in response to the map service request; An allocation module is configured to allocate a computing device for processing the compilation task to any one of the at least one compilation tasks. The sending module is used to send a remote execution command for the compilation task to the computing device, wherein the remote execution command is used to instruct the computing device to execute the compilation task using map data stored in the cloud disk; The return module is used to return the compilation result obtained by at least one computing device based on the remote execution command; The computing devices used to process any map service request compilation task are all mounted to the cloud disk, which is a shared storage cluster composed of multiple storage devices. The shared storage cluster is used to store the map data required for the compilation task. The computing devices in the computing cluster are all mounted to the shared storage cluster. The input data source and result data in the compilation process are all stored in the shared storage cluster. The computing devices do not need to transfer or copy data to each other when executing the compilation task.

13. The apparatus according to claim 12, characterized in that, The allocation module includes: The determining unit is used to determine the computing resources required to process the compilation task based on the compilation information table, wherein the compilation information table is used to record the computing resources required by multiple compilation tasks. The determining unit is also used to determine the idle resources of each of the multiple computing devices in the computing cluster based on the device information table; An allocation unit is used to allocate computing devices with more free resources than computing resources to the compilation task.

14. The apparatus according to claim 13, characterized in that, When the compilation task is associated with a specific device, the allocation unit is used for: If the idle resources of the specific device are greater than the computing resources, the compilation task will be assigned to the specific device; If the idle resources of the specific device are less than or equal to the computing resources, the compilation task is put into a waiting state.

15. The apparatus according to claim 13, characterized in that, In cases where the compilation task is not associated with any specific device, the allocation unit includes: The first allocation subunit is used to allocate a non-specific device that meets the first resource condition to the compilation task; The second allocation subunit is used to allocate a specific device that meets the second resource condition to the compilation task when there is no non-specific device that meets the first resource condition in the computing cluster. The waiting subunit is used to cause the compilation task to enter a waiting state if no specific device meeting the second resource condition exists in the computing cluster.

16. The apparatus according to claim 15, characterized in that, The first allocation subunit is used for: If there are idle non-specific devices in the computing cluster, the idle non-specific devices shall be allocated to the compilation task; If no idle non-specific devices exist in the computing cluster, the compilation task is allocated a non-specific device with idle resources greater than the computing resources.

17. The apparatus according to claim 15, characterized in that, The second allocation subunit is used for: Allocate a specific device that is idle and associated with a downstream task of the compilation task to the compilation task, wherein the downstream task is a task that can only be executed after the compilation task has been completed; If there is no idle specific device in the computing cluster that is associated with a downstream task of the compilation task, allocate a specific device to the compilation task that has more idle resources than the computing resources and is associated with a downstream task of the compilation task.

18. The apparatus according to claim 13, characterized in that, The allocation module further includes: The routing unit is used to route the compilation task to a computing cluster that matches the service type of the map service request.

19. The apparatus according to claim 13, characterized in that, The allocation module further includes: The routing unit is used to route the compilation task to a computing cluster that supports splitting the compilation task into multiple subtasks, provided that the compilation task does not depend on global map data. The routing unit is further configured to route the compilation task to a computing cluster whose computing resources meet the third resource condition when the compilation task depends on global map data.

20. The apparatus according to claim 13, characterized in that, The allocation module further includes: The routing unit is used to determine the calling interface service associated with the compilation task and to route the compilation task to the computing cluster associated with the calling interface service.

21. The apparatus according to claim 12, characterized in that, The device further includes: A blocking module is used to block a target compilation task, which is a compilation task assigned to the failed computing device, in the event of a failure of any computing device. The marking module is used to mark the malfunctioning computing device as unavailable in the device information table; The allocation module is also used to reallocate target computing devices that are in an available state for the target compilation task.

22. The apparatus according to claim 12, characterized in that, The device further includes: The module adds a first computing device to the device information table when a first computing device is added to the computing cluster. A mounting module is used to mount the cloud disk on the first computing device; The configuration module is used to set the environment variables that the compilation task depends on for the first computing device.

23. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the method for compiling map data as described in any one of claims 1 to 11.

24. A storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method for compiling map data as described in any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product includes at least one computer program, which is loaded and executed by a processor to implement the method for compiling map data as described in any one of claims 1 to 11.

Images