A multi-core heterogeneous computing framework and a construction machine

Abstract

The application discloses a kind of multi-core heterogeneous computing framework and engineering machinery, wherein multi-core heterogeneous computing framework includes application layer, public component layer, adaptation layer and heterogeneous engine layer;Application layer includes configuration manager and compilation manager, configuration manager is used to generate configuration file based on the application scene obtained, and compilation manager is used to compile configuration file to generate executable file;Public component layer contains one or more chip platforms public abstract class, and public interface is included in public abstract class;Adaptation layer includes one or more chip platforms adapter, and one or more chip platforms adapter is connected with public interface in communication;Heterogeneous engine layer includes one or more chip platforms engine library, and the engine library of each chip platform is connected with its corresponding adapter in communication.Through the multi-core heterogeneous computing framework described above, application program can be obtained only by obtaining the application scene input by user, and application program can be quickly deployed on different computing platforms.

Description

Technical Field

[0001] This invention relates to the field of control technology for engineering machinery, specifically to a multi-core heterogeneous computing framework and engineering machinery. Background Technology

[0002] The onboard computing platform for construction machinery is the brain and soul of construction machinery vehicles. It is mainly responsible for the perception, calculation, planning and control during the operation and work of construction machinery vehicles, such as the fusion perception of camera image signals and radar signals, map modeling and path planning, vehicle driving planning and control, and planning and control of engineering operation.

[0003] Computing platforms need to process massive amounts of data and perform extensive parallel computing. Therefore, different computing units are integrated into the platform, such as CPUs responsible for system scheduling and control, DSPs (Digital Signal Processors) for digital image processing, and GPUs or NPUs (Digital Processing Units) for accelerating deep learning model inference. Typically, the main control chip, the CPU, acts as the manager, scheduling different computing units to execute functions with different characteristics in order to optimize performance, improve efficiency, reduce costs, and ensure stable operation. Summary of the Invention

[0004] To address the need for multi-task parallel operation on the onboard computing platform of construction machinery, this invention provides a multi-core heterogeneous computing framework and construction machinery.

[0005] According to a first aspect, embodiments of the present invention provide a multi-core heterogeneous computing framework, including an application layer, a common component layer, an adaptation layer, and a heterogeneous engine layer. The application layer includes a configuration manager and a compilation manager. The configuration manager generates a configuration file based on an acquired application scenario, and the compilation manager compiles the configuration file to generate an executable file. The common component layer includes a common abstract class for one or more chip platforms, wherein the common abstract class includes a common interface. The adaptation layer includes adapters for the one or more chip platforms, and each adapter is communicatively connected to the common interface. The heterogeneous engine layer includes engine libraries for the one or more chip platforms, wherein each chip platform's engine library is communicatively connected to its corresponding adapter.

[0006] Specifically, the input to the executable file is an AI model.

[0007] Specifically, the public component layer also includes a model conversion module, which is used to convert and quantize the AI ​​model.

[0008] Specifically, when the application based on the multi-core heterogeneous computing framework is running, the configuration manager is also used to: generate a pipeline based on the configuration file, wherein the pipeline includes multiple computing cores arranged in sequence and identification information of each computing core; the configuration manager connection is also used to, when the executable file is running, call the computing cores corresponding to the pipeline in the order of the pipeline through the adaptation layer and the heterogeneous engine layer.

[0009] Specifically, the common component layer also includes an acceleration module for accelerating the pipeline.

[0010] Specifically, the acceleration module is used to set up a buffer queue between two adjacent computing cores, and the buffer queue adopts a shared memory approach.

[0011] Specifically, the common component layer also includes a management module for memory management and scheduling management.

[0012] Specifically, the engine library of any chip platform includes interfaces for the multiple computing cores contained in that chip.

[0013] Specifically, the heterogeneous engine layer has the ability to access different chip platforms.

[0014] According to a second aspect, embodiments of the present invention also provide an engineering machine, including the multi-core heterogeneous computing framework described in the first aspect or any one of the first aspects.

[0015] The multi-core heterogeneous computing framework and engineering machinery provided in this invention include an application layer, a common component layer, an adaptation layer, and a heterogeneous engine layer. Specifically, the application layer includes a configuration manager and a compilation manager. The configuration manager generates a configuration file based on the acquired application scenario, and the compilation manager compiles the configuration file to generate an executable file. The common component layer contains a common abstract class for one or more chip platforms, wherein the common abstract class includes a common interface. The adaptation layer includes adapters for the one or more chip platforms, and each adapter communicates with the common interface. The heterogeneous engine layer includes engine libraries for the one or more chip platforms, wherein each chip platform's engine library communicates with its corresponding adapter. This multi-core heterogeneous computing framework can generate an application program simply by obtaining the user-input application scenario, enabling rapid deployment of different applications on different computing platforms to meet the onboard operation requirements of various engineering machinery vehicles. Attached Figure Description

[0016] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0017] Figure 1 A schematic diagram of the multi-core heterogeneous computing framework is shown.

[0018] Figure 2 A diagram illustrating an example of generating a configuration file for the configuration manager;

[0019] Figure 3 A diagram illustrating an example of configuration file parsing for a configuration manager;

[0020] Figure 4 This is a schematic diagram of a buffer queue in a pipeline.

[0021] Figure 5 This is a schematic diagram of the buffering mechanism in the buffer queue;

[0022] Figure 6 This is a timing diagram for a single-core multi-stage pipeline.

[0023] Figure 7 This is a timing diagram for a multi-core, multi-stage pipeline. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To address the need for multi-task parallel operation in onboard computing platforms for construction machinery, this invention provides a multi-core heterogeneous computing framework. It should be noted that the computing unit used to perform computing tasks can be called a computing core, where CPU, DSP, GPU, and NPU can be referred to as computing cores. "Multi-core" refers to the presence of multiple computing cores in the computing platform, and "heterogeneous" refers to the different structures for different construction machinery computing platforms.

[0026] Typically, a chip platform includes one CPU, or a chip platform includes one CPU and multiple other computing cores, such as DSPs and GPUs. The multi-core heterogeneous computing framework of this invention is based on a multi-core heterogeneous computing platform (also simply referred to as a computing platform), which may include one or more chip platforms. The types and number of chip platforms included in the computing platform may vary depending on the type of engineering machinery.

[0027] It should be noted that different construction machinery can refer to different types of construction machinery, such as trucks and excavators, or different individual construction machinery, such as excavator A and excavator B.

[0028] Figure 1 This is a schematic diagram of the structure of the multi-core heterogeneous computing framework in an embodiment of the present invention, as shown below. Figure 1 As shown, the multi-core heterogeneous computing framework includes an application layer (i.e., the framework application layer), a common component layer, an adaptation layer, and a heterogeneous engine layer. The application layer includes a configuration manager (i.e., an application configuration manager) and a compilation manager. The configuration manager generates configuration files based on the obtained application scenario, and the compilation manager compiles the configuration files to generate executable files. The common component layer contains common abstract classes for one or more chip platforms, where the common abstract classes include common interfaces. The adaptation layer includes adapters for one or more chip platforms, and each adapter communicates with the common interface. The heterogeneous engine layer includes engine libraries for one or more chip platforms, where each chip platform's engine library communicates with its corresponding adapter.

[0029] It should be noted that different types of construction machinery correspond to different application scenarios. For example, trucks and excavators correspond to different application scenarios. Even for the same type of construction machinery, the corresponding application scenarios may be different. For example, for an excavator, there is one application scenario when it is moving forward and another application scenario when it is moving backward.

[0030] The multi-core heterogeneous computing framework of this invention includes an application layer, a common component layer, an adaptation layer, and a heterogeneous engine layer. The application layer includes a configuration manager and a compilation manager. The configuration manager generates configuration files based on the acquired application scenario, and the compilation manager compiles the configuration files to generate executable files. The common component layer contains common abstract classes for one or more chip platforms, each including a common interface. The adaptation layer includes adapters for one or more chip platforms, all of which communicate with the common interfaces. The heterogeneous engine layer includes engine libraries for one or more chip platforms, each communicating with its corresponding adapter. This multi-core heterogeneous computing framework allows for the creation of applications simply by acquiring the user-input application scenario, enabling rapid deployment of applications across different computing platforms.

[0031] For example, such as Figure 2 As shown, the configuration manager generates two target tasks based on the obtained application scenario: target detection task and point cloud segmentation task, and generates corresponding configuration files.

[0032] Specifically, the compiler manager can use any of the existing technologies to compile the configuration file into an executable file, which will not be elaborated here.

[0033] Specifically, during program execution based on a multi-core heterogeneous computing framework, the configuration manager is also used to: generate pipelines based on configuration files, wherein the pipelines include multiple nodes arranged in sequence and identification information of the computing cores corresponding to each node; the configuration manager is also used to call the computing cores corresponding to the pipelines in the pipeline order through the adaptation layer and the heterogeneous engine layer when the executable file is running.

[0034] For example, when running an application based on a multi-core heterogeneous computing framework, the configuration file needs to be interpreted. The specific role of the configuration manager is as follows: Figure 3 As shown, Figure 3 The configuration file on the left contains input / output configurations and intermediate computation core configurations. This configuration is generated after parsing. Figure 3 The pipeline on the right shows the data flow as follows: In -> Node A1 -> Node A2 -> Node A3 -> Out. It obtains the identification information of the computing cores corresponding to each node from the configuration file. The computing cores corresponding to a node are those capable of completing the computing tasks for that node. Finally, an application matching the application scenario is generated. Therefore, the process of generating and applying the application is simple and convenient; the user only needs to input the application scenario.

[0035] In other words, the framework application layer mainly implements configuration management, configuration file parsing, and application scenario generation, and is directly provided to users. Simultaneously, the framework application layer also includes one-click compilation and deployment scripts, facilitating product deployment for users. Therefore, the multi-core heterogeneous computing framework provided by this embodiment of the invention has the following advantages: a simple and easy-to-use user interface; convenient and quick management of application scenarios through configuration files; user-friendly and convenient; and one-click compilation, running, and deployment for rapid productization.

[0036] Specifically, the input to the executable file is an AI model. In other words, in the multi-core heterogeneous computing framework provided in this embodiment of the invention, the AI ​​model can be easily replaced. The AI ​​model is a combination of parameters and architecture. By importing a large amount of data into a model with a large number of parameters, the machine performs tasks such as "cloze tests" to deeply learn the features and structures contained in the data. Ultimately, it is trained into an artificial intelligence model with logical reasoning and analytical capabilities, i.e., an AI model.

[0037] Specifically, the public component layer also includes a model conversion module, which is used to convert and quantize AI models. The conversion and quantization process converts the 32-bit floating-point parameters in the AI ​​model into 16-bit floating-point or 8-bit integer (or even 4-bit or 2-bit integer) data, while maintaining the model's inference accuracy, thereby reducing bandwidth and computational load.

[0038] Specifically, the common component layer also includes an acceleration module for pipeline-based acceleration management.

[0039] Specifically, the acceleration module is used to set up a buffer queue between two adjacent computing cores, and the buffer queue uses shared memory.

[0040] The defining characteristic of multi-core heterogeneous architectures is the presence of multiple computing cores, which may also be of different types. To fully utilize the multi-core resources of heterogeneous computing architectures and maximize performance, this patented multi-stage pipeline mechanism is designed, as shown in the attached diagram. Figure 4 As shown.

[0041] (1) The pipeline application parsed from the configuration manager is bound to the computing core ID corresponding to each computing core. Each node of pipeline 1 and pipeline 2 is assigned to a different computing core.

[0042] (2) Taking pipeline 1 as an example, buffer queues are designed between the input and node A1, between node A1 and node A2, between node A2 and node A3, and between node A3 and the output.

[0043] (3) Buffer queues all use shared memory, which enables multi-threaded and multi-process data sharing.

[0044] Specifically, a buffer queue is a first-in, first-out (FIFO) buffering mechanism that can speed up data transfer and interaction between different cores and reduce the overhead of memory copying.

[0045] like Figure 5 As shown, data is read from and deleted from the head of the buffer queue, and written to the rear. Its function is to allow two computing cores to run independently and in parallel, consuming data from the buffer queue without waiting, thus improving parallel efficiency and resource utilization of each computing core. For example, when node A2 reads data D0 from the head of the buffer queue, node A1 will only write data D2 to the rear of the queue, without overwriting D0 which is currently being used by node A2. The buffer queue ensures both the ability for parallel operation between nodes A1 and A2 and the accuracy of the data, preventing memory overflow and data modification.

[0046] Furthermore, a maximum length can be set for each buffer queue, thereby preventing the buffer queue from growing indefinitely. For example, when the maximum length is set for the buffer queue between node A1 and node A2, node A1 will discard the current data when the buffer queue is full, preventing the buffer queue from growing indefinitely; at the same time, it will remind the user that data loss has occurred, requiring attention to the system's operating status and timely adjustment of the system load to prevent prolonged data loss.

[0047] Specifically, the common component layer also includes a management module for memory management and scheduling management. This scheduling management function can include managing the acceleration module.

[0048] Furthermore, the timing relationships of multi-stage pipelines can be divided into single-core and multi-core cases.

[0049] Scenario 1:

[0050] The pipeline timing relationship for a single core is as follows: Figure 6 As shown, each node of pipeline 1 runs in parallel on a single-core ARM_0, NPU_0, and DSP_0.

[0051] At time T1, node A1 runs on ARM_0, while NPU_0 and DSP_0 are temporarily idle. Time T1 is the initial moment of pipeline operation.

[0052] At time T2, node A1 runs on ARM_0, node A2 runs on NPU_0, and DSP_0 is idle. Here, time T2 is time T1 plus a preset first duration.

[0053] At time T3, node A1 runs on ARM_0, node A2 runs on NPU_0, and node A3 runs on DSP_0. Time T3 is time T2 plus a preset second duration.

[0054] After time T3, when the pipeline is running normally, node A1 runs on ARM_0, node A2 runs on NPU_0, and node A3 runs on DSP_0. That is, each computing core ARM_0, NPU_0, and DSP_0 is running its corresponding node normally, and there will be no idle situation where they wait for each other.

[0055] The total pipeline runtime is calculated based on the maximum runtime of each computing core. For example, if node A1 runs for 10ms, node A2 runs for 20ms, and node A3 runs for 15ms, then the average pipeline runtime is calculated based on node A2's runtime of 20ms, and its data processing rate is 50 frames of data per second.

[0056] Scenario 2:

[0057] Multi-core pipelines such as Figure 7 As shown, it increases the number of computing cores for each type in the case of a single core, such as... Figure 7 In the scenario shown, the number of ARM cores, NPU cores, and DSP cores are all 2. (Comparison) Figure 6 A single-core pipeline can execute two pipeline tasks in parallel, while a multi-core pipeline can execute two pipeline tasks in parallel without affecting each other's efficiency.

[0058] In summary, the common component layer primarily implements a high degree of abstraction of public functions. For example, the abstract class `Com.h` provides interfaces including initialization interfaces such as `Init`, `Process`, and `Deinit`. Simultaneously, the common component layer code implements all system scheduling and management mechanisms, including memory management, data exchange, and multi-level pipeline parallel acceleration mechanisms, and calls the interfaces of the lower-level adaptation layer. Additionally, the common component layer includes a public AI model conversion tool that can convert and quantize different AI models before finally deploying them to the product.

[0059] It should be noted that the content in the common component layer can be transferred and copied for different application scenarios. That is, in different application scenarios, only the configuration file needs to be modified to generate different tasks, and the code does not need to be adjusted and can be used directly. In addition, when the underlying heterogeneous engine layer chip is changed, the code in the common component layer can be directly applied to the new chip without the need to redevelop the code.

[0060] In this embodiment of the invention, the adaptation layer is a compatibility adaptation interface designed for different chip platforms. For example, if there are chips T, X, and J, then the code for three adaptation layers will be encapsulated into a unified common layer interface. Each adaptation layer code is compiled in a time-sharing manner; that is, at any given time, only the common layer interface Com.h and one adaptation layer code (such as TAdp.c) will be linked. This adaptation layer (such as TAdp.c) then connects to the lower-level heterogeneous engine layer library (such as TEngin.a), generating the corresponding executable file, which is then installed into the target product, achieving one-click deployment.

[0061] Specifically, the engine library of any chip platform includes interfaces to the multiple computing cores contained in that chip. For example, such as... Figure 1 As shown, the heterogeneous engine layer includes a T-chip engine library, an X-chip engine library, a J-chip engine library, and an extension engine library. The T-chip engine library corresponds to the T-chip, the X-chip engine library to the X-chip, the J-chip engine library to the J-chip, and the extension engine library to the extension chips. The T-chip, X-chip, and J-chip represent three different types of chips, while the extension chips are chips to be added. Therefore, by simply replacing or adding the AI ​​inference engine library, the AI ​​computing power card can be replaced or expanded, enabling scalable computing capabilities for the multi-core heterogeneous computing framework.

[0062] Specifically, the heterogeneous engine layer has the ability to access different chip platforms, which allows for rapid expansion or replacement of computing cards and enables different product iterations.

[0063] Therefore, the heterogeneous engine layer, as the underlying engine mechanism for AI acceleration, provides AI interfaces based on specific chips, such as scaling interfaces, color space conversion interfaces, image format conversion interfaces, model inference interfaces, and post-processing interfaces for chips like T; scaling interfaces, color space conversion interfaces, image format conversion interfaces, model inference interfaces, and post-processing interfaces for chips like X; and scaling interfaces, color space conversion interfaces, image format conversion interfaces, model inference interfaces, and post-processing interfaces for chips like J. The adaptation layer ultimately calls the library of this heterogeneous engine layer. Different chips can seamlessly integrate into the entire heterogeneous computing framework as long as they are properly adapted to the standard interfaces.

[0064] Currently, OpenVX and OpenCL have high development costs. Each time a new application scenario is generated, it requires rewriting code, debugging, and testing, making the development process complex and time-consuming. However, based on the multi-core heterogeneous computing framework architecture of this invention, different application scenarios only require modifying configuration files to obtain the corresponding application for each new application scenario; moreover, accessing new chip platforms does not require significant development and time costs.

[0065] In summary, the multi-core heterogeneous computing framework of this invention, based on a multi-core heterogeneous computing platform, has the following advantages:

[0066] 1) The concept of lightweight heterogeneous computing: The user interface is simple and easy to use. Users can manage their application scenarios through configuration files, which is convenient and quick. AI models can be easily replaced.

[0067] 2) User-friendly, convenient, and fast product deployment: One-click compilation, running, and deployment enable rapid productization.

[0068] 3) Portable and generalizable: Common code can be ported and copied, allowing for quick replacement of hardware platforms and shortening development time.

[0069] 4) Scalable computing power: AI computing power cards can be replaced or expanded simply by replacing or adding AI inference engine libraries.

[0070] 5) Computing Acceleration (AI Acceleration): Considering the characteristics of different cores working together in a multi-core heterogeneous computing platform, and taking into account factors such as shared memory, data exchange, and parallel acceleration, a multi-level pipeline mechanism is designed to achieve parallel acceleration of AI inference, make full use of limited computing resources, and enhance product competitiveness.

[0071] The multi-core heterogeneous computing framework of this invention is based on a multi-core heterogeneous computing platform, which integrates hardware modules such as CPU, DSP, MCU, and AI engine. Simultaneously, the platform is designed to be expandable with various accelerator cards (i.e., by adding chip platforms through an expanded engine library) and functional devices (such as cameras) to improve the computing power and device access capabilities of the platform, enabling it to adapt to the autonomous driving and engineering operation requirements of engineering vehicles in different application scenarios.

[0072] Furthermore, based on the aforementioned multi-core heterogeneous computing framework, this embodiment of the invention also provides an engineering machinery, which includes the aforementioned multi-core heterogeneous computing framework. Specifically, the engineering machinery includes, but is not limited to, trucks, excavators, forklifts, cranes, road rollers, etc.

[0073] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multi-core heterogeneous computing framework, characterized in that, include: The application layer includes a configuration manager and a compilation manager. The configuration manager is used to generate configuration files based on the obtained application scenario, and the compilation manager is used to compile the configuration files to generate executable files. The common component layer contains one or more common abstract classes for chip platforms, wherein the common abstract classes include common interfaces; The adaptation layer includes adapters for the one or more chip platforms, all of which are communicatively connected to the common interface. The heterogeneous engine layer includes engine libraries for the one or more chip platforms, wherein the engine library for each chip platform is communicatively connected to its corresponding adapter; When the application obtained based on the multi-core heterogeneous computing framework is running, the configuration manager is also used to: generate a pipeline based on the configuration file, wherein the pipeline includes multiple nodes arranged in sequence and identification information of the computing core corresponding to each node; The configuration manager is also used to call the computing cores corresponding to the pipeline in the pipeline order through the adaptation layer and the heterogeneous engine layer when the executable file is running; wherein the adaptation layer is a compatibility adaptation interface designed for different chip platforms. The engine library of any chip platform includes interfaces for the multiple computing cores contained in that chip.

2. The multi-core heterogeneous computing framework according to claim 1, characterized in that, The input to the executable file is an AI model.

3. The multi-core heterogeneous computing framework according to claim 2, characterized in that, The common component layer also includes a model conversion module, which is used to convert and quantize the AI ​​model.

4. The multi-core heterogeneous computing framework according to claim 1, characterized in that, The common component layer also includes an acceleration module for accelerating the pipeline.

5. The multi-core heterogeneous computing framework according to claim 4, characterized in that, The acceleration module is used to set up a buffer queue between two adjacent computing cores, and the buffer queue adopts a shared memory approach.

6. The multi-core heterogeneous computing framework according to claim 1, characterized in that, The common component layer also includes a management module for memory management and scheduling management.

7. The multi-core heterogeneous computing framework according to claim 1, characterized in that, The heterogeneous engine layer has the ability to access different chip platforms.

8. An engineering machinery, characterized in that, The multi-core heterogeneous computing framework includes any one of claims 1 to 7.

Images