An intelligent identification method and system for unmanned vehicles

Abstract

The application relates to a heterogeneous intelligent identification method and system for unmanned vehicles, and belongs to the technical field of automatic driving, and solves the real-time collaborative optimization problem of deep learning model inference calculation and video graphics rendering in the heterogeneous computing architecture in the prior art unmanned vehicle. A heterogeneous intelligent identification method for unmanned vehicles comprises the following steps: receiving point cloud data and video data of a vehicle sensor; performing quantization processing on a deep learning model, performing inference calculation on the point cloud data by using the quantized model, and obtaining target identification and situation awareness results; sending the video data to a graphics rendering engine through a PCIe link for rendering to generate an image; and finally generating a vehicle control instruction according to the target identification result, the situation awareness result and the rendered image. The application solves the problems that the inference efficiency and accuracy of the deep learning model in the existing unmanned vehicle are difficult to balance, and the real-time performance of data transmission is insufficient.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a heterogeneous intelligent recognition method and system for unmanned vehicles. Background Technology

[0002] With the rapid development of autonomous driving technology, unmanned vehicles are placing higher demands on their real-time environmental perception and decision-making capabilities. Existing unmanned vehicle systems typically employ a heterogeneous computing architecture, using deep learning models to process LiDAR point cloud data for target detection and situational awareness, while simultaneously utilizing graphics processing units (GPUs) for real-time rendering of camera video data. However, deep learning model inference computation consumes significant computing resources and cannot be dynamically adjusted according to the real-time load of the computing units, leading to increased inference latency and even frame drops under high-load scenarios. Furthermore, when point cloud data processing and video rendering tasks share the PCIe bus bandwidth, the lack of an effective traffic scheduling mechanism makes it easy for high-priority perception data and low-priority rendering data to compete for bandwidth, causing congestion in the transmission of critical perception information and affecting the real-time generation of vehicle control commands.

[0003] In implementing the embodiments of the present invention, the prior art has at least the following problems or defects: the existing quantization method cannot dynamically select the quantization mode according to the real-time load of the computing unit, which makes it difficult to balance the model inference efficiency and accuracy; the PCIe link lacks a bandwidth dynamic allocation mechanism based on data priority, which makes it difficult to guarantee the real-time transmission of key sensing data. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a heterogeneous intelligent recognition method for unmanned vehicles, in order to solve the problems of difficulty in balancing inference efficiency and accuracy of deep learning models in existing unmanned vehicles, as well as insufficient real-time data transmission.

[0005] In a first aspect, embodiments of the present invention provide a heterogeneous intelligent recognition method for unmanned vehicles, comprising: Receive raw data from vehicle sensors, including point cloud data and video data; The deep learning model is quantized, and the quantized deep learning model is used to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results. The video data is sent to the graphics rendering engine via the PCIe link, and the video data is rendered to generate a rendered image. The final recognition result is obtained based on the target recognition result, the situational awareness result, and the rendered image.

[0006] Furthermore, the quantization process for the deep learning model includes: All operators in the deep learning model are divided into high-sensitivity operators and low-tolerance operators; Collect the unit utilization rate and memory bandwidth utilization rate of the computing units deployed in the deep learning model, and calculate the current computing load value based on the unit utilization rate and memory bandwidth utilization rate; Based on the current computing power load value and the type of each operator, a target quantization mode is selected for each operator from a variety of preset quantization modes; Each operator is quantized using its corresponding target quantization mode to obtain the quantized deep learning model.

[0007] Furthermore, all operators in the deep learning model are divided into high-sensitivity operators and low-tolerance operators, including: For each operator in the deep learning model, forward sampling computation is performed to obtain the original output data of the operator and the dequantized data of the operator. The quantization error of each operator is calculated based on the original output data and the inverse quantization data of the operators. The quantization error is compared with a preset threshold. When the quantization error is greater than or equal to the preset threshold, the corresponding operator is classified as a high-sensitivity operator; When the quantization error is less than the preset threshold, the corresponding operator is classified as a low-tolerance operator.

[0008] Furthermore, based on the current computing power load value and the type of each operator, selecting a target quantization mode for each operator from a preset range of quantization modes includes: When the operator type is a highly sensitive operator, the first quantization mode is selected as the target quantization mode; When the operator type is a low-tolerance operator, if the current computing power load value is greater than or equal to the first load threshold, the third quantization mode is selected as the target quantization mode; if the current computing power load value is greater than the second load threshold and less than the first load threshold, the second quantization mode is selected as the target quantization mode; if the current computing power load value is less than or equal to the second load threshold, the first quantization mode is selected as the target quantization mode.

[0009] Furthermore, sending video data to the graphics rendering engine via the PCIe link includes: Configure a first virtual channel and a second virtual channel using the PCIe controller, assign a first priority to the first virtual channel, and assign a second priority to the second virtual channel; The bandwidth quota is preset for the first virtual channel and the second virtual channel by the bandwidth quota configuration register, and the first bandwidth quota value and the second bandwidth quota value are obtained. Determine the data type of the video data to be sent, where the data type includes high-priority data and low-priority data; The high-priority data is transmitted to the graphics rendering engine through the first virtual channel, and the low-priority data is transmitted to the graphics rendering engine through the second virtual channel.

[0010] Furthermore, the method also includes: Real-time acquisition of the current bandwidth utilization of the first virtual channel and the second virtual channel; When the transmit buffer of the first virtual channel is not empty and the current bandwidth occupancy rate of the first virtual channel reaches or exceeds the first preset threshold, a first bandwidth adjustment instruction is generated. The bandwidth quota register is modified by the first bandwidth adjustment instruction to increase the bandwidth quota value of the first virtual channel. When the transmit buffer of the first virtual channel is empty and the second virtual channel has video data to be transmitted, a second bandwidth adjustment instruction is generated, and the bandwidth quota value of the first virtual channel is allocated to the second virtual channel through the second bandwidth adjustment instruction.

[0011] Furthermore, the quantization error of each operator is calculated based on the original output data and the inverse quantization data of the operator, as shown in the following formula; ; Where N is the number of samples. The original output data of the operator, For operator inverse quantization data, This represents the operator quantization error.

[0012] Furthermore, the first quantization mode is to perform 16-bit quantization on the activation value and weight in the operator, and to perform 16-bit quantization on the bias in the operator; The second mode involves 12-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator. The third mode involves 8-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator.

[0013] Furthermore, the method also includes: When the transmit buffer of the first virtual channel changes from non-empty to empty, or when the transmit buffer of the first virtual channel changes from empty to non-empty, the bandwidth quota of the first virtual channel is restored to the first bandwidth quota value, and the bandwidth quota of the second virtual channel is restored to the second bandwidth quota value.

[0014] Secondly, embodiments of the present invention provide a heterogeneous intelligent recognition system for unmanned vehicles, the system comprising: The data processing module is used to receive raw data from vehicle sensors, including point cloud data and video data; and to use the target recognition result, situational awareness result, and the rendered image as the final recognition result. The quantization module is used to quantize the deep learning model and use the quantized deep learning model to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results. The sending module is used to send video data to the graphics rendering engine via the PCIe link, perform graphics rendering on the video data, and generate a rendered image.

[0015] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By dividing the operators in the deep learning model into high-sensitivity operators and low-tolerance operators, and dynamically selecting different quantization modes according to the real-time computing load value of the computing unit, an adaptive balance between model accuracy and inference efficiency is achieved. This solves the problem of increased inference latency under high load scenarios with fixed quantization strategies, and ensures the real-time performance and accuracy of point cloud data processing under different computing load conditions.

[0016] 2. By setting up dual virtual channels with different priorities through the PCIe controller and dynamically adjusting the bandwidth quota based on the send buffer status and bandwidth utilization, differentiated transmission guarantee of high-priority perception data and low-priority rendering data is achieved, avoiding transmission blockage caused by bandwidth competition for critical perception information, and improving data transmission efficiency and system reliability under heterogeneous computing architecture.

[0017] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 is a flowchart illustrating a heterogeneous intelligent identification method for unmanned vehicles according to an embodiment of the present invention. Detailed Implementation

[0019] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0020] A specific embodiment of the present invention discloses a heterogeneous intelligent recognition method for unmanned vehicles, such as... Figure 1 As shown.

[0021] Step 1: Receive raw data from vehicle sensors, including point cloud data and video data.

[0022] Vehicle sensors include devices such as lidar, cameras, and millimeter-wave radar installed on autonomous vehicles. In practice, lidar generates point cloud data by emitting laser beams and receiving reflected signals. Each point contains three-dimensional spatial coordinates and reflection intensity information. Cameras collect video data of the vehicle's surrounding environment, with the video data collected in frames. This video data can be used for target classification, traffic sign recognition, lane detection, and scene understanding.

[0023] Step 2: Quantize the deep learning model and use the quantized deep learning model to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results.

[0024] Deep learning models can be object detection models based on convolutional neural networks, such as YOLOv3 or SSD, deployed on dedicated inference computing units, such as NPUs or GPUs. Dynamic quantization refers to dynamically adjusting the quantization precision based on the current computational load and operator sensitivity to balance computational efficiency and model accuracy. Quantized models consume less memory and bandwidth, resulting in faster inference speeds. Using this model to infer point cloud data, it can identify the location, category, and motion state of obstacles as object recognition results, and understand the semantic information of the surrounding environment, such as roads, vegetation, and buildings, as situational awareness results.

[0025] Specifically, quantization of deep learning models includes: Step 201: Divide all operators in the deep learning model into high-sensitivity operators and low-tolerance operators.

[0026] Operators refer to the basic computational units in deep learning models, including convolutional layers, pooling layers, fully connected layers, activation function layers, and attention mechanism layers. High-sensitivity operators are those sensitive to quantization errors and require high precision to ensure model performance, such as the attention operator in the attention mechanism and the fully connected operator in the output layer. Low-tolerance operators are those with a high tolerance for quantization errors and can use lower precision quantization, such as ordinary convolution, pooling, and fully connected operators in intermediate layers. Further, classifying all operators in the deep learning model into high-sensitivity operators and low-tolerance operators includes: Step 2011: Perform forward sampling calculation on each operator in the deep learning model to obtain the original output data and dequantized data of the operator.

[0027] Forward sampling refers to using a small batch of sample data to propagate forward through the network, recording the output of each operator. Sample data can be randomly selected from the training set, for example, 100 images. The original output data is the output value calculated with the original precision, such as FP32. Dequantized data is used to simulate the impact of quantization operations on the operator's output precision. It is the output value obtained by first quantizing the operator and then dequantizing it back to the original precision. In practice, a quantization tool is used to quantize the operator parameters and activation values ​​to a low precision, such as INT8, and then a dequantization operation is performed to dequantize the low-precision values ​​back to the original floating-point precision, such as FP32, obtaining a floating-point value of the same data type as the original output. By quantizing and then dequantizing the operator back to the original precision, the error introduced by quantization can be reproduced in the floating-point domain. Dequantized data can be obtained by inserting quantization nodes in deep learning frameworks such as TensorFlow or PyTorch.

[0028] Step 2012: Calculate the quantization error of each operator based on the original output data and the inverse quantization data of the operators.

[0029] Quantization error measures the loss of precision in the output data caused by the quantization operation; that is, the degree of difference between the original floating-point output and the floating-point output recovered by quantization and then dequantization. The calculation formula is: , Where N is the number of samples. The original output data of the operator, For operator inverse quantization data, This represents the operator quantization error.

[0030] The average absolute error of each output element across all sampled data is calculated to reflect the overall impact of quantization on the operator's output. For example, for a convolutional layer, the absolute error of each pixel in its output feature map can be calculated, and then averaged to obtain the quantization error of that layer.

[0031] Step 2013: Compare the quantization error with a preset threshold. The preset threshold can be set based on experiments or experience, such as 0.05 or 0.1, or it can be adjusted according to the model accuracy requirements and actual test results.

[0032] Step 2014: When the quantization error is greater than or equal to the preset threshold, the corresponding operator is classified as a high-sensitivity operator. High-sensitivity operators experience significant accuracy loss after quantization, requiring a high-precision quantization mode to ensure the final model performance. When the quantization error is less than the preset threshold, the corresponding operator is classified as a low-tolerance operator. Low-tolerance operators are not sensitive to quantization and can use a lower-precision quantization mode to improve computational efficiency.

[0033] Step 202: Collect the unit utilization rate and memory bandwidth utilization rate of the computing units deployed in the deep learning model, and calculate the current computing load value based on the unit utilization rate and memory bandwidth utilization rate.

[0034] Unit utilization can be obtained in real time through hardware monitoring interfaces, such as using the xpu-smi tool of Kunlun NPU or the nvidia-smi command of NVIDIA, to collect the percentage of computing units that are in working state per unit time. Memory bandwidth utilization can be obtained by monitoring the utilization of memory read and write bandwidth through the monitoring interface.

[0035] The current computing load value is calculated using the following formula; Calculate, where α and β are weighting coefficients, for example =0.7、 =0.3, which can be adjusted according to the actual scenario. For unit utilization rate, This represents the utilization rate of video memory bandwidth. The load value reflects the current level of computing resource strain, ranging from 0 to 100%.

[0036] Step 203: Based on the current computing power load value and the type of each operator, select a target quantization mode for each operator from a variety of preset quantization modes.

[0037] The quantization modes include a first quantization mode, a second quantization mode, and a third quantization mode, each corresponding to different activation values ​​and weighted quantization bit widths. Specifically, when the operator type is a high-sensitivity operator, the first quantization mode is selected as the target quantization mode. When the operator type is a low-tolerance operator, if the current computing power load value is greater than or equal to the first load threshold, the third quantization mode is selected as the target quantization mode; if the current computing power load value is greater than the second load threshold and less than the first load threshold, the second quantization mode is selected as the target quantization mode; if the current computing power load value is less than or equal to the second load threshold, the first quantization mode is selected as the target quantization mode.

[0038] The preset first load threshold can be set to 80%, and the second load threshold can be set to 50%. When the current computing power load is greater than or equal to the first load threshold, the system is in a high-load state, and the third quantization mode is selected to maximize computational efficiency; when the current computing power load is greater than the second load threshold but less than the first load threshold, the system is in a medium-load state, and the second quantization mode is selected to balance accuracy and efficiency; when the current computing power load is less than or equal to the second load threshold, the system is in a low-load state, and the first quantization mode is selected to obtain optimal accuracy. Furthermore, The first quantization mode performs 16-bit quantization on the activation values ​​and weights in the operator, and also performs 16-bit quantization on the bias in the operator, as shown in the following equation. ; The second mode involves 12-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator, as shown in the following formula. ; The third mode involves 8-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator, as shown in the following formula. .

[0039] Where X represents the original FP32 data. and These are the minimum and maximum values ​​of the original data for this operator, respectively.

[0040] The activation values ​​and weights of the operators are converted from their original floating-point precision to integer representations with a target bit width through quantization, while the bias is maintained at 16-bit quantization to ensure numerical stability.

[0041] Step 204: Quantize each operator using the corresponding target quantization mode to obtain the quantized deep learning model.

[0042] During implementation, the Kunlun NPU's dedicated instruction set and quantization tools can be used to complete the model conversion, and the quantized model can be deployed on the NPU to finally obtain the quantized deep learning model.

[0043] Step 3: Send the video data to the graphics rendering engine via the PCIe link, perform graphics rendering on the video data, and generate a rendered image.

[0044] PCIe (Peripheral Component Interconnect Express) is a high-speed serial bus used to connect devices such as CPUs, NPUs, and GPUs (Graphics Processing Units). GPUs perform graphics rendering tasks, such as converting 3D models into 2D images and adding lighting and textures. This application efficiently transmits video data to the GPU via a PCIe link. After rendering, the GPU outputs a rendered image that can be displayed or further processed, such as overlaying a virtual model of the vehicle's surroundings onto the video screen for visualization-assisted driving or human-machine interaction.

[0045] Step 301: Configure the first virtual channel and the second virtual channel through the PCIe controller, assign a first priority to the first virtual channel, and assign a second priority to the second virtual channel.

[0046] Multiple logical channels can be created on the same physical link using PCIe virtual channels, each with independently configurable priority and bandwidth. The first virtual channel is used to transmit high-priority data, and the second virtual channel is used to transmit normal data. Priority settings can be configured through the virtual channel configuration register of the PCIe controller; for example, setting the VC number of the first virtual channel to 0 and assigning it high priority, and setting the VC number of the second virtual channel to 1 and assigning it normal priority. Data packets on the high-priority channel receive priority processing during transmission, thereby reducing latency.

[0047] Step 302: Preset bandwidth quotas for the first virtual channel and the second virtual channel through the bandwidth quota configuration register to obtain the first bandwidth quota value and the second bandwidth quota value.

[0048] By configuring bandwidth quotas, the maximum available bandwidth of each virtual channel is limited, preventing one channel from consuming excessive resources and affecting other channels. For example, the bandwidth of the first virtual channel can be set to 2.67 GB / s, and the bandwidth of the second virtual channel to 5.33 GB / s, with a ratio of 1:2. The quota values ​​are implemented by writing them into the Quality of Service (QoS) bandwidth quota register of the PCIe controller.

[0049] Step 303: Determine the data type of the video data to be sent. The data types include high-priority data and low-priority data. Data type can be determined based on packet size; for example, keyframe data can be considered high-priority, and ordinary video stream data can be considered low-priority. This can be distinguished by parsing application layer data requests: packets smaller than or equal to 64 bytes are keyframe data, and packets larger than or equal to 1 KB are ordinary video stream data. Alternatively, frame order can be used, such as setting I-frames as high-priority and P-frames and B-frames as low-priority.

[0050] Step 304: Transmit the high-priority data to the graphics rendering engine through the first virtual channel, and transmit the low-priority data to the graphics rendering engine through the second virtual channel.

[0051] Data to be transmitted is placed in the send buffer of the corresponding virtual channel. The PCIe controller schedules data according to priority and bandwidth quota, ensuring that high-priority data is sent first. For example, using weighted round-robin scheduling, where the weight is consistent with the bandwidth quota, the controller schedules data in a loop, transmitting one packet through the first virtual channel and two packets through the second virtual channel. This ensures low latency for critical data while making full use of bandwidth.

[0052] Furthermore, it also includes the following steps: The current bandwidth utilization of the first and second virtual channels is acquired in real time. By reading the bandwidth monitoring register of the PCIe controller, information such as the real-time transmission rate, the amount of data transmitted, and the link idle time of each channel can be obtained. The monitoring method combines interruption and polling. The bandwidth monitoring register is queried every 100μs. When the bandwidth utilization exceeds a preset threshold, an interrupt is triggered for timely adjustment.

[0053] When the transmit buffer of the first virtual channel is not empty and the current bandwidth occupancy rate of the first virtual channel reaches or exceeds the first preset threshold, a first bandwidth adjustment instruction is generated. The bandwidth quota register is modified by the first bandwidth adjustment instruction to increase the bandwidth quota value of the first virtual channel.

[0054] For example, if the bandwidth utilization of the first virtual channel exceeds 90% and there is still data waiting to be sent, the bandwidth quota is temporarily increased to the same level as the second virtual channel, such as 4GB / s, to ensure that high-priority data is not blocked.

[0055] When the transmit buffer of the first virtual channel is empty and the second virtual channel has video data to be transmitted, a second bandwidth adjustment instruction is generated. This instruction allocates the bandwidth quota value of the first virtual channel to the second virtual channel. At this time, the bandwidth quota of the first virtual channel is set to 0, and the bandwidth quota of the second virtual channel is set to the full bandwidth, such as 8GB / s, to maximize the transmission efficiency of batch data.

[0056] When the transmit buffer of the first virtual channel changes from non-empty to empty, or when the transmit buffer of the first virtual channel changes from empty to non-empty, the bandwidth quota of the first virtual channel is restored to the first bandwidth quota value, and the bandwidth quota of the second virtual channel is restored to the second bandwidth quota value.

[0057] When the send buffer of the first virtual channel changes from non-empty to empty, it means that all the high-priority data accumulated in the first virtual channel has been sent and the burst traffic has ended. At this time, the bandwidth resources that were temporarily allocated to improve its transmission efficiency should be reclaimed and restored to the default quota so that the second virtual channel can obtain a reasonable bandwidth share.

[0058] When the transmit buffer of the first virtual channel changes from empty to non-empty, it indicates that new high-priority video data has arrived on the first virtual channel. If the bandwidth quota of the second virtual channel was temporarily allocated to it due to channel idleness, the bandwidth usage right should be immediately revoked and the default quota restored to ensure that the newly arrived high-priority data can be transmitted in a timely manner and avoid delays in critical information.

[0059] Step 4: Use the target recognition result, situational awareness result, and rendered image as the final recognition result.

[0060] The final recognition result can be determined based on the mission requirements and application scenarios of the unmanned vehicle. For example, the final recognition result can be used for vehicle path planning, enabling autonomous driving. Furthermore, the target recognition result, situational awareness result, and rendered image can be packaged and sent to a remote monitoring center for command and dispatch or accident analysis; alternatively, the target recognition result, situational awareness result, and the rendered image can be stored in the onboard data recorder as driving data logs for subsequent algorithm training or fault diagnosis. In semi-autonomous mode with human intervention, the rendered image can be displayed on the human-machine interface, providing visual assistance for target recognition and situational awareness results, helping the driver or remote operator understand the vehicle's surrounding environment. Taking vehicle path planning based on the final recognition result as an example, the steps include: Step 401: Analyze the target recognition result to obtain obstacle information.

[0061] Target recognition results typically include the obstacle's 3D coordinates, category (e.g., vehicle, pedestrian, roadblock), and motion status (e.g., speed, direction). Obstacle information can be extracted through analysis. For example, the precise location and velocity of the obstacle can be obtained by combining the bounding box and category probability output by the YOLOv3 model with LiDAR point clouds. Alternatively, deep neural networks based on voxels or point clouds, such as PointPillars, VoxelNet, or SECOND, can output the obstacle's 3D bounding box, category probability, and orientation angle.

[0062] Step 402: Analyze the situational awareness results to obtain environmental information.

[0063] Situational awareness results include semantic information such as lane lines, curbs, and traffic signs, as well as the motion trajectories of surrounding dynamic targets, which are then parsed to obtain environmental information. For example, semantic segmentation models can be used to classify road regions, such as using fully convolutional networks (DeepLabV3+, PSPNet, etc.) to classify images and obtain semantic regions such as roads, lane lines, curbs, traffic signs, pedestrian crossings, and vegetation. For point cloud data, point-based semantic segmentation networks (PointNet++, RandLA-Net, etc.) can be used to obtain semantic labels for 3D points.

[0064] Step 403: Fuse the obstacle information with the environmental information to generate a comprehensive situational map of the vehicle's surrounding environment.

[0065] Fusion methods, such as Kalman filtering and Bayesian networks, align information from different sources to the same spatiotemporal coordinate system, forming a unified environmental model that includes both static and dynamic elements. For example, obstacle locations are mapped onto a vehicle-centric grid map, with each grid cell marked with its occupancy probability; lane lines and curbs are represented as vector lines, and dynamic targets are represented as rectangles with velocity indicators. The final result is a comprehensive 2D or 3D situational awareness map for path planning.

[0066] Based on the comprehensive situation map and a pre-defined path planning algorithm, the desired driving trajectory of the vehicle is generated. Path planning algorithms, such as A*, Dijkstra's algorithm, or sampled RRT, search for a collision-free path from the current position to the target point in the comprehensive situation map and generate a smooth trajectory considering dynamic constraints. For example, the path planning module in Model Predictive Control (MPC) can be used, combined with obstacle avoidance rules, to output a series of trajectory points, each containing position, velocity, and heading angle information.

[0067] Another specific embodiment of the present invention discloses a heterogeneous intelligent recognition system for unmanned vehicles, the system comprising: The data processing module is used to receive raw data from vehicle sensors, including point cloud data and video data; and to use the target recognition result, situational awareness result, and the rendered image as the final recognition result. The quantization module is used to quantize the deep learning model and use the quantized deep learning model to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results. The sending module is used to send video data to the graphics rendering engine via the PCIe link, perform graphics rendering on the video data, and generate a rendered image.

[0068] It is understandable that the modules described in this heterogeneous intelligent recognition system for unmanned vehicles are similar to those in the reference system. Figure 1 The steps described in the heterogeneous intelligent recognition method for autonomous vehicles correspond to each other. Therefore, the operations, features, and beneficial effects described above for the heterogeneous intelligent recognition method for autonomous vehicles are also applicable to the heterogeneous intelligent recognition system for autonomous vehicles and its included modules, and will not be repeated here.

[0069] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0070] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous intelligent recognition method for unmanned vehicles, characterized in that, include: Receive raw data from vehicle sensors, including point cloud data and video data; The deep learning model is quantized, and the quantized deep learning model is used to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results. The video data is sent to the graphics rendering engine via the PCIe link, and the video data is rendered to generate a rendered image. The target recognition result, situational awareness result, and rendered image are used as the final recognition result.

2. The method according to claim 1, characterized in that, The quantization process for the deep learning model includes: All operators in the deep learning model are divided into high-sensitivity operators and low-tolerance operators; Collect the unit utilization rate and memory bandwidth utilization rate of the computing units deployed in the deep learning model, and calculate the current computing load value based on the unit utilization rate and memory bandwidth utilization rate; Based on the current computing power load value and the type of each operator, a target quantization mode is selected for each operator from a variety of preset quantization modes; Each operator is quantized using its corresponding target quantization mode to obtain the quantized deep learning model.

3. The method according to claim 2, characterized in that, All operators in the deep learning model are divided into high-sensitivity operators and low-tolerance operators, including: For each operator in the deep learning model, forward sampling computation is performed to obtain the original output data of the operator and the dequantized data of the operator. The quantization error of each operator is calculated based on the original output data and the inverse quantization data of the operators. The quantization error is compared with a preset threshold. When the quantization error is greater than or equal to the preset threshold, the corresponding operator is classified as a high-sensitivity operator; When the quantization error is less than the preset threshold, the corresponding operator is classified as a low-tolerance operator.

4. The method according to claim 3, characterized in that, Based on the current computing power load value and the type of each operator, the target quantization mode is selected for each operator from a variety of preset quantization modes, including: When the operator type is a highly sensitive operator, the first quantization mode is selected as the target quantization mode; When the operator type is a low-tolerance operator, if the current computing power load value is greater than or equal to the first load threshold, the third quantization mode is selected as the target quantization mode; if the current computing power load value is greater than the second load threshold and less than the first load threshold, the second quantization mode is selected as the target quantization mode; if the current computing power load value is less than or equal to the second load threshold, the first quantization mode is selected as the target quantization mode.

5. The method according to claim 1, characterized in that, Sending video data to the graphics rendering engine via the PCIe link includes: Configure a first virtual channel and a second virtual channel using the PCIe controller, assign a first priority to the first virtual channel, and assign a second priority to the second virtual channel; The bandwidth quota is preset for the first virtual channel and the second virtual channel by the bandwidth quota configuration register, and the first bandwidth quota value and the second bandwidth quota value are obtained. Determine the data type of the video data to be sent, where the data type includes high-priority data and low-priority data; The high-priority data is transmitted to the graphics rendering engine through the first virtual channel, and the low-priority data is transmitted to the graphics rendering engine through the second virtual channel.

6. The method according to claim 5, characterized in that, The method further includes: Real-time acquisition of the current bandwidth utilization of the first virtual channel and the second virtual channel; When the transmit buffer of the first virtual channel is not empty and the current bandwidth occupancy rate of the first virtual channel reaches or exceeds the first preset threshold, a first bandwidth adjustment instruction is generated. The bandwidth quota register is modified by the first bandwidth adjustment instruction to increase the bandwidth quota value of the first virtual channel. When the transmit buffer of the first virtual channel is empty and the second virtual channel has video data to be transmitted, a second bandwidth adjustment instruction is generated, and the bandwidth quota value of the first virtual channel is allocated to the second virtual channel through the second bandwidth adjustment instruction.

7. The method according to claim 3, characterized in that, The quantization error of each operator is calculated based on the original output data and the inverse quantization data of the operator, as shown in the following formula; ； Where N is the number of samples. The original output data of the operator, For the operator to dequantize the data, This represents the operator quantization error.

8. The method according to claim 4, characterized in that, The first quantization mode is to perform 16-bit quantization on the activation values ​​and weights in the operator, and to perform 16-bit quantization on the bias in the operator; The second mode involves 12-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator. The third mode involves 8-bit quantization of the activation values ​​and weights in the operator, and 16-bit quantization of the bias in the operator.

9. The method according to claim 6, characterized in that, The method further includes: When the transmit buffer of the first virtual channel changes from non-empty to empty, or when the transmit buffer of the first virtual channel changes from empty to non-empty, the bandwidth quota of the first virtual channel is restored to the first bandwidth quota value, and the bandwidth quota of the second virtual channel is restored to the second bandwidth quota value.

10. A heterogeneous intelligent recognition system for unmanned vehicles, characterized in that, The system includes: The data processing module is used to receive raw data from vehicle sensors, including point cloud data and video data; and to use the target recognition result, situational awareness result, and the rendered image as the final recognition result. The quantization module is used to quantize the deep learning model and use the quantized deep learning model to perform inference calculations on the point cloud data to obtain target recognition results and situational awareness results. The sending module is used to send video data to the graphics rendering engine via the PCIe link, perform graphics rendering on the video data, and generate a rendered image.

Images