Self-adaptive multi-mode edge vehicle detection and identification system

Abstract

The invention discloses a self-adaptive multi-mode edge vehicle detection and identification system. The system is composed of a sensor collection module, an edge calculation module, a software deployment module, a communication module and an energy management module, the sensor collection module is used for obtaining images, distances and point cloud information around a vehicle, and the edge calculation module locally executes multi-modal fusion deep learning reasoning to achieve vehicle detection and recognition. The software deployment module dynamically manages reasoning tasks based on a Docker container and a lightweight arrangement system, the communication module has an event triggering mechanism, uploads original data when detection is abnormal or the confidence coefficient is low, and only transmits and recognizes abstracts in a normal state, and the energy management module automatically adjusts computing power and power consumption according to real-time loads, so that low-energy-consumption and high-performance operation is realized. According to the system, vehicle identification and classification can be completed at the edge end, the cloud computing pressure and communication delay are remarkably reduced, and the real-time performance, stability and energy efficiency in an intelligent traffic environment are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent transportation technology, and in particular to an intelligent vehicle detection and recognition system that combines edge computing and multi-sensor fusion, specifically applied to scenarios such as intelligent roads, autonomous driving and traffic management. Background Technology

[0002] With the development of Intelligent Transportation Systems (ITS), the real-time performance and reliability of vehicle detection and identification have become crucial for traffic safety and autonomous driving decisions.

[0003] Existing technologies typically rely on cloud computing platforms to centrally process large amounts of data from sensors such as cameras and millimeter-wave radar. While this provides high detection accuracy, it suffers from issues such as high network latency, high bandwidth consumption, and insufficient privacy and security.

[0004] In recent years, the introduction of edge computing technology has made it possible to perform deep learning inference near the data source. However, current edge vehicle detection systems still have the following shortcomings: It relies heavily on a single sensor and its recognition accuracy is insufficient under complex lighting or inclement weather conditions. The algorithm model is fixed, making it difficult to adjust the inference strategy according to changes in the real-time scenario; Communication bandwidth is wasted, and a large amount of redundant image or point cloud data is uploaded to the cloud. Containerized deployment lacks adaptive scheduling, which can easily lead to a waste of computing power on edge devices; It is difficult to balance energy consumption and performance, and long-term operation can lead to system overheating or instability.

[0005] Therefore, there is an urgent need for a vehicle detection and recognition system that can achieve multi-sensor fusion, adaptive model switching, intelligent energy consumption management, and event-triggered communication at the edge to improve real-time performance and reliability. Summary of the Invention

[0006] This invention aims to overcome the problems of high detection latency, high energy consumption, and high fusion complexity in existing technologies. It proposes an adaptive multimodal edge vehicle detection and recognition system, which achieves low latency, high robustness, and high energy efficiency in vehicle detection and recognition through dynamic model scheduling, multimodal fusion, event-triggered communication, and containerized deployment collaboration.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive multimodal edge vehicle detection and recognition system includes: The module comprises a sensor acquisition module, an edge computing module, a software deployment module, a communication module, and an energy management module; among which: The sensor acquisition module includes a camera, millimeter-wave radar, and lidar, used to acquire images, distance, and point cloud information of the vehicle's surrounding environment; the sensor acquisition module has a unified time synchronization interface and a multimodal data fusion intermediate layer, used to perform time alignment and format unification of multi-source data at the edge. The edge computing module uses an embedded AI computing device with a GPU, is equipped with an embedded Linux operating system and an artificial intelligence development kit, and runs a deep learning model based on a deep learning inference acceleration engine. The edge computing module dynamically selects detection models of different precision according to the complexity of the traffic scene, so as to reduce the computational load while maintaining detection accuracy. The software deployment module uses a lightweight container orchestration system to run vehicle detection and recognition tasks with application containers as the basic unit; the orchestration system can dynamically create, pause, or migrate containers according to the load of the edge computing module to achieve adaptive resource scheduling. The communication module is used to transmit the detection results generated by the edge computing module to the cloud via 4G / 5G or wired network; the communication module has an event-triggered upload mechanism, which only uploads the original data when the detection result is abnormal or the recognition confidence is lower than a preset threshold, and only transmits the detection summary information in other cases; The energy management module monitors the real-time load and temperature status of the edge computing module. Under low load conditions, it shuts down some inference containers or switches to low power mode, and under high load conditions, it automatically allocates GPU computing power to achieve dynamic energy balance.

[0008] The edge computing module includes two modes: a low-precision model and a high-precision model. The system determines the scene state by indicators such as ambient lighting, traffic density, or image complexity, and calls the low-precision model when the complexity is low and the high-precision model when the complexity is high.

[0009] The multimodal data fusion intermediate layer employs a timestamp alignment algorithm and a feature-level fusion network, including an image feature extraction subnetwork, a radar feature extraction subnetwork, and a point cloud feature extraction subnetwork. The outputs of these three subnetworks are correlated in the fusion layer and then output as vehicle category probabilities via a fully connected layer.

[0010] The software deployment module implements container orchestration based on a lightweight container orchestration system; each container encapsulates an independent deep learning model inference service and communicates with each other through a REST interface to support multi-task collaborative inference; the container orchestration system can adjust the number and concurrency of parallel inference containers according to the load.

[0011] The communication module supports a hierarchical communication mechanism, including: First layer: Local real-time feedback layer, used to output detection results to the vehicle display terminal; The second layer: the summary upload layer, which only reports vehicle location and category information; The third layer is the anomaly return layer, which is used to upload video or point cloud data corresponding to low-confidence results.

[0012] According to the system of claim 1, a feedback loop is provided between the energy management module and the edge computing module; the feedback loop adjusts the working mode according to the container GPU utilization: when the GPU utilization is below 30%, it is determined to be a low load condition, and when the GPU utilization is above 80%, it is determined to be a high load condition; and adjusts the inference frequency, batch size and thread priority in real time accordingly.

[0013] The system can automatically pull new model versions and perform hot updates within a container based on model update instructions issued from the cloud, without stopping the inference service.

[0014] The system has an adaptive control closed loop, including: Traffic conditions are detected by a sensor acquisition module; The edge computing module adjusts the inference model based on the complexity of the scenario. The software deployment module dynamically schedules containers and computing power accordingly; The energy management module adjusts the energy consumption mode according to the operating load; The communication module uploads key data according to event-triggered rules; The closed-loop execution enables the system to learn and optimize itself.

[0015] The edge computing module is configured with a two-stage inference mechanism. First, a low-precision model is used to quickly screen the image or point cloud data and score the potential vehicle target areas. When the target score is higher than a set threshold, a high-precision model is called to perform fine-grained verification and inference in the local area. The high- and low-precision models share some basic feature extraction modules and support dynamic switching and resource reuse, so as to significantly reduce the overall inference load and power consumption while ensuring detection accuracy.

[0016] The communication module is configured with an edge summary caching mechanism, including an interest region extractor and a latency control module; The system will only upload the image or point cloud fragment corresponding to the region of interest when the detection result is abnormal or the recognition confidence is lower than a preset threshold between 0.5 and 0.7. The cloud can make a preliminary judgment on the data based on the characteristics of the summary information, and if necessary, send a supplementary transmission command to the edge to obtain the original data.

[0017] The model inference service in the edge computing module adopts a plug-in structure, with the core backbone network and task head module separated. Different task modules run independently in the form of containers, supporting hot reloading and on-demand scheduling; The system dynamically loads or unloads specific functional modules based on task priority and computing power status to achieve functional flexibility and resource optimization for inference services.

[0018] The multimodal data fusion intermediate layer introduces a motion compensation prediction module; This module predicts the feature offsets of each sensor in the current frame based on the target motion state of the previous frame, and performs temporal and spatial alignment in advance. The converged network supports parallel asynchronous processing of different modal data channels to reduce latency in the convergence process.

[0019] The system has a self-learning mechanism, including: The edge device caches all low-confidence or failed-to-identify samples locally and periodically uploads them to the cloud when the communication module allows the bandwidth. After the cloud completes the re-labeling and model update, the improved model is pushed to edge devices through the model update interface to achieve continuous improvement in inference accuracy.

[0020] The edge device refers to the intelligent node deployed at the edge of the traffic site, which includes components such as an edge computing module, a communication module, a sensor acquisition module, and an energy management module; among them, the edge computing module is the core computing power unit.

[0021] Through real-time feedback and resource collaboration between modules, this invention achieves the following comprehensive technical effects: (1) Balancing low latency and high accuracy: The system is based on a dynamic model adaptive mechanism, which automatically selects a lightweight or high-precision model under different traffic scenarios to achieve real-time inference at the edge and maintains high detection accuracy in complex lighting and high-density traffic environments. (2) Improved communication efficiency and bandwidth optimization: An event-triggered upload mechanism is adopted, which only uploads the original data when the detection confidence is low or the result is abnormal. In other cases, only the summary information is transmitted, reducing more than 90% of unnecessary data uploads. (3) Dynamic balance between resource utilization and energy consumption: Combining the K3s lightweight container orchestration and energy management feedback module, the inference frequency, batch size and computing power allocation are automatically adjusted according to the real-time GPU load and temperature to achieve adaptive scheduling of computing power and optimal control of power consumption. (4) System scalability and self-learning capability: Through the plug-in model structure and container hot update mechanism, new modules or updated models can be dynamically loaded without stopping the inference service; edge devices can also periodically upload low confidence samples, and the cloud can complete retraining and send back updated models to form a self-evolutionary closed loop of continuous optimization. (5) Enhanced system stability and reliability: The multimodal data fusion layer unifies timestamps and data formats, simplifying the sensor synchronization process; the dual-layer feedback mechanism of energy management and container scheduling ensures that the system maintains stable temperature and performance during long-term operation.

[0022] In summary, this invention constructs an edge intelligent detection system with adaptive and self-optimizing capabilities through an innovative combination of multimodal fusion, dynamic model switching, resource self-scheduling, and energy closed-loop management. It achieves a unified balance of high precision, low latency, and low power consumption, demonstrating outstanding substantive features and significant technological progress. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall system architecture of the present invention, in which solid arrows represent data / result transmission and dashed arrows represent energy consumption adjustment and update instructions. Detailed Implementation

[0024] The specific implementation process of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0025] The term "edge device" as used in this specification refers to an edge intelligent node deployed at the traffic site, which includes components such as an edge computing module, a communication module, a sensor acquisition module, and an energy management module, among which the edge computing module is the core computing power unit.

[0026] Example:

[0027] System overall structure: like Figure 1 As shown, this system comprises five main modules: sensor acquisition module 1, edge computing module 2, software deployment module 3, communication module 4, and energy management module 5. These modules are connected via industrial Ethernet and the ROS communication framework.

[0028] Sensor acquisition module 1 includes three types of sensors: camera, millimeter-wave radar, and lidar. These sensors are connected to the edge computing module via a unified synchronization interface, with time errors controlled within ±2ms. The system is configured with a multimodal data fusion intermediate layer, employing a feature-level fusion network to achieve synchronous extraction and fusion of image, radar, and point cloud features. A motion compensation prediction module is introduced into the multimodal data fusion intermediate layer.

[0029] Detailed implementation of the motion compensation prediction module:

[0030] Specifically, the motion compensation prediction module of the multimodal data fusion intermediate layer uses the Kalman filter algorithm to track the state of the vehicle target. The system defines a state vector. S t = [ x, y, v x，v y ],in( x, y () represents the coordinates of the target center. v x ，v y () represents the velocity component. Based on the previous frame. t -1 is the optimal estimated state, used to predict the current frame. t The target position offset. Since the data acquisition frequencies of the camera and LiDAR may differ (e.g., camera 30Hz, LiDAR 10Hz), the system utilizes the predicted position offset Δ... d = v ×△ t Alignment compensation is performed on the timestamps of different sensors, and the feature map of the hysteresis mode is mapped to the spatial coordinate system of the current moment, thereby eliminating dynamic errors in high-speed moving scenarios.

[0031] Edge computing module 2 is based on NVIDIA Jetson Xavier NX or equivalent GPU platform, running Ubuntu and JetPack SDK; it deploys the TensorRT inference engine to accelerate deep learning models; and it has a dynamic model adaptation mechanism: automatically selects a lightweight model or a high-precision model for inference based on scene features such as traffic density, light intensity, and weather conditions.

[0032] Software deployment module 3 adopts Docker containerization and is dynamically managed through the K3s lightweight orchestration system; containers can be automatically created, paused, or migrated based on the real-time load of edge devices; each model runs as a REST service, supporting multi-container collaborative inference. Each container service uses a REST API to call inference tasks and interact with results.

[0033] Communication module 4 supports 4G / 5G cellular and wired network transmission; it adopts an event-triggered communication mechanism: raw data is uploaded only when an anomaly is detected or the confidence level is below a threshold; under normal circumstances, only detection summaries (target category, location coordinates, confidence level, etc.) are transmitted. The communication module can receive model update instructions from the cloud and hot-update the model in the container. A latency control module optimizes the upload timing based on network conditions, and the cloud can issue retransmission instructions based on the summary information.

[0034] The energy management module 5 monitors GPU load and temperature in real time; it shuts down some inference containers or switches to low-power mode under low load; and it dynamically allocates GPU computing power, adjusts batch size and inference frequency under high load to achieve energy balance and stable device operation. The energy management module can adjust the GPU scheduling order according to task thread priority.

[0035] Multimodal fusion mechanism: The system employs a three-way convolutional feature extraction sub-network: The image feature network is based on a lightweight YOLOv8 architecture; The radar feature network employs a convolutional sparse representation model; The point cloud feature network adopts point cloud feature encoding based on PointNet++.

[0036] The outputs of the three data sources are concatenated and then input into the fusion unit of the multimodal data fusion intermediate layer. The outputs are vehicle category probabilities and location regression results, and duplicate targets are removed by non-maximum suppression (NMS).

[0037] Detailed implementation of multimodal feature fusion In the feature fusion stage, to effectively integrate image and point cloud information, this embodiment employs a frustum-based feature association method. First, the 3D point cloud features are projected onto the 2D image plane to generate a sparse depth feature map. Then, the RGB semantic features output by the image feature extraction sub-network are concatenated with the projected point cloud geometric features along the channel dimension. The concatenated hybrid feature vector is then input into a fusion convolutional layer (1x1 convolution) for dimensionality reduction and weight redistribution. This fusion network automatically learns the complementary relationship between image texture and point cloud depth through an attention mechanism, for example, automatically increasing the weight of point cloud features when there is insufficient light at night, thereby outputting a joint feature representation containing rich multimodal information.

[0038] Dynamic model adaptive inference: The model inference service in the edge computing module adopts a plug-in architecture, separating the core backbone network from the task head module. Networks such as YOLOv8 and PointNet++ can be used as specific implementations of the backbone network. Different task heads can correspond to different inference models, and the corresponding detection or recognition model can be dynamically loaded according to the task type. The system sets up two sets of inference models in the edge computing module: a low-precision model and a high-precision model.

[0039] When the ambient brightness is high and the traffic is sparse, the system calls a low-precision or high-efficiency model (INT8 quantization or lightweight structure), and the frame rate is increased to 40fps. When there is insufficient lighting or dense targets, the system automatically switches to a high-precision model (FP16 or FP32 inference) to ensure detection accuracy.

[0040] This strategy is determined by the energy management module based on real-time GPU utilization and input frame complexity, forming a closed-loop self-adjustment mechanism.

[0041] Event-triggered communication: The communication module has three preset communication levels: Local real-time layer: Detection results are displayed directly on the vehicle screen; Summary upload layer: Sends the recognition results to the cloud in JSON format; Anomaly feedback layer: When the detection confidence is low or an unknown target is detected, the corresponding image frame and point cloud data are automatically uploaded.

[0042] This mechanism significantly reduces network bandwidth usage and supports automatic cloud-based distribution of supplementary training data tasks, achieving a data closed loop.

[0043] Dynamic container orchestration and energy management: A K3s cluster manages multiple Docker containers, each running an independent model service.

[0044] The energy management module monitors the GPU utilization of each container. When the total utilization is below 30%, it shuts down some containers and reduces the GPU frequency. When utilization exceeds 80%, the system automatically increases the parallel execution units of the inference containers through container orchestration and distributes computing tasks to available GPU cores to achieve computing power expansion and load balancing. It can also dynamically adjust the concurrency of the inference service according to the load. This mechanism enables the device to maintain efficient and stable operation under different loads.

[0045] Cloud-based collaboration and hot model updates: The communication module can pull algorithm update packages from the cloud and complete hot replacement of model files inside the container without stopping inference; The version control mechanism automatically rolls back to the old model to prevent update anomalies, ensuring system reliability and maintainability.

[0046] This invention proposes the following five optimization mechanisms based on the original system architecture: 1. Two-stage inference module: The edge computing module integrates a low-precision prior detection module and a high-precision refinement module. The former is used to quickly filter candidate target regions, while the latter performs fine identification and classification of the filtered regions. Both models share a feature extraction backbone network and run in conjunction with the TensorRT inference engine, which ensures detection accuracy while reducing overall computational power consumption, making it suitable for high-density traffic environments.

[0047] 2. Partial Upload and Cloud Feedback Mechanism: The communication module embeds a region of interest extractor and a summary generation module, allowing for the upload of only a local area of ​​the target image or point cloud when confidence is low or detection is uncertain. The cloud determines whether to request re-upload based on the received summary information, thereby reducing bandwidth consumption while maintaining recognition reliability.

[0048] 3. Plug-in Inference Models and Hot Container Reloading: Each inference model is encapsulated as an independent container service using a "backbone network + functional plugins" architecture. This allows for the flexible activation or suspension of specific functional plugins based on the computing power of edge nodes and task requirements, supporting various target detection and scene analysis tasks, thereby achieving resource reuse and rapid response. The task header module can be considered a type of functional plugin, with different plugins corresponding to different detection or recognition tasks.

[0049] 4. Asynchronous and Compensation Mechanism for Multimodal Fusion: The system pre-aligns sensor features using historical frame motion information, making asynchronous fusion more accurate. The fusion layer employs streaming processing to reduce multi-channel synchronization latency to within 100ms, adapting to real-time sensing requirements.

[0050] 5. Self-learning and model feedback closed loop: Edge devices automatically cache samples with low detection confidence or those that fail to be identified, and periodically aggregate and upload them to the cloud. After manual annotation or semi-automatic augmentation in the cloud, the samples are packaged into new models and deployed to the edge via a container hot update mechanism, forming a self-evolutionary closed loop that continuously improves detection robustness.

[0051] In summary, the adaptive multimodal edge vehicle detection and recognition system of the present invention achieves an adaptive closed-loop architecture of perception fusion, dynamic reasoning, energy management, and communication feedback through multiple collaborative innovative designs. Compared with existing technologies, it has higher real-time performance, lower energy consumption, and stronger environmental adaptability, and is suitable for various scenarios such as smart cities, autonomous driving, and road monitoring.

[0052] Furthermore, this invention introduces optimization modules including a two-stage inference mechanism, a local upload and feedback communication strategy, a pluggable model structure, a multimodal compensation fusion mechanism, and a self-learning feedback closed loop, which further enhances the scalability and adaptability of the system performance.

Claims

1. An adaptive multimodal edge vehicle detection and recognition system, characterized in that, include: The module comprises a sensor acquisition module, an edge computing module, a software deployment module, a communication module, and an energy management module; among which: The sensor acquisition module includes a camera, millimeter-wave radar, and lidar, used to acquire images, distance, and point cloud information of the vehicle's surrounding environment; the sensor acquisition module has a unified time synchronization interface and a multimodal data fusion intermediate layer, used to perform time alignment and format unification of multi-source data at the edge. The edge computing module uses an embedded AI computing device with a GPU, is equipped with an embedded Linux operating system and an artificial intelligence development kit, and runs a deep learning model based on a deep learning inference acceleration engine. The edge computing module dynamically selects detection models of different precision according to the complexity of the traffic scene, so as to reduce the computational load while maintaining detection accuracy. The software deployment module uses a lightweight container orchestration system to run vehicle detection and recognition tasks with application containers as the basic unit; the orchestration system can dynamically create, pause, or migrate containers according to the load of the edge computing module to achieve adaptive resource scheduling. The communication module is used to transmit the detection results generated by the edge computing module to the cloud via 4G / 5G or wired network; the communication module has an event-triggered upload mechanism, which only uploads the original data when the detection result is abnormal or the recognition confidence is lower than a preset threshold, and only transmits the detection summary information in other cases; The energy management module monitors the real-time load and temperature status of the edge computing module. Under low load conditions, it shuts down some inference containers or switches to low power mode, and under high load conditions, it automatically allocates GPU computing power to achieve dynamic energy balance.

2. The system according to claim 1, characterized in that, The edge computing module includes two modes: a low-precision model and a high-precision model. The system determines the scene state by indicators such as ambient lighting, traffic density, or image complexity, and calls the low-precision model when the complexity is low and the high-precision model when the complexity is high.

3. The system according to claim 1, characterized in that, The multimodal data fusion intermediate layer adopts a timestamp alignment algorithm and a feature-level fusion network, and introduces a motion compensation prediction module; the motion compensation prediction module predicts the feature offset of each sensor in the current frame based on the target motion state of the previous frame, and performs time and space alignment in advance. It includes an image feature extraction subnetwork, a radar feature extraction subnetwork, and a point cloud feature extraction subnetwork. The outputs of the three are correlated in the fusion layer and support parallel asynchronous processing. Finally, the vehicle category probability is output through a fully connected layer.

4. The system according to claim 1, characterized in that, The software deployment module implements container orchestration based on a lightweight container orchestration system; each container encapsulates an independent deep learning model inference service and communicates with each other through a REST interface to support multi-task collaborative inference; the container orchestration system can adjust the number and concurrency of parallel inference containers according to the load; the model inference service adopts a plug-in structure, with the core backbone network and task head module separated, and different task modules running independently in the form of containers, supporting hot loading and on-demand scheduling.

5. The system according to claim 1, characterized in that, The communication module supports a hierarchical communication mechanism, including: First layer: Local real-time feedback layer, used to output detection results to the vehicle display terminal; The second layer: the summary upload layer, which only reports vehicle location and category information; The third layer is the anomaly return layer, which is used to upload video or point cloud data corresponding to low-confidence results.

6. The system according to claim 1, characterized in that, A feedback loop is provided between the energy management module and the edge computing module. The feedback loop adjusts the working mode according to the container GPU utilization: when the GPU utilization is below 30%, it is determined to be a low load condition, and when the GPU utilization is above 80%, it is determined to be a high load condition. Based on this, the inference frequency, batch size and thread priority are adjusted in real time.

7. The system according to claim 1, characterized in that, The system can automatically pull new model versions and perform hot updates within a container based on model update instructions issued from the cloud, without stopping the inference service.

8. The system according to claim 1, characterized in that, The system has an adaptive control closed loop, including: a sensor acquisition module detecting traffic status; an edge computing module adjusting the inference model based on scene complexity; a software deployment module dynamically scheduling containers and computing power accordingly; an energy management module adjusting energy consumption mode according to operating load; and a communication module uploading key data according to event triggering rules. The closed loop is executed cyclically to achieve system self-learning and self-optimization. The self-learning mechanism includes: edge devices caching low-confidence or failed identification samples locally and periodically uploading them; after the cloud completes re-labeling and model updates, it pushes the improved model to the edge devices through the model update interface.

9. The system according to claim 2, characterized in that: The edge computing module is configured with a two-stage inference mechanism. First, a low-precision model is used to quickly screen the image or point cloud data and score the potential vehicle target areas. When the target score is higher than a set threshold, a high-precision model is called to perform fine-grained verification and inference in the local area. The high- and low-precision models share some basic feature extraction modules and support dynamic switching and resource reuse, so as to significantly reduce the overall inference load and power consumption while ensuring detection accuracy.

10. The system according to claim 1, characterized in that: The communication module is configured with an edge summary caching mechanism, including an interest region extractor and a latency control module; The system will only upload the image or point cloud fragment corresponding to the region of interest when the detection result is abnormal or the recognition confidence is lower than a preset threshold between 0.5 and 0.

7. The cloud can make a preliminary judgment on the data based on the characteristics of the summary information, and if necessary, send a supplementary transmission command to the edge to obtain the original data.

Images