Multi-mode real-time target detection and tracking system

Abstract

The invention relates to the technical field of target detection and tracking, and discloses a multi-modal real-time target detection and tracking system, which is characterized in that a multi-modal data acquisition module integrates a high-definition camera, a millimeter-wave radar, a laser radar and an infrared sensor and is used for acquiring target information from multiple dimensions such as visual images, distance, speed and angle, three-dimensional point cloud and thermal radiation; and the data preprocessing module is used for carrying out denoising, enhancement, normalization, filtering, coordinate conversion and temperature correction processing on the original data acquired by the multi-modal data acquisition module. Multiple sensors are integrated to collect multi-dimensional data, after preprocessing, efficient fusion is achieved through hierarchical fusion and an attention mechanism, the detection module is combined with an improved algorithm and a dynamic threshold value, the tracking module fuses multiple features and has the online learning ability, and the system control module achieves intelligent management. The system greatly improves the accuracy, the real-time performance and the stability of detection and tracking, has remarkable advantages of an innovative technology, and provides a new scheme for related fields.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and more particularly to a multimodal real-time target detection and tracking system. Background Art

[0002] Currently, target detection and tracking technologies are widely used in numerous fields, including security monitoring, autonomous driving, and intelligent robotics. Traditional single-modal target detection and tracking systems, such as visual detection systems that rely solely on cameras, are significantly affected by factors such as lighting, occlusion, and viewing angle, resulting in poor detection and tracking accuracy and stability in complex environments. Systems based solely on sensors such as radar, while highly adaptable to the environment, lack target feature recognition and struggle to accurately distinguish between different types of targets. While multimodal target detection and tracking systems have developed, existing multimodal systems suffer from low data fusion efficiency, difficulty adapting algorithms to different scenarios and target changes, and insufficient real-time performance. These systems are unable to meet the high-precision and real-time requirements for target detection and tracking in complex and changing environments. Therefore, a multimodal, real-time target detection and tracking system is urgently needed to address these issues. Summary of the Invention

[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a multimodal real-time target detection and tracking system to solve the problems existing in the above-mentioned background technology.

[0004] The present invention provides the following technical solution: a multimodal real-time target detection and tracking system, comprising:

[0005] Multimodal data acquisition module, integrating high-definition cameras, millimeter-wave radars, lidars, and infrared sensors, to collect target information from multiple dimensions, including visual images, distance, speed, and angle, 3D point clouds, and thermal radiation;

[0006] A data preprocessing module performs denoising, enhancement, normalization, filtering, coordinate conversion, temperature correction, and time synchronization and spatial alignment on the raw data acquired by the multimodal data acquisition module;

[0007] The multimodal data fusion module adopts a layered fusion strategy, including direct merging of raw data at the data layer, fusion of feature vectors of each sensor at the feature layer, and comprehensive decision-making based on the independent detection results of each sensor at the decision layer. It also introduces an attention mechanism to dynamically adjust the fusion weight of each sensor data.

[0008] The target detection module builds a multimodal target detection model based on the improved YOLO algorithm and Transformer structure, sets a multimodal feature fusion layer, uses transfer learning for pre-training and fine-tuning, and has a dynamic detection threshold mechanism that automatically adjusts according to target size, distance, and environmental complexity;

[0009] The target tracking module extracts the visual, motion, and spatial features of the target and fuses them to construct a feature descriptor. It uses the Kalman filter algorithm to predict the target's motion state, combines the Hungarian algorithm for data association, and introduces an online learning mechanism to update the target feature model.

[0010] The system control and management module dynamically adjusts the operating parameters of each sensor in the multimodal data acquisition module according to the target detection and tracking results, monitors the system operation status in real time, performs fault alarms and processing, and provides a human-computer interaction interface to realize parameter setting, result viewing and remote control management.

[0011] Furthermore, in the multimodal data fusion module, data layer fusion is to merge camera image data, millimeter-wave radar and lidar point cloud data, and infrared sensor thermal image data according to rules; during feature layer fusion, convolutional neural networks are used to extract image and infrared data features, and point cloud processing networks such as PointNet are used to extract millimeter-wave radar and lidar data features, and then feature vectors are fused through a fully connected layer; decision layer fusion adopts a weighted voting method to assign weights according to the performance of each sensor in different scenarios.

[0012] Furthermore, in the target detection module, the multimodal target detection model that combines the improved YOLO algorithm with the Transformer structure further fuses and extracts features from the feature representation output by the multimodal data fusion module by adding a multimodal feature fusion layer.

[0013] Furthermore, in the target tracking module, the visual features include color histogram and HOG features, the motion features obtain the speed and acceleration by calculating the position change of the target in consecutive frames, and the spatial features are the position coordinates and size information of the target.

[0014] Furthermore, the system control and management module adopts an embedded control system, realizes the operation control and resource management of the system by writing a control program, and uses a sensor status monitoring circuit and a software algorithm to monitor the system operation status.

[0015] Furthermore, the high-definition camera in the multimodal data acquisition module is an industrial-grade high-resolution, high-frame-rate camera, and the millimeter-wave radar and lidar are selected with corresponding detection range and accuracy models according to the application scenario requirements, and are precisely installed and calibrated.

[0016] Furthermore, in the data preprocessing module, Gaussian filtering and histogram equalization are used to enhance the image data, Kalman filtering is used for millimeter wave radar and lidar data, and high-precision clock synchronization technology and joint calibration are used to achieve time synchronization and spatial alignment of multimodal data.

[0017] Furthermore, the attention mechanism of the multimodal data fusion module dynamically adjusts the weight of each sensor data in the fusion process according to the input multimodal data by training the attention model.

[0018] Furthermore, during training, the target detection module is first pre-trained on large-scale public datasets such as COCO and ImageNet, and then fine-tuned on datasets collected for specific application scenarios.

[0019] Furthermore, the online learning mechanism of the target tracking module updates the target's feature model using new detection results when the target's appearance or motion pattern changes.

[0020] Technical effects and advantages of the present invention:

[0021] The present invention integrates multiple sensors to collect multi-dimensional data, and uses hierarchical fusion and attention mechanisms to achieve efficient fusion after preprocessing. The detection module combines improved algorithms with dynamic thresholds, the tracking module integrates multiple features and has online learning capabilities, and the system control module realizes intelligent management. The system greatly improves the accuracy, real-time performance and stability of detection and tracking, and has significant innovative technological advantages, providing new solutions for related fields.

[0022] Specifically:

[0023] Significantly improve detection and tracking accuracy: The multimodal data acquisition module integrates high-definition cameras, millimeter-wave radars, lidars, and infrared sensors to collect target information from multiple dimensions, including visual images, distance, speed, and angle, three-dimensional point clouds, and thermal radiation, thus making up for the limitations of single-modal sensors. For example, in environments with insufficient lighting, infrared sensors and millimeter-wave radars can work normally to obtain target information; in complex occlusion scenarios, the three-dimensional point cloud data of the lidar can assist in determining the target's position and shape. The multimodal data fusion module adopts a layered fusion strategy and an attention mechanism. At the data layer, the original data is directly merged to retain the underlying information. The feature layer deeply fuses the feature vectors of each sensor. The decision layer makes a comprehensive decision based on weighted voting. At the same time, the attention mechanism dynamically adjusts the fusion weights, enabling the system to fully utilize the complementarity of multimodal data and accurately identify targets. The target detection module is based on the improved YOLO algorithm and Transformer structure, combined with a multimodal feature fusion layer and a dynamic detection threshold mechanism, which can effectively extract target features and reduce missed detection of small targets and false detection of similar targets; the target tracking module constructs a feature descriptor by fusing visual, motion, and spatial features, and cooperates with Kalman filtering and the Hungarian algorithm, as well as an online learning mechanism. Even if the target appearance and motion pattern change, it can continue to track accurately, greatly improving the accuracy of detection and tracking.

[0024] Efficiently ensure the real-time performance of the system: In the data processing flow, the data preprocessing module uses efficient algorithms such as Gaussian filtering and Kalman filtering to quickly process multimodal data, ensuring data quality while reducing processing time. The layered fusion strategy and attention model of the multimodal data fusion module have been optimized to reduce the computational complexity of data fusion. The target detection module is based on a lightweight improved YOLO algorithm and Transformer structure. By reducing redundant calculations and optimizing network parameters, it improves detection speed while ensuring detection accuracy. The Kalman filter and Hungarian algorithm of the target tracking module use a fast implementation method to reduce the time overhead of data association and state prediction. At the same time, the system control and management module dynamically adjusts the operating parameters of each sensor according to the target detection and tracking results to avoid resource waste, further improve the system's operating efficiency, and enable the entire system to meet real-time requirements. It is suitable for scenarios such as autonomous driving and real-time security monitoring that require extremely high response speed.

[0025] Comprehensively enhance system stability: The coordinated operation of multiple sensors enables the system to stably acquire target information under diverse environmental conditions. For example, millimeter-wave radar and infrared sensors can still function normally in inclement weather such as rain, snow, and fog. At night or in bright light, infrared sensors and lidar can compensate for the shortcomings of cameras. The data preprocessing module's time synchronization and spatial alignment mechanisms ensure the consistency of multimodal data, providing a reliable foundation for subsequent processing. The system control and management module monitors the system's operating status in real time. When sensor failures or data anomalies occur, it can promptly issue alarms and implement troubleshooting measures, such as switching to backup sensors and adjusting data fusion strategies. The target tracking module's online learning mechanism can adapt to target changes, ensuring continuous and stable tracking. These designs work together to ensure the system's long-term stable operation in complex and changing environments, reducing the probability of system failures and misjudgments.

[0026] Outstanding innovative technical advantages: This invention innovatively combines the hierarchical fusion strategy with the Transformer-based attention mechanism in multimodal data fusion. Compared with traditional fusion methods, it can more efficiently fuse multi-source heterogeneous data. The target detection module improves the YOLO algorithm and introduces the Transformer structure, which enhances the extraction and processing capabilities of multimodal features. The dynamic detection threshold mechanism further optimizes the detection performance. The target tracking module integrates multiple features and adopts an online learning mechanism to improve the adaptability to target changes. The system control and management module realizes intelligent control and management of the system. These innovations work together to form a complete and advanced multimodal real-time target detection and tracking system. It has outstanding substantive features and significant progress in technology. Compared with existing technologies, it shows stronger innovation and competitiveness, and provides new ideas and solutions for the field of target detection and tracking technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an architecture diagram of the multimodal real-time target detection and tracking system of the present invention;

[0028] Figure 2 This is a workflow diagram of the multimodal data fusion module in the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] It will be understood that the terms "first," "second," etc. used herein may be used to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element.

[0031] In order to help those skilled in the art better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of this application.

[0032] Embodiment: The present invention provides a multimodal real-time target detection and tracking system, comprising:

[0033] 1. Multimodal Data Acquisition Module

[0034] The multimodal data acquisition module integrates a high-definition camera, millimeter-wave radar, lidar, and infrared sensor. The high-definition camera uses an industrial-grade 4K resolution and 60fps frame rate. Connected to the system via an M12 interface, it is installed 4 meters above the monitoring area, with a 120° horizontal viewing angle, ensuring clear visual capture of the target. During installation, the camera is calibrated horizontally and vertically using professional calibration tools to ensure accurate image acquisition.

[0035] The millimeter-wave radar, operating in the 76-81 GHz frequency band and with a detection range of 200 meters, is connected to the system via a CAN bus interface and installed 0.5 meters below the camera, tilted 15 degrees downward to accurately detect the target's distance, speed, and angle. Prior to deployment, the millimeter-wave radar underwent multiple field tests, and its detection parameters were adjusted based on the test results. For example, detection accuracy was set to ±0.1 meters (distance), ±1 km / h (speed), and ±1° (angle).

[0036] The LiDAR utilizes a 16-line mechanical LiDAR with a 360° detection range and a maximum detection distance of 100 meters. It communicates with the system via an Ethernet interface and is installed in the center of the device, ensuring full 3D scanning of the target. After installation, the LiDAR is calibrated using a high-precision calibration plate to establish its spatial relationship with the other sensors.

[0037] The infrared sensor uses an uncooled type with a thermal sensitivity of 0.05°C and an operating wavelength range of 8-14μm. It connects to the system via a USB port and is installed next to the camera to acquire thermal radiation information from the target in low-visibility environments. During installation, the infrared sensor's field of view is adjusted to closely overlap with the camera's field of view to facilitate subsequent data fusion.

[0038] During operation, each sensor collects data at a preset frequency. The high-definition camera collects 60 frames of images per second, the millimeter-wave radar updates the detection data 20 times per second, the lidar scans 10 times per second, and the infrared sensor collects 30 frames of thermal images per second. The collected data is transmitted to the data preprocessing module in real time.

[0039] 2. Data Preprocessing Module

[0040] For image data captured by a high-definition camera, a Gaussian filter algorithm is first used for denoising. The Gaussian filter kernel size is set to 3×3, with a standard deviation of 1.5. This operation effectively removes Gaussian noise from the image. Next, histogram equalization is used to enhance the image, stretching the grayscale range to the entire grayscale range and improving the image contrast. Finally, the image pixel values are normalized to the range [0, 1] to facilitate subsequent processing. The normalization formula is:

[0041]

[0042] Where I is the original image pixel value, I min and I max are the minimum and maximum pixel values in the image, respectively.

[0043] Millimeter-wave radar and lidar data are filtered using the Kalman filter algorithm. Taking millimeter-wave radar as an example, a state-space model is established that includes the target's position, velocity, and acceleration. The process noise covariance matrix Q and the observation noise covariance matrix R are set based on the radar's detection accuracy. After each radar data is received, the target's state is estimated through the prediction and update steps of the Kalman filter to remove noise interference from the data. Simultaneously, the millimeter-wave radar and lidar data are converted to a unified world coordinate system based on the installation position and attitude parameters of each sensor. This conversion process involves the calculation of rotation and translation matrices, and the relevant parameters are obtained through precise calibration.

[0044] Infrared sensor data is temperature-corrected based on its temperature response characteristics. Using the temperature correction coefficients provided by the sensor manufacturer, the raw thermal image data is corrected to eliminate temperature errors caused by the sensor's inherent characteristics. Infrared images are enhanced using adaptive histogram equalization. By dividing the image into sub-blocks and performing histogram equalization on each sub-block, this method avoids the noise amplification that can be caused by global histogram equalization.

[0045] For time synchronization, the system utilizes high-precision GPS-based clock synchronization technology. Each sensor receives a precise timestamp from GPS signals, ensuring temporal consistency across all sensor data. Time synchronization accuracy reaches the microsecond level. For spatial alignment, a high-precision calibration plate is placed within the monitoring area, while each sensor simultaneously collects observation data from the plate. The camera is calibrated using techniques such as the Zhang Zhengyou calibration method to obtain its internal and external parameters. For millimeter-wave radar and lidar, a conversion relationship is established between them and the camera to achieve spatial alignment of multimodal data, with spatial alignment error controlled to the centimeter level.

[0046] 3. Multimodal Data Fusion Module

[0047] During data layer fusion, the camera's RGB image data, millimeter-wave radar point cloud data (including distance, speed, and angle information), lidar's 3D point cloud data, and infrared sensor's thermal image data are matched according to timestamps and directly merged into a single data set. For example, for data collected at the same time, the image data, millimeter-wave radar data, lidar data, and infrared data are combined into a multidimensional data structure as the result of data layer fusion.

[0048] For feature-level fusion, ResNet-50 was used as the backbone network for image data, extracting 2048-dimensional feature vectors in the final layer. For infrared data, the same ResNet-50 network was used to extract features. For millimeter-wave radar and lidar data, the point cloud data was processed using the PointNet network, extracting 1024-dimensional feature vectors. These feature vectors were then fused using a fully connected layer consisting of two layers: the first layer had 4096 neurons, and the second layer had 2048 neurons. A nonlinear transformation was performed using the ReLU activation function to obtain a feature representation containing multimodal information.

[0049] At the decision-making level, each sensor performs independent target detection. The camera uses the traditional YOLOv5 algorithm for target detection, the millimeter-wave radar and lidar use their own data processing algorithms, and the infrared sensor uses a thermal image segmentation algorithm. After obtaining the detection results from each sensor, a weighted voting method is used to make a comprehensive decision. Through extensive experiments, the weights of each sensor are determined for different scenarios (such as urban roads, indoor environments, and night scenes). For example, in daytime urban road scenes, the camera has a weight of 0.5, the millimeter-wave radar has a weight of 0.3, the lidar has a weight of 0.15, and the infrared sensor has a weight of 0.05. In nighttime scenes, the camera has a weight of 0.3, the millimeter-wave radar has a weight of 0.3, the lidar has a weight of 0.2, and the infrared sensor has a weight of 0.2.

[0050] Furthermore, dynamic weight adjustment is achieved by training a Transformer-based attention model. The feature representation output by the multimodal data fusion module is used as input to the attention model, which then outputs weight coefficients for each sensor data point. During training, the model is trained using a stochastic gradient descent algorithm with a learning rate of 0.001 for 100 epochs, using target detection accuracy and recall as optimization goals. This allows the attention model to dynamically adjust the weights of each sensor data point during the fusion process based on the input multimodal data, highlighting key information.

[0051] 4. Object Detection Module

[0052] A multimodal object detection model was constructed based on the improved YOLO algorithm and Transformer architecture. The network architecture retains the backbone network of YOLOv5, while a multimodal feature fusion layer is added to the neck network. This fusion layer consists of three layers: the first is a convolutional layer with a 3×3 kernel size and 256 channels, which performs preliminary processing on the features output by the multimodal data fusion module; the second is a Transformer layer, which includes an 8-head multi-attention mechanism, each with a dimension of 64, to capture long-range dependencies between data of different modalities; the third is an upsampling layer, which restores the feature map to an appropriate size through bilinear interpolation and fuses it with the feature map output by the backbone network.

[0053] During the training process, the model is first pre-trained on the large-scale public datasets COCO and ImageNet. During pre-training, the Adam optimizer is used, with a learning rate of 0.001 and a batch size of 64. Training is performed for 30 epochs to enable the model to learn common target features. Then, fine-tuning is performed on a dataset collected for a specific application scenario. During fine-tuning, the learning rate is reduced to 0.0001, the batch size is adjusted to 32, and training is performed for 15 epochs to adapt the model to the object detection task in the specific scenario.

[0054] The dynamic detection threshold mechanism automatically adjusts based on target size, distance, and environmental complexity. By analyzing extensive data, a mapping between target size and detection threshold is established. For example, when the target size in the image is smaller than 50×50 pixels, the detection threshold is set to 0.3; when the target size is larger than 200×200 pixels, the detection threshold is set to 0.5. Furthermore, the detection threshold is dynamically adjusted based on the target distance detected by the millimeter-wave radar and lidar, as well as environmental lighting and occlusion. For example, at long distances and in complex environments, the detection threshold is appropriately lowered to improve the detection recall rate.

[0055] 5. Target Tracking Module

[0056] Based on target detection, multiple features of the target are extracted. In terms of visual features, the color histogram of the target area is calculated. Using the HSV color space, the H channel is divided into 16 intervals, and the S channel and V channel are divided into 8 intervals each, resulting in a 2048-dimensional color histogram feature. At the same time, the HOG algorithm is used to extract the directional gradient histogram features of the target. The window size is set to 64×64 pixels, the block size is 16×16 pixels, and the cell size is 8×8 pixels. Each cell contains 9 gradient directions, and finally a 3780-dimensional HOG feature is obtained. The motion feature calculates the position change of the target in three consecutive frames and uses the least squares method to fit the target's velocity and acceleration. The spatial feature directly obtains the target's position coordinates (x, y) and size information (width w, height h). These features are connected in series to construct a high-dimensional target feature descriptor.

[0057] The Kalman filter algorithm is used to predict the target's motion state. A state-space model is established, including the target's position, velocity, and acceleration. The process noise covariance matrix Q and the observation noise covariance matrix R are set based on the target's motion characteristics and the sensor's detection accuracy. In each frame, the target's state is predicted based on the previous frame's state. The detection results output by the target detection module are then used as observations to update the predicted state, resulting in an accurate estimate of the target's motion state.

[0058] Combined with the Hungarian algorithm for data association, the distance between the target detection result and the prediction result in the current frame is used as the cost matrix. The Hungarian algorithm is used to solve the cost matrix and find the optimal matching relationship between the targets in different frames, thus achieving continuous tracking of the target.

[0059] The online learning mechanism is activated when the target's appearance or motion pattern changes. When the detected feature descriptor of a target differs from the previous feature by more than a certain threshold (set experimentally to 0.5), the target's feature model is updated using the new detection result. This update is performed by weighted fusion of the new feature descriptor with historical features, with weights assigned based on time. Features closer to the current time have higher weights. This allows the target tracking module to adapt to target changes, improving tracking accuracy and stability.

[0060] 6. System Control and Management Module

[0061] The system control and management module uses an embedded control system based on the ARM Cortex-A72 architecture and runs the Linux operating system. A C++ control program is written to control the entire system's operation and manage its resources. During system startup, the control program initializes each module and configures the operating mode of each sensor in the multimodal data acquisition module according to preset parameters.

[0062] Based on the results of target detection and tracking, the operating parameters of each sensor in the multimodal data acquisition module are dynamically adjusted. When a distant target is detected, the HD camera's focal length is adjusted from 25mm to 50mm, while the millimeter-wave radar's detection range is increased to 250 meters and the lidar's scanning frequency is increased to 15 times per second. When the system is in low-power mode, the operating frequency of each sensor is reduced, for example, the HD camera frame rate is reduced to 30fps, the millimeter-wave radar updates detection data 10 times per second, and the lidar scans 5 times per second.

[0063] The system's operating status is monitored in real time using sensor status monitoring circuits and software algorithms. The sensor status monitoring circuit determines whether the sensor is functioning properly by detecting parameters such as the sensor's power supply voltage and communication signal. The software algorithm analyzes the data output by each module in real time. When an anomaly is detected (e.g., no valid detection results for five consecutive frames, excessive sensor data fluctuations), an alarm is issued and appropriate troubleshooting measures are implemented. For example, if a millimeter-wave radar failure is detected, the system automatically switches to a backup millimeter-wave radar (if available). If a backup radar is unavailable, the system reduces its reliance on millimeter-wave radar data, prioritizing target detection and tracking using data from other sensors. Fault information is displayed through the human-computer interface, prompting maintenance personnel to perform repairs.

[0064] Develop a human-computer interaction interface based on the Qt framework, through which users can set system parameters, such as adjusting the category of target detection, setting the initial value of the weight of multimodal data fusion, and modifying the working parameters of the sensor; view the detection and tracking results, and display the target's position, motion trajectory, category and other information in a visual manner; and realize remote control and management of the system through the network. Users can log in to the system account on the remote terminal to monitor and operate the system in real time, such as remotely starting or stopping the system, adjusting the camera's shooting angle, etc.

[0065] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0067] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal real-time target detection and tracking system, characterized by: include: Multimodal data acquisition module, integrating high-definition cameras, millimeter-wave radars, lidars, and infrared sensors, to collect target information from multiple dimensions, including visual images, distance, speed, and angle, 3D point clouds, and thermal radiation; A data preprocessing module performs denoising, enhancement, normalization, filtering, coordinate conversion, temperature correction, and time synchronization and spatial alignment on the raw data acquired by the multimodal data acquisition module; The multimodal data fusion module adopts a layered fusion strategy, including direct merging of raw data at the data layer, fusion of feature vectors of each sensor at the feature layer, and comprehensive decision-making based on the independent detection results of each sensor at the decision layer. It also introduces an attention mechanism to dynamically adjust the fusion weight of each sensor data. The target detection module builds a multimodal target detection model based on the improved YOLO algorithm and Transformer structure, sets a multimodal feature fusion layer, uses transfer learning for pre-training and fine-tuning, and has a dynamic detection threshold mechanism that automatically adjusts according to target size, distance, and environmental complexity; The target tracking module extracts the visual, motion, and spatial features of the target and fuses them to construct a feature descriptor. It uses the Kalman filter algorithm to predict the target's motion state, combines the Hungarian algorithm for data association, and introduces an online learning mechanism to update the target feature model. The system control and management module dynamically adjusts the operating parameters of each sensor in the multimodal data acquisition module according to the target detection and tracking results, monitors the system operation status in real time, performs fault alarms and processing, and provides a human-computer interaction interface to realize parameter setting, result viewing and remote control management.

2. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: In the multimodal data fusion module, data layer fusion is to merge camera image data, millimeter wave radar and lidar point cloud data, and infrared sensor thermal image data according to rules; during feature layer fusion, convolutional neural networks are used to extract image and infrared data features, and point cloud processing networks such as PointNet are used to extract millimeter wave radar and lidar data features, and then feature vectors are fused through a fully connected layer; The decision-making layer fusion adopts a weighted voting method to assign weights according to the performance of each sensor in different scenarios.

3. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: In the target detection module, the multimodal target detection model that combines the improved YOLO algorithm with the Transformer structure further fuses and extracts features from the feature representation output by the multimodal data fusion module by adding a multimodal feature fusion layer.

4. The multimodal real-time target detection and tracking system according to claim 1, wherein: In the target tracking module, the visual features include color histogram and HOG features, the motion features are obtained by calculating the position change of the target in consecutive frames to obtain the speed and acceleration, and the spatial features are the position coordinates and size information of the target.

5. The multimodal real-time target detection and tracking system according to claim 1, wherein: The system control and management module adopts an embedded control system, realizes the operation control and resource management of the system by writing a control program, and uses a sensor status monitoring circuit and a software algorithm to monitor the system operation status.

6. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: The high-definition camera in the multimodal data acquisition module is an industrial-grade high-resolution, high-frame-rate camera. The millimeter-wave radar and lidar are selected with corresponding detection range and accuracy models according to the application scenario requirements, and are precisely installed and calibrated.

7. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: In the data preprocessing module, Gaussian filtering and histogram equalization are used to denoise image data, and Kalman filtering is used for millimeter wave radar and lidar data. High-precision clock synchronization technology and joint calibration are used to achieve time synchronization and spatial alignment of multimodal data.

8. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: The attention mechanism of the multimodal data fusion module dynamically adjusts the weight of each sensor data in the fusion process according to the input multimodal data by training the attention model.

9. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: During training, the object detection module is first pre-trained on large-scale public datasets such as COCO and ImageNet, and then fine-tuned on datasets collected for specific application scenarios.

10. The multimodal real-time target detection and tracking system according to claim 1, characterized in that: The online learning mechanism of the target tracking module uses the new detection results to update the target's feature model when the target's appearance or movement pattern changes.

Images