A multi-modal dynamic target perception and tracking early warning method and system

Abstract

The application discloses a kind of multimodal dynamic target perception and tracking early warning method and system, obtain the multimodal perception data of monitoring area, and carry out time synchronization and space registration, establish unified reference coordinate system;Extract each modal target candidate observation information and obtain perception quality parameter;According to the short-time prediction state generated in the last time fusion state, determine dynamic fusion weight in combination with perception quality parameter, detection confidence and consistency parameter, complete target fusion tracking;When target is lost, according to the search area generated in the short-time prediction state and state estimation confidence coefficient generated in the fusion state before losing, and through re-association score restores original target trajectory;Further according to the deviation between the prediction trajectory generated in target historical trajectory sequence and actual trajectory, carry out abnormality determination and output graded warning result;The application can improve the accuracy and stability of dynamic target perception, continuous tracking and abnormality warning in complex scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a multimodal dynamic target perception, tracking and early warning method and system. Background Technology

[0002] With the continuous expansion of application scenarios such as intelligent security, intelligent transportation, park monitoring, and unmanned platform inspection, continuous perception, stable tracking, and anomaly early warning of dynamic targets have become important research directions in related technical fields. Therefore, a Chinese patent (application number CN202111569801.6) discloses a dynamic target perception system and method integrating a camera and a lidar. This patented technology, through the cooperation of a monocular camera, lidar, and controller, utilizes methods such as two-dimensional target detection, three-dimensional region of interest generation, point cloud clustering fitting, inter-frame matching, tracker management, and trajectory prediction to achieve the detection and tracking of three-dimensional dynamic targets and predict the target's motion trend based on historical trajectories, thereby improving the accuracy and reliability of dynamic target perception to a certain extent. However, this patent mainly focuses on the collaborative achievement of target detection, tracking, and trend prediction by the camera and lidar. In complex monitoring scenarios, the completeness of continuous target tracking and the forward-looking nature of anomaly warning are still insufficient. Therefore, those skilled in the art provide a multimodal dynamic target perception and tracking warning method and system to solve the problems mentioned in the background art. Summary of the Invention

[0003] This invention provides a multimodal dynamic target perception, tracking, and early warning method and system to improve the accuracy and stability of target perception and tracking in complex scenarios.

[0004] In a first aspect, the present invention provides a multimodal dynamic target perception, tracking, and early warning method, comprising the following steps: S1. Acquire multimodal sensing data of the monitoring area, and perform time synchronization and spatial registration of the multimodal sensing data to establish a unified reference coordinate system; S2. Extract target candidate observation information based on the synchronized and registered modal sensing data, and obtain the sensing quality parameters corresponding to each modality; S3. When there is a fusion state of the target at the previous moment, generate the short-term prediction state at the current moment based on the fusion state at the previous moment; when there is no fusion state of the target at the previous moment, establish the initial fusion state of the target based on the target candidate observation information of each mode at the current moment. S4. Based on the perception quality parameters of each modality, the detection confidence level corresponding to each target candidate observation information, and the consistency parameters between the target candidate observation information of each modality and the short-term predicted state, determine the dynamic fusion weight corresponding to each modality; fuse the target candidate observation information of each modality based on the dynamic fusion weight corresponding to each modality to obtain the fusion state of the target at the current moment, and update the target trajectory information. S5. When the target trajectory fails to complete the association and matching with the new target candidate observation information in the current frame within consecutive preset frames, the corresponding target trajectory is switched to the lost tracking cache state, and the prediction search area is generated based on the fusion state before the target lost tracking, the latest short-term prediction state, and the state before the loss of tracking to estimate the confidence coefficient. S6. Retrieve new target candidate observation information under each mode within the predicted search area, and calculate the re-association score between the new target candidate observation information and the target trajectory in the lost-tracking state; when the re-association score meets the preset recovery threshold condition, restore the association of the new target candidate observation information to the original target trajectory, and keep the original target identifier unchanged; S7. Generate a predicted trajectory for a future preset sampling time based on the target's historical trajectory sequence in an active or recovered state. S8. At the future preset sampling time, obtain the actual observation state of the target, and form the actual trajectory of the target based on the actual observation state of the target; S9. Calculate the trajectory prediction deviation between the actual trajectory and the predicted trajectory; when the trajectory prediction deviation meets the preset anomaly judgment condition, determine that the target has abnormal behavior and output the corresponding warning result.

[0005] Preferably, the multimodal sensing data includes two or more of the following: visible light image data, infrared image data, laser point cloud data, and millimeter-wave radar data. Among them, the perception quality parameters corresponding to the image mode include one or more of image sharpness and occlusion ratio; the perception quality parameters corresponding to the laser point cloud mode include one or more of point cloud density and point cloud contour integrity; and the perception quality parameters corresponding to the millimeter-wave radar mode include one or more of echo signal-to-noise ratio, velocity measurement stability, and range continuity.

[0006] Preferably, in step S4, the consistency parameter is used to characterize the degree of matching between the candidate observation information of each modality target and the short-term predicted state, and the degree of matching includes one or more of position deviation, velocity deviation and orientation deviation; All other things being equal, the larger the perceived quality parameter, the higher the detection confidence, and the higher the degree of matching represented by the consistency parameter, the greater the dynamic fusion weight of the corresponding modality.

[0007] Preferably, in step S5, the state estimation reliability coefficient is determined based on the dynamic fusion weights of each mode before the loss of tracking, and is used to characterize the reliability of the target fusion state before the loss of tracking. The method for generating the predicted search region is as follows: the basic search region is determined based on the fusion position, fusion speed and fusion direction before the target is lost; the basic search region is expanded based on the number of frames of loss; and the expanded search region is corrected by combining the state estimation confidence coefficient. The smaller the state estimation confidence coefficient, the larger the prediction search area.

[0008] Preferably, in step S6, the re-association score is determined based on at least two of the following: spatial location similarity, velocity direction similarity, target category consistency, and cross-modal mapping consistency. When the re-association score is greater than the first re-association threshold, or when the re-association score is greater than the second re-association threshold in consecutive preset frames, the preset recovery threshold condition is determined to be met.

[0009] Preferably, in step S9, the trajectory prediction deviation includes one or more of the following: spatial deviation between predicted position and actual position, velocity deviation between predicted velocity and actual velocity, and angular deviation between predicted direction and actual direction. When the trajectory prediction deviation is greater than the first deviation threshold within a consecutive preset number of samplings, or when the trajectory prediction deviation is greater than the second deviation threshold and the spatial deviation vector from the predicted position to the actual position points to a preset sensitive area, the trajectory prediction deviation is determined to meet the preset abnormality determination condition.

[0010] Preferably, the early warning results are output in a graded manner based on the magnitude of the trajectory prediction deviation, the duration of the deviation, the target category, the risk level of the area where the target is located, and the reliability of the trajectory; The reliability of the trajectory is determined at least by the target's continuous tracking stability and the re-association recovery.

[0011] Secondly, the present invention also provides a multimodal dynamic target perception and tracking early warning system applicable to the above-mentioned method, the system comprising the following modules: The multimodal data acquisition module is used to acquire multimodal sensing data of the monitoring area; The spatiotemporal synchronization and calibration module is connected to the multimodal data acquisition module and is used to perform time synchronization and spatial registration of the multimodal sensing data in order to establish a unified reference coordinate system. The target detection and feature extraction module is connected to the spatiotemporal synchronization and calibration module and is used to extract target candidate observation information in each modality. The perception quality assessment module is connected to the spatiotemporal synchronization and calibration module and is used to obtain the perception quality parameters corresponding to each mode. The state initialization and short-term prediction module is connected to the fusion tracking module. When there is a fusion state of the target at the previous moment, it generates the short-term prediction state of the current moment based on the fusion state of the previous moment; when there is no fusion state of the target at the previous moment, it establishes the initial fusion state of the target based on the target candidate observation information of each modality at the current moment. The dynamic fusion weight allocation module is connected to the target detection and feature extraction module, the perception quality assessment module, and the state initialization and short-term prediction module, respectively. It is used to determine the dynamic fusion weight corresponding to each modality based on the perception quality parameters, detection confidence, and consistency parameters between the target candidate observation information and the short-term prediction state of each modality. The fusion tracking module, connected to the dynamic fusion weight allocation module and the target detection and feature extraction module, is used to fuse the target candidate observation information of each modality based on the dynamic fusion weight corresponding to each modality, so as to obtain the fusion state of the target at the current moment and update the target trajectory information. The tracking loss management and re-association recovery module is connected to the fusion tracking module, the state initialization and short-term prediction module, the dynamic fusion weight allocation module, and the target detection and feature extraction module. It is used to switch the corresponding target trajectory to a tracking loss cache state when the target trajectory fails to complete the association matching with the new target candidate observation information in the current frame within a consecutive preset frame. It generates a prediction search area based on the fusion state before the target tracking loss, the latest short-term prediction state, and the confidence coefficient of the state estimation before the tracking loss. It calculates the re-association score in the prediction search area and restores the original target trajectory while keeping the original target identifier unchanged when the preset recovery threshold condition is met. The trajectory prediction and deviation analysis module is connected to the fusion tracking module and the tracking loss management and re-association recovery module. It is used to generate a predicted trajectory for a future preset sampling time based on the target's historical trajectory sequence in an active or recovered state, and to form the target's actual trajectory after obtaining the target's actual observation state at the future preset sampling time. Then, it calculates the trajectory prediction deviation between the actual trajectory and the predicted trajectory. The graded early warning output module is connected to the trajectory prediction and deviation analysis module. It is used to determine that the target has abnormal behavior and output the corresponding early warning result when the trajectory prediction deviation meets the preset abnormal judgment conditions.

[0012] Preferably, the tracking loss management and re-association recovery module is specifically used to determine the state estimation reliability coefficient based on the dynamic fusion weights of each mode before tracking loss, and to determine the basic search area based on the fusion position, fusion speed and fusion direction before the target tracking loss, and then to expand the basic search area based on the number of frames of tracking loss, and to correct the expanded search area in combination with the state estimation reliability coefficient. The trajectory prediction and deviation analysis module is specifically used to determine the trajectory prediction deviation based on at least one of the spatial deviation between the predicted position and the actual position, the velocity deviation between the predicted velocity and the actual velocity, and the angular deviation between the predicted direction and the actual direction. The graded early warning output module is specifically used to grade and output early warning results based on the magnitude of trajectory prediction deviation, duration of deviation, target category, risk level of the target's location area, and trajectory reliability.

[0013] The present invention has the following beneficial effects: This invention effectively overcomes the aforementioned deficiencies in existing technologies by constructing a technical solution that combines multimodal perception, state prediction, dynamic fusion, tracking loss recovery, and deviation warning. First, this invention dynamically determines the fusion weights based on the perception quality parameters, detection confidence, and consistency parameters between each modality and the short-term predicted state. This allows different modalities to adaptively participate in target state estimation under different environmental conditions, improving the accuracy and stability of target perception and tracking in complex scenarios compared to a fixed fusion method. Second, when a target fails to complete association matching for several consecutive frames, this invention does not directly terminate the target trajectory but switches it to a tracking loss buffer state. It then generates a prediction search area by combining the fusion state before tracking loss, the short-term predicted state, and the state estimation confidence coefficient. The original target trajectory is then restored through re-association scoring, effectively reducing trajectory breaks and target label jumps caused by short-term occlusion, cross-movement, or local detection deficiencies. Furthermore, this invention uses the trajectory prediction deviation between the actual trajectory and the predicted trajectory as a basis for judging abnormal behavior. This not only identifies whether a target has entered a sensitive area but also identifies abnormal approaching, abnormal turning, or abnormal acceleration behaviors that deviate from the existing movement trend in advance, thereby improving the timeliness and accuracy of warnings. Attached Figure Description

[0014] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0016] Reference Figure 1 In some embodiments, a multimodal dynamic target perception and tracking early warning method, wherein step S1 "acquiring multimodal perception data of the monitoring area and performing time synchronization and spatial registration on the multimodal perception data to establish a unified reference coordinate system" can be specifically implemented as follows: Multimodal sensing units are deployed within the monitoring area. Each multimodal sensing unit includes at least two different types of sensors, preferably including a visible light camera, an infrared camera, a lidar, and a millimeter-wave radar. The visible light camera is used to collect visible light image data, the infrared camera is used to collect infrared image data, the lidar is used to collect point cloud data, and the millimeter-wave radar is used to collect radar detection data related to target distance, azimuth, and velocity. Each sensor is installed at a corresponding position within the monitoring area so that its monitoring range covers the preset monitoring area.

[0017] During data acquisition, the multimodal data acquisition module receives the raw data output by each sensor and adds timestamp information under a unified time reference to each modal data. For modal data with different sampling frequencies, time window alignment, nearest neighbor matching, interpolation alignment, or trigger synchronization can be used based on a unified time axis to correlate visible light image frames, infrared image frames, laser point cloud frames, and millimeter-wave radar detection frames at the same or adjacent times to obtain time-aligned multimodal synchronization data.

[0018] After time synchronization is completed, spatial registration is performed on the data of each modality. Specifically, the installation position and attitude parameters of each sensor relative to a unified reference coordinate system are pre-calibrated, and the data in each sensor's own coordinate system are converted to the unified reference coordinate system. For image data, the target position in the image can be mapped to the unified reference coordinate system according to the camera parameters. For laser point cloud data, the point cloud coordinates can be converted to the unified reference coordinate system according to the lidar extrinsic parameters. For millimeter-wave radar data, the distance, azimuth, and velocity information can be converted into spatial position and motion information in the unified reference coordinate system according to the millimeter-wave radar installation parameters. The unified reference coordinate system can be the world coordinate system of the monitoring area or a preset global coordinate system.

[0019] After the above processing, multimodal synchronization data with unified time labels and located in a unified reference coordinate system are obtained, which serves as the data basis for subsequent steps to extract target candidate observation information, obtain perception quality parameters, and perform multimodal fusion, tracking loss correlation and trajectory prediction deviation analysis.

[0020] Based on the multimodal synchronous data obtained in step S1, located in a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light and infrared image data, target candidate regions can be obtained using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target location, target boundary range, target category, and detection confidence score can be extracted. For laser point cloud data, target candidate point clouds can be extracted through clustering segmentation, target bounding box fitting, or point cloud detection methods, and the corresponding spatial location, geometric contour, direction of motion, and detection confidence score can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence score can be obtained based on point aggregation, pre-tracking detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time is formed for each modality.

[0021] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters corresponding to each mode. For image mode, the perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity. For laser point cloud mode, the perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or number of effective points. For millimeter-wave radar mode, the perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. The perception quality parameters are used to characterize the effective perception degree of each mode of the target at the current moment.

[0022] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0023] Based on the multimodal synchronous data obtained in step S1 under a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light image data and infrared image data, target candidate regions can be obtained by using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target position, target boundary range, target category, and detection confidence can be extracted. For laser point cloud data, target candidate point clouds can be extracted by clustering segmentation, target box fitting, or point cloud detection methods, and the corresponding spatial position, geometric contour, motion direction, and detection confidence can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence can be obtained based on point aggregation, target tracking pre-detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time under each modality is formed.

[0024] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters corresponding to each mode. For image mode, the perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity. For laser point cloud mode, the perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or number of effective points. For millimeter-wave radar mode, the perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. The perception quality parameters are used to characterize the effective perception degree of each mode of the target at the current moment.

[0025] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0026] Based on the multimodal synchronous data obtained in step S1, located in a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light and infrared image data, target candidate regions can be obtained using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target location, target boundary range, target category, and detection confidence score can be extracted. For laser point cloud data, target candidate point clouds can be extracted through clustering segmentation, target bounding box fitting, or point cloud detection methods, and the corresponding spatial location, geometric contour, direction of motion, and detection confidence score can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence score can be obtained based on point aggregation, pre-tracking detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time is formed for each modality.

[0027] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters corresponding to each mode. For image mode, the perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity. For laser point cloud mode, the perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or number of effective points. For millimeter-wave radar mode, the perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. The perception quality parameters are used to characterize the effective perception degree of each mode of the target at the current moment.

[0028] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0029] Based on the multimodal synchronous data obtained in step S1 under a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light image data and infrared image data, target candidate regions can be obtained by using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target position, target boundary range, target category, and detection confidence can be extracted. For laser point cloud data, target candidate point clouds can be extracted by clustering segmentation, target box fitting, or point cloud detection methods, and the corresponding spatial position, geometric contour, motion direction, and detection confidence can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence can be obtained based on point aggregation, target tracking pre-detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time under each modality is formed.

[0030] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters corresponding to each mode. For image mode, the perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity. For laser point cloud mode, the perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or number of effective points. For millimeter-wave radar mode, the perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. The perception quality parameters are used to characterize the effective perception degree of each mode of the target at the current moment.

[0031] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0032] Based on the multimodal synchronous data obtained in step S1 under a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light image data and infrared image data, target candidate regions can be obtained by using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target position, target boundary range, target category, and detection confidence can be extracted. For laser point cloud data, target candidate point clouds can be extracted by clustering segmentation, target box fitting, or point cloud detection methods, and the corresponding spatial position, geometric contour, motion direction, and detection confidence can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence can be obtained based on point aggregation, target tracking pre-detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time under each modality is formed.

[0033] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters corresponding to each mode. For image mode, the perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity. For laser point cloud mode, the perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or number of effective points. For millimeter-wave radar mode, the perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. The perception quality parameters are used to characterize the effective perception degree of each mode of the target at the current moment.

[0034] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0035] Based on the multimodal synchronous data obtained in step S1, located in a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light and infrared image data, target candidate regions can be obtained using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target location, target boundary range, target category, and detection confidence score can be extracted. For laser point cloud data, target candidate point clouds can be extracted through clustering segmentation, target bounding box fitting, or point cloud detection methods, and the corresponding spatial location, geometric contour, direction of motion, and detection confidence score can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence score can be obtained based on point aggregation, pre-tracking detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time is formed for each modality.

[0036] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters for each mode. Specifically, for the image mode, perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity; for the laser point cloud mode, perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or the number of effective points; and for the millimeter-wave radar mode, perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. These perception quality parameters characterize the effective perception degree of each mode of the target at the current moment.

[0037] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0038] Based on the multimodal synchronous data obtained in step S1, located in a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light and infrared image data, target candidate regions can be obtained using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target location, target boundary range, target category, and detection confidence score can be extracted. For laser point cloud data, target candidate point clouds can be extracted through clustering segmentation, target bounding box fitting, or point cloud detection methods, and the corresponding spatial location, geometric contour, direction of motion, and detection confidence score can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence score can be obtained based on point aggregation, pre-tracking detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time is formed for each modality.

[0039] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters for each mode. Specifically, for the image mode, perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity; for the laser point cloud mode, perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or the number of effective points; and for the millimeter-wave radar mode, perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. These perception quality parameters characterize the effective perception degree of each mode of the target at the current moment.

[0040] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0041] Based on the multimodal synchronous data obtained in step S1, located in a unified reference coordinate system, target detection or target extraction processing is performed on each modal data to obtain target candidate observation information for each modality. Specifically, for visible light and infrared image data, target candidate regions can be obtained using target detection networks, region extraction algorithms, or foreground segmentation methods, and the corresponding target location, target boundary range, target category, and detection confidence score can be extracted. For laser point cloud data, target candidate point clouds can be extracted through clustering segmentation, target bounding box fitting, or point cloud detection methods, and the corresponding spatial location, geometric contour, direction of motion, and detection confidence score can be obtained. For millimeter-wave radar data, the target's distance, azimuth, velocity, and detection confidence score can be obtained based on point aggregation, pre-tracking detection, or target-level output results. After the above processing, a set of candidate observation information for the target at the same time is formed for each modality.

[0042] While extracting target candidate observation information, the data quality corresponding to each mode is evaluated to obtain the perception quality parameters for each mode. Specifically, for the image mode, perception quality parameters can be determined based on image sharpness, occlusion ratio, brightness distribution, or target imaging integrity; for the laser point cloud mode, perception quality parameters can be determined based on point cloud density, point cloud contour integrity, or the number of effective points; and for the millimeter-wave radar mode, perception quality parameters can be determined based on echo signal-to-noise ratio, velocity measurement stability, and range continuity. These perception quality parameters characterize the effective perception degree of each mode of the target at the current moment.

[0043] Furthermore, to facilitate dynamic fusion weight allocation in subsequent steps, the target candidate observation information extracted from each modality and the corresponding perception quality parameters can be correlated and output according to a unified target time. This provides an input basis for determining the dynamic fusion weights of each modality based on detection confidence, consistency parameters, and perception quality parameters, and provides data support for subsequent target trajectory updates, tracking loss correlation recovery, and anomaly warning judgment.

[0044] In some embodiments, a multimodal dynamic target perception and tracking early warning system is also disclosed, including: a multimodal data acquisition module for acquiring multimodal perception data of the monitoring area; The spatiotemporal synchronization and calibration module is connected to the multimodal data acquisition module and is used to perform time synchronization and spatial registration of the multimodal sensing data in order to establish a unified reference coordinate system. The target detection and feature extraction module is connected to the spatiotemporal synchronization and calibration module and is used to extract target candidate observation information in each modality. The perception quality assessment module is connected to the spatiotemporal synchronization and calibration module and is used to obtain the perception quality parameters corresponding to each mode. The state initialization and short-term prediction module is connected to the fusion tracking module. When there is a fusion state of the target at the previous moment, it generates the short-term prediction state of the current moment based on the fusion state of the previous moment; when there is no fusion state of the target at the previous moment, it establishes the initial fusion state of the target based on the target candidate observation information of each modality at the current moment. The dynamic fusion weight allocation module is connected to the target detection and feature extraction module, the perception quality assessment module, and the state initialization and short-term prediction module, respectively. It is used to determine the dynamic fusion weight corresponding to each modality based on the perception quality parameters, detection confidence, and consistency parameters between the target candidate observation information and the short-term prediction state of each modality. The fusion tracking module, connected to the dynamic fusion weight allocation module and the target detection and feature extraction module, is used to fuse the target candidate observation information of each modality based on the dynamic fusion weight corresponding to each modality, so as to obtain the fusion state of the target at the current moment and update the target trajectory information. The tracking loss management and re-association recovery module is connected to the fusion tracking module, the state initialization and short-term prediction module, the dynamic fusion weight allocation module, and the target detection and feature extraction module. It is used to switch the corresponding target trajectory to a tracking loss cache state when the target trajectory fails to complete the association matching with the new target candidate observation information in the current frame within a consecutive preset frame. It generates a prediction search area based on the fusion state before the target tracking loss, the latest short-term prediction state, and the confidence coefficient of the state estimation before the tracking loss. It calculates the re-association score in the prediction search area and restores the original target trajectory while keeping the original target identifier unchanged when the preset recovery threshold condition is met. The trajectory prediction and deviation analysis module is connected to the fusion tracking module and the tracking loss management and re-association recovery module. It is used to generate a predicted trajectory for a future preset sampling time based on the target's historical trajectory sequence in an active or recovered state, and to form the target's actual trajectory after obtaining the target's actual observation state at the future preset sampling time. Then, it calculates the trajectory prediction deviation between the actual trajectory and the predicted trajectory. The graded early warning output module is connected to the trajectory prediction and deviation analysis module. It is used to determine that the target has abnormal behavior and output the corresponding early warning result when the trajectory prediction deviation meets the preset abnormal judgment conditions.

[0045] Specifically, the tracking loss management and re-association recovery module is used to determine the state estimation confidence coefficient based on the dynamic fusion weights of each mode before tracking loss, and to determine the basic search area based on the fusion position, fusion speed and fusion direction before tracking loss. Then, the basic search area is expanded based on the number of frames of tracking loss, and the expanded search area is corrected in combination with the state estimation confidence coefficient. Specifically, the trajectory prediction and deviation analysis module is used to determine the trajectory prediction deviation based on at least one of the spatial deviation between the predicted position and the actual position, the velocity deviation between the predicted velocity and the actual velocity, and the angular deviation between the predicted direction and the actual direction. Specifically, the graded early warning output module is used to grade and output early warning results based on the magnitude of trajectory prediction deviation, duration of deviation, target category, risk level of the target's location area, and trajectory reliability.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal dynamic target perception, tracking, and early warning method, characterized in that, Includes the following steps: S1. Acquire multimodal sensing data of the monitoring area, and perform time synchronization and spatial registration of the multimodal sensing data to establish a unified reference coordinate system; S2. Extract target candidate observation information based on the synchronized and registered modal sensing data, and obtain the sensing quality parameters corresponding to each modality; S3. When there is a fusion state of the target at the previous moment, generate the short-term prediction state of the current moment based on the fusion state at the previous moment. When there is no previous fusion state of the target, the initial fusion state of the target is established based on the target candidate observation information of each mode at the current time. S4. Based on the perception quality parameters of each modality, the detection confidence level corresponding to each target candidate observation information, and the consistency parameter between the target candidate observation information of each modality and the short-term prediction state, the consistency parameter is used to characterize the matching degree between the target candidate observation information of each modality and the short-term prediction state. The matching degree includes one or more of position deviation, velocity deviation, and direction deviation. Determine the dynamic fusion weight corresponding to each modality. Based on the dynamic fusion weight corresponding to each modality, fuse the target candidate observation information of each modality to obtain the fusion state of the target at the current moment, and update the target trajectory information. S5. When the target trajectory fails to complete the association and matching with the new target candidate observation information in the current frame within consecutive preset frames, the corresponding target trajectory is switched to the lost tracking cache state, and the prediction search area is generated based on the fusion state before the target lost tracking, the latest short-term prediction state, and the state before the loss of tracking to estimate the confidence coefficient. S6. Retrieve new target candidate observation information under each mode within the predicted search area, and calculate the re-association score between the new target candidate observation information and the target trajectory in the lost-tracking state; when the re-association score meets the preset recovery threshold condition, restore the association of the new target candidate observation information to the original target trajectory, and keep the original target identifier unchanged; S7. Generate a predicted trajectory for a future preset sampling time based on the target's historical trajectory sequence in an active or recovered state. S8. At the future preset sampling time, obtain the actual observation state of the target, and form the actual trajectory of the target based on the actual observation state of the target; S9. Calculate the trajectory prediction deviation between the actual trajectory and the predicted trajectory; when the trajectory prediction deviation meets the preset anomaly judgment condition, determine that the target has abnormal behavior and output the corresponding warning result.

2. The multimodal dynamic target perception, tracking, and early warning method according to claim 1, characterized in that, The multimodal sensing data includes two or more of the following: visible light image data, infrared image data, laser point cloud data, and millimeter-wave radar data. Among them, the perception quality parameters corresponding to the image mode include one or more of image sharpness and occlusion ratio; the perception quality parameters corresponding to the laser point cloud mode include one or more of point cloud density and point cloud contour integrity; and the perception quality parameters corresponding to the millimeter-wave radar mode include one or more of echo signal-to-noise ratio, velocity measurement stability, and range continuity.

3. The multimodal dynamic target perception, tracking, and early warning method according to claim 1, characterized in that, In step S5, the state estimation confidence coefficient is determined based on the dynamic fusion weights of each mode before the loss of tracking, and is used to characterize the confidence level of the target fusion state before the loss of tracking. The method for generating the predicted search region is as follows: the basic search region is determined based on the fusion position, fusion speed and fusion direction before the target is lost; the basic search region is expanded based on the number of frames of loss; and the expanded search region is corrected by combining the state estimation confidence coefficient. The smaller the state estimation confidence coefficient, the larger the prediction search area.

4. The multimodal dynamic target perception, tracking, and early warning method according to claim 1, characterized in that, In step S6, the re-association score is determined based on at least two of the following: spatial location similarity, velocity direction similarity, target category consistency, and cross-modal mapping consistency. When the re-association score is greater than the first re-association threshold, or when the re-association score is greater than the second re-association threshold in consecutive preset frames, the preset recovery threshold condition is determined to be met.

5. The multimodal dynamic target perception, tracking, and early warning method according to claim 1, characterized in that, In step S9, the trajectory prediction deviation includes one or more of the following: spatial deviation between predicted position and actual position, velocity deviation between predicted velocity and actual velocity, and angular deviation between predicted direction and actual direction. When the trajectory prediction deviation is greater than the first deviation threshold within a consecutive preset number of samplings, or when the trajectory prediction deviation is greater than the second deviation threshold and the spatial deviation vector from the predicted position to the actual position points to a preset sensitive area, the trajectory prediction deviation is determined to meet the preset abnormality determination condition.

6. The multimodal dynamic target perception, tracking, and early warning method according to claim 1, characterized in that, The early warning results are output in a graded manner based on the magnitude of trajectory prediction deviation, the duration of deviation, the target category, the risk level of the area where the target is located, and the reliability of the trajectory. The reliability of the trajectory is determined at least by the target's continuous tracking stability and the re-association recovery.

7. A multimodal dynamic target perception, tracking, and early warning system, characterized in that, Includes the following modules: The multimodal data acquisition module is used to acquire multimodal sensing data of the monitoring area; The spatiotemporal synchronization and calibration module is connected to the multimodal data acquisition module and is used to perform time synchronization and spatial registration of the multimodal sensing data in order to establish a unified reference coordinate system. The target detection and feature extraction module is connected to the spatiotemporal synchronization and calibration module and is used to extract target candidate observation information in each modality. The perception quality assessment module is connected to the spatiotemporal synchronization and calibration module and is used to obtain the perception quality parameters corresponding to each mode. The state initialization and short-term prediction module is connected to the fusion tracking module and is used to generate the short-term prediction state of the current moment based on the fusion state of the previous moment when there is a fusion state of the target at the previous moment. When there is no previous fusion state of the target, the initial fusion state of the target is established based on the target candidate observation information of each mode at the current time. The dynamic fusion weight allocation module is connected to the target detection and feature extraction module, the perception quality assessment module, and the state initialization and short-term prediction module, respectively. It is used to determine the dynamic fusion weight corresponding to each modality based on the perception quality parameters, detection confidence, and consistency parameters between the target candidate observation information and the short-term prediction state of each modality. The consistency parameters are used to characterize the matching degree between the target candidate observation information and the short-term prediction state of each modality. The matching degree includes one or more of position deviation, velocity deviation, and orientation deviation. The fusion tracking module, connected to the dynamic fusion weight allocation module and the target detection and feature extraction module, is used to fuse the target candidate observation information of each modality based on the dynamic fusion weight corresponding to each modality, so as to obtain the fusion state of the target at the current moment and update the target trajectory information. The tracking loss management and re-association recovery module is connected to the fusion tracking module, the state initialization and short-term prediction module, the dynamic fusion weight allocation module, and the target detection and feature extraction module. It is used to switch the corresponding target trajectory to a tracking loss cache state when the target trajectory fails to complete the association matching with the new target candidate observation information in the current frame within a consecutive preset frame. It generates a prediction search area based on the fusion state before the target tracking loss, the latest short-term prediction state, and the confidence coefficient of the state estimation before the tracking loss. It calculates the re-association score in the prediction search area and restores the original target trajectory while keeping the original target identifier unchanged when the preset recovery threshold condition is met. The trajectory prediction and deviation analysis module is connected to the fusion tracking module and the tracking loss management and re-association recovery module. It is used to generate a predicted trajectory for a future preset sampling time based on the target's historical trajectory sequence in an active or recovered state, and to form the target's actual trajectory after obtaining the target's actual observation state at the future preset sampling time. Then, it calculates the trajectory prediction deviation between the actual trajectory and the predicted trajectory. The graded early warning output module is connected to the trajectory prediction and deviation analysis module. It is used to determine that the target has abnormal behavior and output the corresponding early warning result when the trajectory prediction deviation meets the preset abnormal judgment conditions.

8. A multimodal dynamic target perception and tracking early warning system according to claim 7, characterized in that, The lost tracking management and re-association recovery module is specifically used to determine the state estimation reliability coefficient based on the dynamic fusion weight of each mode before lost tracking, and to determine the basic search area based on the fusion position, fusion speed and fusion direction before the target lost tracking, and then expand the basic search area based on the number of frames of lost tracking, and to correct the expanded search area in combination with the state estimation reliability coefficient. The trajectory prediction and deviation analysis module is specifically used to determine the trajectory prediction deviation based on at least one of the spatial deviation between the predicted position and the actual position, the velocity deviation between the predicted velocity and the actual velocity, and the angular deviation between the predicted direction and the actual direction. The graded early warning output module is specifically used to grade and output early warning results based on the magnitude of trajectory prediction deviation, duration of deviation, target category, risk level of the target's location area, and trajectory reliability.

Images