A target recognition method based on multi-modal spatio-temporal feature fusion

Abstract

The application discloses a target recognition method based on multi-modal space-time feature fusion, comprising the following steps: S1, collecting multi-source heterogeneous data, and performing space registration on the multi-source heterogeneous data; S2, candidate region extraction based on deep instance segmentation: adopting a YOLOv8-seg model to process an RGB image, and outputting a candidate region set; S3, performing geometric checking on each candidate region; S4, material identification of the candidate region based on thermodynamic gradient; S5, distinguishing rigid bodies and flexible bodies of the candidate region through time sequence dynamics characteristics; S6, comprehensive confidence evaluation and decision output. The application constructs a three-in-one layered filtering system of "geometry-thermology-dynamics", and through multi-modal data fusion and physical feature checking, can realize accurate recognition of non-rigid interference targets.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, multimodal data fusion, intelligent sensing, and intelligent security technology. Specifically, it relates to a target recognition method based on multimodal spatiotemporal feature fusion that can penetrate visual interference. By integrating geometric dimensions, thermodynamic features, and temporal dynamic features, it provides an intelligent recognition method and system that effectively distinguishes between two-dimensional perspective graffiti on the ground, inflatable flexible advertising models, and real industrial equipment or vehicles. Background Technology

[0002] In scenarios such as complex industrial site monitoring, temporary municipal road occupancy operations, and field surveys, management often employs low-cost visual interference methods for temporary marking or flexible display models to meet requirements such as safety isolation, work boundary marking, personnel flow guidance, and visual warnings. Common interference methods include:

[0003] (1) Two-dimensional perspective graffiti: Using the principle of visual illusion, draw the outline of equipment or temporary markings with a three-dimensional sense directly on the ground or unhardened area.

[0004] (2) Flexible inflatable model: a highly realistic display model or temporary isolation facility made of polymer composite materials, which is filled with gas to maintain its shape.

[0005] The components or patterns used in the aforementioned interference methods generally possess characteristics such as high-saturation colors, periodic textures, quasi-periodic deformation caused by wind / vibration, and transient high-gloss responses to illumination / vehicle headlights. These characteristics frequently affect the normal operation of automated monitoring systems. In the foreground extraction, motion detection, target classification, and anomaly detection stages of fixed-view machine vision, these interferences are easily misidentified as intrusion targets, flashing lights, abandoned objects, or background drift sources, resulting in a large number of semantically irrelevant alarms, leading to high false alarm rates and wasted resources (such as security personnel and computing resources). Therefore, in automated monitoring systems, target recognition technology is needed to eliminate these interferences. However, existing target recognition technologies have the following significant drawbacks:

[0006] (1) Single-modal dependence: Existing systems mostly rely on a single high-definition visible light imaging (RGB). Although the resolution is high, it is easily affected by lighting conditions, shadows, surface coatings and visual deception (such as two-dimensional graffiti), making it difficult to accurately distinguish between genuine and fake.

[0007] (2) Lack of physical property perception: Traditional algorithms can only identify shape and texture, but cannot perceive the physical properties of the target (such as mass, hardness, thermal inertia). This makes it impossible for the system to distinguish between "rigid bodies" (such as metal equipment, vehicles) and "soft bodies" (such as inflatable models, plastic products).

[0008] (3) Lack of temporal features: Static image analysis ignores the dynamic features of the target in the time dimension. The inflatable model will produce slight low-frequency pulsations (breathing effect) under the influence of the environmental wind field, while the real metal target remains absolutely rigid. Existing algorithms lack the ability to extract this key fingerprint feature.

[0009] (4) Data silos: Although some high-end platforms are equipped with LiDAR or infrared sensors, they lack effective back-end fusion algorithms. The data are independent of each other and fail to form a "1+1>2" analytical synergy.

[0010] Therefore, there is an urgent need for an intelligent recognition method that can integrate multi-dimensional physical characteristics, especially the ability to quantify the essential difference between the "rigidity" and "flexibility" of a target. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a target recognition method based on multimodal spatiotemporal feature fusion. This invention utilizes the elevation information of LiDAR point clouds to eliminate two-dimensional graffiti drawn on the ground, and uses the thermal inertia characteristics of infrared thermal imaging to eliminate isothermal false targets without internal heat sources; it also uses video temporal analysis to capture the micro-motion characteristics of inflatable models to eliminate flexible interference objects. It enables fully automatic and real-time target detection.

[0012] The objective of this invention is achieved through the following technical solution: a non-rigid interference target recognition method based on multimodal spatiotemporal feature fusion, comprising the following steps:

[0013] S1. Collect multi-source heterogeneous data and perform spatial registration on the multi-source heterogeneous data; including the following sub-steps:

[0014] S1-1. Acquiring multi-source heterogeneous data: Using three types of sensors—visible light imaging unit, LiDAR detection unit, and mid-wave infrared thermal imaging unit—visible light images, three-dimensional point cloud data maps, and infrared images of the same area are acquired respectively.

[0015] S1-2. Spatial registration of the collected multi-source heterogeneous data; the specific operation method is as follows:

[0016] S1-2-1, LiDAR point cloud projection: The coordinates of the 3D point cloud data are converted into homogeneous coordinates. The homogeneous coordinates are then projected onto the camera coordinate system through a rigid body transformation matrix, and then projected onto the visible light coordinate system through an intrinsic parameter matrix to obtain 2D point cloud coordinates. The point cloud data is then converted into a LiDAR depth map corresponding to the RGB image.

[0017] S1-2-2, Infrared-Visible Registration: The ORB feature point matching algorithm is used to affinely transform the infrared image to the visible coordinate system, forming an infrared temperature map corresponding to the RGB image;

[0018] S2. Candidate Region Extraction Based on Deep Instance Segmentation: The YOLOv8-seg model is used to process the RGB image, outputting a set of candidate regions; each candidate region... Includes pixel-level masks for candidate regions Candidate region bounding box Preliminary category confidence of candidate regions ;

[0019] S3. Perform geometric verification on each candidate region;

[0020] S4. Material identification of candidate regions based on thermodynamic gradient;

[0021] S5. Differentiate rigid and flexible bodies in candidate regions using temporal dynamic characteristics;

[0022] S6. Comprehensive confidence assessment and decision output, including two decision modes: strict triple verification mode or weighted confidence mode; under strict triple verification mode, if all verifications in S3~S5 are passed, it is determined to be the real target; under weighted confidence mode, the probability of determining it to be the real target is output through confidence level.

[0023] The beneficial effects of this invention are: This invention constructs a three-in-one hierarchical filtering system integrating geometry, thermal dynamics, and kinetics, achieving accurate identification of non-rigid interference targets through multimodal data fusion and physical feature verification, specifically manifested in:

[0024] 1. Use the elevation information of LiDAR point clouds to remove two-dimensional graffiti drawn on the ground.

[0025] 2. Utilize the thermal inertia characteristics of infrared thermal imaging to eliminate isothermal false targets (such as inflatable models) without internal heat sources.

[0026] 3. Utilize video time-series analysis to capture the micro-motion characteristics of the inflatable model and eliminate flexible interference objects.

[0027] 4. Construct a multimodal feature fusion model to output a high-confidence target authenticity determination result, thereby optimizing the allocation of valuable security or monitoring resources.

[0028] Therefore, this invention proposes a method / system for identifying and excluding such "legitimate scenario soft false targets" in an automated monitoring pipeline, so that monitoring attention can be focused on the security events that really need to be addressed, rather than conflicting with the necessary on-site security management measures. Attached Figure Description

[0029] Figure 1 This is a flowchart of the target recognition method of the present invention;

[0030] Figure 2 This explains the principle behind LiDAR's ability to identify ground graffiti using elevation variance.

[0031] Figure 3 Time-series displacement diagrams for rigid and flexible bodies;

[0032] Figure 4 The time-series amplitude diagrams are for rigid and flexible bodies. Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, a target recognition method based on multimodal spatiotemporal feature fusion according to the present invention includes the following steps:

[0035] S1. Collect multi-source heterogeneous data and perform spatial registration on the multi-source heterogeneous data; including the following sub-steps:

[0036] S1-1. Acquiring multi-source heterogeneous data: The system is equipped with a multi-sensor array, using three types of sensors: visible light imaging unit, LiDAR detection unit, and mid-wave infrared thermal imaging unit, to acquire visible light images, 3D point cloud data maps, and infrared images of the same area respectively; covering three perception dimensions: optical, geometric, and thermal, and achieving spatiotemporal consistency of multimodal data through hardware-level synchronization and software-level registration.

[0037] Visible light imaging unit: Employs a full-color high-definition industrial camera with a zoom lens to acquire visible light images with a ground resolution (GSD) better than 0.1m. The camera outputs RGB images. , Represents the coordinates of a point on a visible light image. express The pixel value of a point. This camera supports 1080P@30fps real-time output for capturing target texture, color, and semantic information (such as equipment coatings and markings).

[0038] LiDAR detection unit: Emits laser pulses of a preset wavelength, receives the echoes, and generates a 3D point cloud data map based on the echoes. ,in For the first The three-dimensional coordinates of a point cloud data The total number of point cloud data. For the first The laser reflection intensity of point cloud data. High-density three-dimensional point cloud data is acquired using a LiDAR detection unit to analyze the physical dimensions and elevation characteristics of the target.

[0039] Mid-wave infrared thermal imaging unit: Uses a cooled mid-wave infrared detector to acquire infrared images. This detector responds to thermal radiation signals with wavelengths of 3μm-5μm and outputs a single-channel temperature matrix with a resolution of 640×512. With a temperature sensitivity of ⩽50mK, it is used to analyze the thermophysical properties of a target (such as internal heat sources and thermal inertia).

[0040] Before data collection using the sensor, the sensor's time is first corrected using the time difference between the BeiDou / GPS timestamp and the sensor's local timestamp; let the BeiDou / GPS timestamp be... The local timestamps of the three types of sensors—visible light imaging unit, LiDAR detection unit, or mid-wave infrared thermal imaging unit—are as follows: The time deviation is Utilizing time deviation The data timestamps are corrected using the following formula:

[0041] (1);

[0042] That is, using the corrected time Replace the original time .

[0043] S1-2. Spatial registration of the acquired multi-source heterogeneous data; the specific operation method is as follows: pre-complete the sensor intrinsic and extrinsic parameter calibration, and store the intrinsic parameter matrix of the industrial camera. Rigid body transformation matrix from LiDAR detection unit to industrial camera The heterogeneous data is mapped to the visible light coordinate system through the following steps:

[0044] S1-2-1, LiDAR Point Cloud Projection: Converting the coordinates of 3D point cloud data into homogeneous coordinates, and then transforming the homogeneous coordinates using a rigid body transformation matrix. Projected onto the camera coordinate system, and then through the intrinsic parameter matrix. Projecting the perspective image onto the visible light coordinate system yields the 2D point cloud coordinates. The point cloud data is converted into a LiDAR depth map corresponding to the RGB image. (Including 2D point cloud coordinates and the laser reflection intensity corresponding to each coordinate point), the size of the LiDAR depth map is the same as that of the RGB image, and the 2D point cloud coordinates on the LiDAR depth map correspond one-to-one with the coordinates of the RGB image. The value of each point on the LiDAR depth map is the laser reflection intensity at that point; the reprojection error verification formula is:

[0045] (2);

[0046] in, These are the true coordinates of the image feature points;

[0047] like Then recalibrate the rigid body transformation matrix. Reproject;

[0048] S1-2-2, Infrared-Visible Registration: The ORB feature point matching algorithm is used to extract 500 feature points from both the infrared and visible light images. A brute-force matcher (BFMatcher, Hamming distance) is used to filter high-quality matching pairs. When the number of matches exceeds 10, the affine transformation matrix is ​​calculated using estimateAffine2D to warp the infrared image to the visible light coordinate system, forming an infrared temperature map corresponding to the RGB image. (The coordinates of the infrared temperature map correspond one-to-one with the coordinates of the RGB image, and the value of each coordinate point is the infrared temperature value of that point.)

[0049] S2. Candidate Region Extraction Based on Depth Instance Segmentation: After registration and normalization in S1, the three modalities of data have been unified to the visible light image plane. To reduce the computational load of subsequent high-precision physical verification, this step uses the YOLOv8-seg model to perform fast instance segmentation and coarse screening of the RGB image, outputting a set of candidate regions. , The number of candidate regions output by the model; each candidate region Includes pixel-level masks for candidate regions Candidate region bounding box Preliminary category confidence of candidate regions Regions outside the candidate bounding boxes are no longer included in subsequent calculations, thus significantly reducing the computational load of the entire image and enabling rapid localization of potential target regions. For each candidate region, a continuous 30-second video sequence (30fps, 900 frames in total) is associated for subsequent dynamic analysis.

[0050] Model architecture design:

[0051] ① Backbone network: The CSPDarknet structure is adopted. The feature map is divided into two parts through the cross-stage local network (CSPNet). One part is processed by convolution, and the other part is directly passed to the next stage to enhance gradient flow propagation, reduce computational redundancy, and extract multi-scale deep features (such as the contour and texture of the target).

[0052] ② Neck network: A path aggregation network (PANet) is adopted; a bidirectional feature fusion path is constructed from top to bottom and bottom to top to fully integrate deep semantic information (such as target category) with shallow position information (such as edge details) to improve the detection capability of small targets (such as distant interfering targets) and occluded targets (such as semi-buried targets);

[0053] Loss function design: Total loss The weighted sum of classification loss, bounding box loss, and segmentation loss:

[0054] (3);

[0055] in, , , These are the classification loss weights, bounding box loss weights, and segmentation loss weights, respectively; the default values ​​for each weight coefficient are as follows: Classification loss weights (Balancing classification and regression tasks); Bounding box loss weights (Focusing on positioning accuracy); Segmentation loss weights (Prioritize mask accuracy).

[0056] For classification loss, binary cross-entropy loss is used to measure the probability that the target is a "distractor target". The calculation formula is as follows:

[0057] (4);

[0058] The labels are real (1 for real target, 0 for background). For the model to predict the first The probability that a candidate region is the real target. This represents the number of candidate regions.

[0059] For the bounding box loss, CIoU loss is used, and the calculation formula is as follows:

[0060] (5);

[0061] in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. The Euclidean distance between the center points of the two frames. The minimum diagonal length of the two enclosing frames. For aspect ratio consistency, The balance coefficient is set to 0.5. The CIoU loss takes into account overlap, center point distance, and aspect ratio consistency.

[0062] For segmentation loss, Dice loss is used to measure the overlap between the predicted mask and the ground truth mask. The calculation formula is as follows:

[0063] (6);

[0064] For predicting the mask (Boolean matrix). For a realistic mask, This is an element-wise product.

[0065] The inference and selection process of the YOLOv8-seg model is as follows:

[0066] ① Input: RGB image used as a reference ;

[0067] ② Inference Output: The model outputs a set of candidate regions. Each candidate region Include:

[0068] Pixel-level mask : A Boolean matrix that marks the candidate regions in the image;

[0069] bounding box : The bounding rectangle of the candidate region;

[0070] Preliminary Category Confidence The probability that the candidate region is an "interference target";

[0071] The candidate regions output by the YOLOv8-seg model are processed as follows:

[0072] ③ Non-maximum suppression (NMS): For overlapping candidate regions and Calculate its intersection-union ratio ,reserve For the region <0.5, the formula is:

[0073] (7);

[0074] ④ Coarse sieve, the rule is: only retain Candidate regions that are selected will proceed to the next stage of the process, while other regions will be filtered directly. Simultaneously, the retained preliminary category confidence scores will undergo temperature scaling calibration, using the following formula:

[0075] (8);

[0076] in The temperature coefficient enhances the reliability of the confidence level. and These represent the initial category confidence levels before and after calibration, respectively.

[0077] S3. Perform geometric verification on each candidate region to filter out interfering targets. This step targets 2D perspective graffiti on the ground (such as temporary ground markings), filtering out interfering targets by verifying the 3D volumetric features of the targets. Specifically, it includes the following sub-steps:

[0078] S3-1, Mask and Point Cloud Matching: Based on the correspondence between the LiDAR depth map and the RGB image established in S1-2-1, all candidate regions are found on the LiDAR depth map; then, based on the correspondence between the coordinates of the 3D point cloud data and the coordinates of the 2D point cloud, all candidate regions are found on the 3D point cloud data map, and the 3D point cloud data set of each candidate region is extracted. .

[0079] Point cloud sparsity adaptive verification: Calculate the average target distance from the candidate region point cloud to the sensor origin. (Unit: m) Obtained from LiDAR point cloud computing: Then, based on the target distance... The formula for adjusting the minimum point cloud number threshold is:

[0080] (9);

[0081] The greater the distance, the more point cloud data is required; if The number of point clouds within is less than the minimum point cloud number threshold. If the point cloud is determined to be sparse (e.g., the target is too small or is occluded), the candidate region is directly rejected to avoid misjudgment; otherwise, the candidate region is retained and further step S3-2 is executed.

[0082] S3-2, Calculate elevation variance; extract... All of the points Using axis coordinates as elevation values, calculate the average elevation:

[0083] ;

[0084] For the first Elevation values ​​of each point for The number of midpoints;

[0085] Calculate the elevation variance:

[0086] ;

[0087] Chi-square test is used to determine whether a candidate region is an entity target: the preset threshold is... null hypothesis It is determined to be two-dimensional graffiti; alternative hypothesis It is determined to be an entity target; the principle is as follows: Figure 2 As shown;

[0088] The test statistic is:

[0089] (10);

[0090] like > If so, then refuse. ,choose The candidate region is identified as an entity target, and its geometric score is output. Otherwise, it is judged as a 2D doodle and output. And remove the candidate region, where, At the significance level, For degrees of freedom The critical value of the chi-square distribution.

[0091] If the candidate region is judged to be a sparse point cloud or a two-dimensional graffiti in this step ( If the material is not specified, it will be directly excluded and will no longer participate in the material identification of S4 and the rigidity identification of S5.

[0092] S4. Material identification of candidate regions based on thermodynamic gradient; This step is for flexible inflatable dummy targets (such as rubber / fabric models) and distinguishes them from real metal entities by the difference in thermal characteristics.

[0093] Specifically, it includes the following sub-steps:

[0094] S4-1. Thermal feature extraction, including the following sub-steps:

[0095] S4-1-1, Calculation of candidate target region temperature: After registration in S1-2-2, the infrared temperature frame has been resampled to the visible light image pixel grid, forming an infrared temperature map of the same size as the RGB image. (Unit: °C). Based on the correspondence between the infrared temperature image and the RGB image obtained after registration in S1-2-2-2, each candidate region is found on the infrared temperature image; based on the mask of the candidate region... Extract the infrared temperature value of each point within the target area. And calculate the average temperature of the target area. and the temperature variance of the target area ; target area is ;

[0096] S4-1-2, Calculation of candidate background region temperature: via mask inversion. Extract the infrared temperature value of each point within the background area. And calculate the average temperature of the background area. and background temperature variance The background area is ;

[0097] S4-1-3, Calculation of relative temperature difference: ;

[0098] S4-1-4 Calculate the coefficient of variation of background temperature The formula for measuring the degree of background fluctuation is:

[0099] (11);

[0100] in The standard deviation of the background temperature. Background average temperature; requirements (Background temperature fluctuation <5%)

[0101] S4-2. Set a threshold and make a judgment: Real metal targets generate heat due to internal electronic components (such as drive motors and servo systems), exhibiting a "hot inside, cold outside" gradient. Larger surface temperature distribution, and uneven surface temperature distribution. (Higher); while inflatable dummy targets have no internal heat source and conduct heat evenly. Approaching 0, the temperature distribution is highly uniform and the thermal inertia is small. Therefore, this invention sets the following three threshold values:

[0102] Relative temperature difference threshold This means that the target needs to be at least 5°C hotter than the background temperature.

[0103] Target temperature variance threshold That is, the target itself needs to have a background with sufficient temperature texture changes;

[0104] Temperature variance threshold This indicates that the background temperature needs to be relatively stable;

[0105] If both conditions are met , , , If the target is determined to be a real target, a thermal score is output. =100; otherwise, it is judged as an inactive false target and outputs... =0, and remove the target.

[0106] S5. Rigidity Identification Based on Micro-Pulsating Spectrum: This step is the core innovation. Targeting the "breathing effect" (micro-amplitude, low-frequency pulsations under wind conditions) of the inflatable model, it distinguishes between rigid and flexible bodies in the candidate region using temporal dynamic characteristics. The micro-pulsation diagrams of rigid and flexible bodies are shown below. Figure 3 and Figure 4 As shown. Among them, Figure 3 The time-series displacement diagrams are for rigid and flexible bodies. Figure 4 The figures show the time-series amplitude diagrams for rigid and flexible bodies. In the figures, the solid line (with dots) represents the power spectral density of the rigid entity target, with no significant peaks in the low-frequency range; the dashed line / peaks represent the concentration of low-frequency resonance energy in the flexible inflatable model.

[0107] The specific operation includes the following sub-steps:

[0108] S5-1, Time Sequence Validation: Based on the current system sampling time... Using time as a reference, RGB images of candidate regions over a past period are extracted from the system-cached video stream to form a historical data frame sequence. , The total number of frames in the image; if If the time sequence is insufficient, a rigid score is output for the candidate region. At the same time, the candidate region is marked as "dynamic features not verified" to avoid missed detections. Then, step S5-2 is executed, which analyzes the spatial continuity, temperature evolution trend, and point cloud density changes of the candidate target in the historical sequence to comprehensively verify the authenticity and physical stability of the current detection result. In this embodiment, a continuous image sequence of the candidate region within the past 30 seconds is obtained. The frame rate is set to 30fps. A total of 900 frames: .

[0109] S5-2, Dense Optical Flow Field Calculation: Converting RGB Images to Grayscale Images Using the Brightness Formula:

[0110] =0.299R+0.587G+0.114B;

[0111] For adjacent grayscale frame pairs The dense optical flow algorithm employs Farnebäck polynomial expansion: based on the assumption of grayscale / brightness conservation. Linearization yields first-order constraints:

[0112] (12);

[0113] in, , It is the optical flow vector; , For spatial gradient, For time gradient;

[0114] And introduce a second-order polynomial model in the local neighborhood , Find the unique solution and output the dense optical flow field for this frame pair. , Represents pixels The displacement vector between two frames. The optical flow field sequence is obtained by sequentially calculating the optical flow field for all adjacent frame pairs within this 30-second time period. The algorithm then statistically analyzes the mean direction and magnitude distribution of optical flow vectors within the candidate region to assess the spatiotemporal consistency of motion in that region, helping to distinguish between real rigid targets and dynamic background interference. The algorithm parameters are set as follows: pyramid scale pyr_scale = 0.5, pyramid levels = 3, window size winsize = 15, iterations = 3, polynomial neighborhood size poly_n = 5, and Gaussian standard deviation poly_sigma = 1.2 to ensure optical flow accuracy.

[0115] S5-3, Displacement Amplitude Statistics: Convert the optical flow field to polar coordinates and extract the displacement amplitude (magnitude). , for the A light flow field In the mask of the candidate region Internal calculation of average displacement amplitude ; The displacement amplitude sequence is obtained. .

[0116] S5-4, Spectrum Analysis and Feature Extraction: [The text abruptly ends here, likely due to an in Performing a Fast Fourier Transform (FFT) to calculate the power spectral density (PSD) to quantify the energy distribution includes the following steps:

[0117] S5-4-1, To Perform mean-reduction operation: for displacement amplitude sequences First, calculate the sequence mean. And by subtracting the mean from the original sequence, a zero-mean sequence D~(t) is generated: To eliminate the influence of DC component caused by overall translation or fixed deviation;

[0118] The technical effect is to eliminate constant offsets in the sequence caused by slight camera shake, slow overall background movement, or gradual changes in lighting, ensuring that the spectral analysis focuses on the AC components generated by the target's own motion.

[0119] S5-4-2, Positive Frequency Extraction: Perform the following on a mean-reduced zero-mean sequence: Point FFT yields the frequency range as follows: complex spectrum , The sampling rate for time-series analysis is 30 fps, as the visible light imaging unit of the system operates at this rate. Because the FFT result of a real-valued signal is conjugate symmetric, and the negative frequency component is a redundant mirror image of the positive frequency, only the first two frequencies need to be retained. 1 data point, corresponding to frequencies from 0 to the Nyquist frequency. The positive frequency component.

[0120] S5-4-3, Power Spectral Density Calculation:

[0121] (13);

[0122] in The sampling rate for time series analysis (e.g., 30Hz sampling frequency). The length of the sampling sequence; in actual calculations, the positive frequency point is used. The discrete one-sided power spectral density is obtained. .

[0123] S5-4-4, Calculation of Total Energy and Low-Frequency Energy: Total Energy Low-frequency energy (0.1Hz−2Hz, corresponding to the resonant frequency of the fabric being blown by the wind). = ;

[0124] S5-4-5, Calculate the proportion of low-frequency energy ;

[0125] S5-5, Dynamic threshold adaptive adjustment: based on real-time wind speed. (Unit: m / s) Adjust threshold The formula is:

[0126] (14);

[0127] like If the target is determined to be a flexible inflatable dummy target, a rigidity score is output. Remove the target; otherwise, output... .

[0128] The judgment logic is as follows: A real rigid body target (such as metal equipment) has extremely small deformation, and its displacement amplitude mainly comes from its own motion (such as rotation and movement), with a frequency spectrum exhibiting a broadband white noise distribution. The amplitude is relatively small (typically <0.6); while the inflatable model, affected by the wind field, produces periodic micro-pulsations, with peaks appearing in the spectrum in the 0.1Hz-2Hz frequency band. Significantly increased (usually >0.6). If If the wind speed is too high, it is determined to be a flexible inflatable false target; otherwise, it is determined to be a rigid target. At the same time, the present invention does not use a completely fixed threshold, but dynamically and adaptively adjusts the threshold through formula (16). The higher the wind speed, the higher the threshold is set, which can avoid misjudgment.

[0129] S6. Comprehensive confidence assessment and decision output, including two decision modes: strict triple verification mode or weighted confidence mode. In strict triple verification mode, if all verifications in S3 to S5 are passed, the target is determined to be the real target. In weighted confidence mode, the probability of determining the target to be the real target is output based on the confidence level. The two modes can be switched according to task requirements.

[0130] The specific operation method of the weighted confidence model is as follows:

[0131] S6-1, Dynamic Weight Adjustment: The weights of each verification item are adaptively adjusted based on environmental conditions (day and night, wind speed), assuming environmental factors... .in, As a diurnal factor, during the daytime At night (The weight of the test is increased in hot weather at night). The wind speed factor is used when the wind speed is <5 m / s. When 5 ≤ wind speed ≤ 10 m / s, When the wind speed is >10 m / s, (The weighting of windy weather has been increased);

[0132] Let the original weights corresponding to the geometric, thermal, and kinetic verifications be respectively... , , The initial values ​​can be set to 0.4, 0.3, or 0.3 (geometric verification has the highest priority); the original weights are adjusted using environmental factors to obtain the new weights as follows:

[0133] (15);

[0134] (16);

[0135] (17);

[0136] S6-2. Construct a linear weighted model and calculate the final confidence level:

[0137] (18);

[0138] S6-3, Bayesian confidence calibration: Introducing prior probability (Representing the base probability of the true target, set to 0.1), calculate the posterior confidence level. :

[0139] (19);

[0140] in, This is the likelihood probability, which corresponds to the weighted score. ;

[0141] S6-4, Decision Output Rule: If If a target is identified as a high-confidence real industrial target, priority processing instructions (such as key monitoring and resource scheduling) are generated, and the target coordinates are added to the key monitoring target database.

[0142] like If a target is identified as a suspected target, a manual review process is triggered (multimodal data is pushed to the monitoring center), and the operator confirms and decides whether to take action.

[0143] like If a target is identified as a distracting false target, its coordinates will be added to the geofence database to prevent duplicate processing and avoid wasting monitoring resources due to repeated identification.

[0144] The identification system of this invention features full-process closed-loop control: the system end-to-end latency is <100ms, adapting to the real-time monitoring closed-loop requirements of "Observation-Adjustment-Decision-Action (OODA)," and can be directly connected to urban security management platforms or industrial IoT systems.

[0145] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A target recognition method based on multimodal spatiotemporal feature fusion, characterized in that, Includes the following steps: S1. Collect multi-source heterogeneous data and perform spatial registration on the multi-source heterogeneous data; Includes the following sub-steps: S1-1. Acquiring multi-source heterogeneous data: Using three types of sensors—visible light imaging unit, LiDAR detection unit, and mid-wave infrared thermal imaging unit—visible light images, three-dimensional point cloud data maps, and infrared images of the same area are acquired respectively. S1-2. Spatial registration of the collected multi-source heterogeneous data; the specific operation method is as follows: S1-2-1, LiDAR point cloud projection: The coordinates of the 3D point cloud data are converted into homogeneous coordinates. The homogeneous coordinates are then projected onto the camera coordinate system through a rigid body transformation matrix, and then projected onto the visible light coordinate system through an intrinsic parameter matrix to obtain 2D point cloud coordinates. The point cloud data is then converted into a LiDAR depth map corresponding to the RGB image. S1-2-2, Infrared-Visible Registration: The ORB feature point matching algorithm is used to affinely transform the infrared image to the visible coordinate system, forming an infrared temperature map corresponding to the RGB image; S2. Candidate Region Extraction Based on Deep Instance Segmentation: The YOLOv8-seg model is used to process the RGB image, outputting a set of candidate regions; each candidate region... Includes pixel-level masks for candidate regions Candidate region bounding box Preliminary category confidence of candidate regions ; S3. Perform geometric verification on each candidate region; S4. Material identification of candidate regions based on thermodynamic gradient; S5. Differentiate rigid and flexible bodies in candidate regions using temporal dynamic characteristics; S6. Comprehensive confidence assessment and decision output, including two decision modes: strict triple verification mode or weighted confidence mode; under strict triple verification mode, if all verifications in S3~S5 are passed, it is determined to be the real target; under weighted confidence mode, the probability of determining it to be the real target is output through confidence level.

2. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, In step S1-1, before data collection using the sensor, the time of the sensor is corrected by using the time deviation between the BeiDou / GPS timestamp and the local timestamp of the sensor.

3. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, In step S2, the candidate regions output by the YOLOv8-seg model are processed as follows: Non-maximum suppression: for overlapping candidate regions and Calculate its intersection-union ratio ,reserve Candidate regions with a value < 0.5; Coarse sieve: retain only The candidate regions are selected, and the remaining regions are filtered directly.

4. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, Step S3 specifically includes the following sub-steps: S3-1, Mask and Point Cloud Matching: Based on the correspondence between the LiDAR depth map and the RGB image established in S1-2-1, find all candidate regions on the LiDAR depth map; Based on the correspondence between the coordinates of the 3D point cloud data and the coordinates of the 2D point cloud, all candidate regions are identified on the 3D point cloud data map, and the 3D point cloud data set of each candidate region is extracted. ;like If the number of point clouds within a region is less than the minimum point cloud number threshold, it is determined to be a sparse point cloud and the candidate region is rejected; otherwise, the candidate region is retained and step S3-2 is further executed. S3-2, Calculate elevation variance; extract... All of the points Using axis coordinates as elevation values, calculate the average elevation: ； For the first Elevation values ​​of each point for The number of midpoints; Calculate the elevation variance: ； The chi-square test is used to determine whether a candidate region is a real target. If it is, the geometric score of the candidate region is output. Otherwise, output And remove the candidate region.

5. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, Step S4 specifically includes the following sub-steps: S4-1. Thermal feature extraction, including the following sub-steps: S4-1-1, Target Area Temperature Calculation: Based on the correspondence between the infrared temperature image and the RGB image obtained in S1-2-2-2, find each candidate area on the infrared temperature image; based on the mask of the candidate area... Extract the infrared temperature value of each point within the target area. ; And calculate the average temperature of the target area. and the temperature variance of the target area ; target area is ; S4-1-2, Background Area Temperature Calculation: By inverting the mask ~ Extract the infrared temperature value of each point within the background area. And calculate the average temperature of the background area. and background temperature variance The background area is ; S4-1-3, Calculation of relative temperature difference: ; S4-1-4 Calculate the coefficient of variation of background temperature : ； in The standard deviation of the background temperature. The background average temperature; S4-2, Set thresholds and perform judgments: Set the following three thresholds: relative temperature difference threshold. Target temperature variance threshold Temperature variance threshold If both conditions are met , , , If the target is determined to be a real target, a thermal score is output. =100; Otherwise, it is judged as an inactive false target and outputs... =0, and remove the target.

6. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, Step S5 specifically includes the following sub-steps: S5-1, Time Sequence Validation: Based on the current system sampling time... Using time as a reference, RGB images of candidate regions over a past period are extracted from the system-cached video stream to form a historical data frame sequence. , The total number of frames in the image; if If the time sequence is insufficient, a rigid score is output for the candidate region. At the same time, the candidate region is marked as "dynamic features not verified"; if Then execute S5-2; S5-2, Dense Optical Flow Field Calculation: Convert the RGB image to a grayscale image according to the brightness formula, and perform calculations on adjacent grayscale frame pairs. The optical flow field sequence was obtained by employing the Farnebäck polynomial expansion dense optical flow algorithm. ; S5-3, Displacement Amplitude Statistics: Convert the optical flow field to polar coordinates, extract the displacement amplitude, and obtain the displacement amplitude sequence. ; S5-4, Spectrum Analysis and Feature Extraction: [The text abruptly ends here, likely due to an in Performing a fast Fourier transform to calculate the power spectral density to quantize the energy distribution includes the following steps: S5-4-1, To Perform a mean-reduction operation; S5-4-2, Positive Frequency Extraction: Perform the following on a mean-reduced zero-mean sequence: Point FFT yields the frequency range as follows: complex spectrum ; S5-4-3, Power Spectral Density Calculation: （13）； in The sampling rate for time series analysis. The length of the sampling sequence; S5-4-4, Calculation of Total Energy and Low-Frequency Energy: Total Energy Low-frequency energy = ; S5-4-5, Calculate the proportion of low-frequency energy ; S5-5, Dynamic threshold adaptive adjustment: based on real-time wind speed. Adjust threshold The formula is: （14）； like If the target is determined to be a flexible inflatable dummy target, a rigidity score is output. Remove the target; otherwise, output... .

7. The target recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, In step S6, the operation method of the weighted confidence pattern is as follows: S6-1, Dynamic Weight Adjustment: Assume environmental factors... ;in, As a diurnal factor, during the daytime At night ; The wind speed factor is used when the wind speed is <5 m / s. When 5 ≤ wind speed ≤ 10 m / s, When the wind speed is >10 m / s, ; Let the original weights corresponding to the geometric, thermal, and kinetic verifications be respectively... , , The original weights were adjusted using environmental factors to obtain the following new weights: （17）； （18）； （19）； S6-2. Construct a linear weighted model and calculate the final confidence level: （20）； S6-3, Bayesian confidence calibration: Introducing prior probability Calculate the posterior confidence level : （21）； in, ; S6-4, Decision Output Rules: like : Determined to be a high-confidence, real industrial target; like : Determined to be a suspected target; like It was determined to be a false target used for interference.

Images