A target recognition and tracking method and system based on multi-source fusion imaging

Abstract

The present invention discloses a target recognition and tracking method and system based on multi-source fusion imaging, including: acquiring multi-modal environmental monitoring data and performing preprocessing, performing image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data; extracting multi-modal features from the image-enhanced multi-modal environmental monitoring data, introducing a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features; constructing a cascaded target recognition model, inputting the cross-modal fusion features to perform target recognition on the current monitoring scene; tracking the scene recognition targets in the current monitoring scene to generate scene target tracking information and storing it in a preset trajectory memory pool; when a tracking target disconnection occurs during target tracking, acquiring a mismatched tracking trajectory and combining it with the trajectory memory pool to repair the tracking trajectory disconnection. Thereby improving the recognition accuracy and tracking reliability of targets in the monitoring scene.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition and tracking, and particularly to a target recognition and tracking method and system based on multi-source fusion imaging. Background Art

[0002] In the technical field of target recognition and tracking, traditional methods usually rely on a single sensor to collect data, facing significant technical bottlenecks in complex environments. Visible light imaging is vulnerable to changes in lighting, adverse weather, and target occlusion. Although infrared thermal imaging has the ability to perceive at night, its spatial resolution is limited. Millimeter-wave radar has inherent defects in target contour recognition. Existing methods based on multi-modal data fusion have improved environmental adaptability to a certain extent, but there are still problems in practical applications, such as insufficient spatio-temporal alignment accuracy of multi-source data, low cross-modal feature association efficiency, and the lack of a trajectory repair mechanism after target loss. In addition, the phenomenon of trajectory interruption caused by environmental interference or sensor disconnection during target tracking is widespread. Traditional algorithms usually rely on linear prediction or trajectory interpolation for repair, and it is difficult to ensure the accuracy and continuity of trajectory association in unstructured dynamic scenarios. These technical defects seriously restrict the demand for continuous and stable tracking of multiple targets in complex scenarios in key fields such as intelligent monitoring, and there is an urgent need for a technical breakthrough through multi-modal deep collaborative perception and an intelligent repair mechanism. Summary of the Invention

[0003] The present invention overcomes the defects of the prior art and provides a target recognition and tracking method and system based on multi-source fusion imaging.

[0004] To achieve the above object, the first aspect of the present invention provides a target recognition and tracking method based on multi-source fusion imaging, including:

[0005] Performing environmental monitoring on a target scene to obtain multi-modal environmental monitoring data and performing preprocessing, and performing image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data;

[0006] Performing multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, and introducing a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features;

[0007] Constructing a cascaded target recognition model, inputting the cross-modal fusion features to perform target recognition on the current monitoring scene to obtain scene target recognition information;

[0008] Tracking the scene recognition targets in the current monitoring scene based on the cross-modal fusion features and the scene target recognition information to generate scene target tracking information and storing it in a preset trajectory memory pool;

[0009] When a disconnection occurs in the tracked target during target tracking, the mismatched tracking trajectory is obtained, and the disconnection of the tracking trajectory is repaired in combination with the trajectory memory pool.

[0010] In this solution, the environmental monitoring of the target scene to obtain multi-modal environmental monitoring data and perform preprocessing specifically includes:

[0011] Install a heterogeneous sensor array in the target scene, and use the installed heterogeneous sensor array to perform environmental monitoring on the target scene to obtain multi-modal environmental monitoring data, where the heterogeneous sensor array includes a high-resolution imaging camera, an infrared thermal imaging camera, and a millimeter-wave radar;

[0012] Based on the obtained multi-modal environmental monitoring data, perform spatial registration on the data obtained from different monitoring sources. Introduce the ORB feature extraction algorithm, extract the scale pyramid features from the initial visible light image, and eliminate the illumination difference through gray value normalization;

[0013] After eliminating the illumination difference, extract the image region features of the initial visible light image after eliminating the illumination difference, and perform threshold segmentation processing on the corresponding region of the initial infrared image through the extracted image region features;

[0014] Extract feature points from the initial visible light image and the initial infrared image respectively, perform preliminary matching on the extracted feature points through the Hamming distance to generate a set of matching points. After completing the preliminary matching, use the RANSAC algorithm to iteratively screen the inlier set;

[0015] Randomly select 4 pairs of feature points each time to calculate the affine transformation matrix, and calculate the reprojection error of all matching segments on this affine transformation matrix. Select the points less than the preset reprojection error and include them in the inlier set. After repeated iteration, retain the final affine transformation matrix and the corresponding inlier set to obtain the spatial registration result of the initial visible light image and the initial infrared image;

[0016] Extract the radar point cloud data from the multi-modal environmental monitoring data, convert the polar coordinates corresponding to the radar point cloud data into Cartesian coordinate system three-dimensional coordinates, and perform rigid body transformation projection onto the imaging plane of the visible light camera in combination with the external parameter calibration matrix of the sensor, and finally generate a two-dimensional point cloud heat map that is spatially aligned with the optical image;

[0017] Combine the spatial registration result of the initial visible light image and the initial infrared image and the two-dimensional point cloud heat map to obtain the preprocessed multi-modal environmental monitoring data.

[0018] In this solution, the image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data specifically includes:

[0019] Obtain the preprocessed multi-modal environmental monitoring data, input it into the HSI model to convert the original RGB image to the HSI color space, and obtain the hue component, saturation component, and brightness component through the HSI color space;

[0020] Introduce the Retinex algorithm to decompose the brightness component into a reflection component and an illumination component, perform digital domain processing on the reflection component to enhance the dark area of the image, perform non-linear adaptive adjustment on the separated saturation component, and enhance the color concentration through a dynamic range expansion function;

[0021] Use the Canny edge detection operator to extract the full-image structure features and form an edge map, perform pixel-level weighted fusion of the edge map and the enhanced brightness component. After processing each component, re-fuse the optimized H, S, and I channels, and reconstruct the RGB image through reverse color space conversion to obtain the initial enhanced image;

[0022] Introduce the Markov algorithm to reprocess the initial enhanced image, import the initial enhanced image into the convolutional autoencoder network for encoding and mapping it to the latent space, and generate an initial latent variable as the starting state of the Markov chain;

[0023] Define the state distribution through the generated initial latent variable and calculate the state transition probability, generate candidate latent variables according to the state transition probability, and use the Metropolis criterion to judge whether to accept the target candidate latent variable;

[0024] Perform iterative optimization until the stop criterion is met, then output the final latent variable. Input the final latent variable into the pre-trained U-Net decoder network, and gradually reconstruct the high-resolution image through the deconvolution layer and skip connections to obtain the image-enhanced multi-modal environmental monitoring data.

[0025] In this solution, for the multi-modal feature extraction of the image-enhanced multi-modal environmental monitoring data, a cross-attention mechanism is introduced to fuse the extracted multi-modal features to construct cross-modal fusion features, which specifically includes:

[0026] Obtain the image-enhanced multi-modal environmental monitoring data, and input the image-enhanced multi-modal environmental monitoring data into a multi-channel feature extraction network for multi-modal feature extraction. The multi-channel feature extraction network includes a first feature extraction channel, a second feature extraction channel, and a third feature extraction channel;

[0027] Import the visible light image into the first feature extraction channel to extract multi-scale context features based on the hybrid dilated convolutional layer, capture context information under different receptive fields according to the parallel convolution branches, and then compress it into a multi-dimensional global vector through global average pooling to obtain the first feature extraction information;

[0028] Import the infrared image into the second feature extraction channel for spatial feature extraction, input the extracted spatial features into the SE attention module to perform global average pooling to generate a channel description vector, use the channel description vector to obtain a channel weight matrix, multiply it with the original feature map, and then extract the local thermal feature vector to obtain the second feature extraction information;

[0029] Import the radar point cloud data into the third feature extraction channel to extract three-dimensional spatial features, select k key points as the initial point set through farthest point sampling, and combine the local aggregation mechanism to capture the geometric contour and motion trajectory of the target object to generate the third feature extraction information;

[0030] Introduce the cross-attention mechanism to perform feature cross-modal association based on the first feature extraction information, the second feature extraction information, and the third feature extraction information. Use the visible light image feature as the query, and the infrared image feature and the radar point cloud feature as the key values. Calculate the spatial correlation weight through multi-head attention to generate the cross-modal fusion feature.

[0031] In this solution, to construct the cascaded target recognition model, input the cross-modal fusion feature to perform target recognition on the current monitoring scene to obtain the scene target recognition information, which specifically includes:

[0032] Construct a cascaded target recognition model. The cascaded target recognition model is constructed with Bi-LSTM and a multi-scale detection head as the framework. Obtain the cross-modal fusion feature and input it into the cascaded target recognition model to analyze the current scene;

[0033] Use the Bi-LSTM network to perform temporal context modeling based on the input cross-modal fusion feature. Based on the stacked two-layer LSTM units, capture the spatio-temporal context features of the objects in the target scene by forward and backward propagation respectively, and output the spatio-temporal enhanced features;

[0034] Input the spatio-temporal enhanced features into the multi-scale detection head for target parsing. Use the lightweight GhostNet module to perform multi-level feature abstraction to obtain multiple groups of low-rank feature maps, and use the cross-stage local connection module to extract local detail features;

[0035] Perform depthwise separable convolution compression on the obtained multiple groups of low-rank feature maps to obtain the channel dimension, perform channel concatenation with the local detail features to obtain the cascaded fusion feature, introduce spatial pyramid pooling and input the cascaded fusion feature, and extract the cross-receptive field features through parallel multi-scale max pooling and splice them with the original feature map to obtain a three-level feature pyramid;

[0036] Based on the obtained three-level feature pyramid, perform scene target recognition on the target scene to generate several candidate recognition frames, and perform non-maximum suppression and mask threshold filtering on the candidate recognition frames to obtain the scene target recognition information.

[0037] In this solution, based on the cross-modal fusion features and the scene target recognition information, the scene recognition target in the current monitoring scene is tracked to generate scene target tracking information and stored in a preset trajectory memory pool, which specifically includes:

[0038] Obtain the cross-modal fusion features and the scene target recognition information, extract the motion state features of the scene recognition target based on the cross-modal fusion features, and import the extracted motion state features into an extended Kalman filter to construct a state vector;

[0039] Based on the constructed state vector, predict the position distribution of the scene recognition target in the next frame image to obtain a target prediction box, calculate the Mahalanobis distance between the target prediction box and the target detection box in combination with the scene target recognition information, and generate a motion cost matrix;

[0040] Obtain the appearance state features of the scene recognition target through the cross-modal fusion features, calculate the cosine similarity with the target appearance embedding in the adjacent frame image, generate an appearance cost matrix, and perform weighted fusion with the motion cost matrix to generate a comprehensive cost matrix as the input of the Hungarian algorithm for solving the optimal matching pair. Generate scene target tracking information according to the obtained optimal matching pair and store it in a preset trajectory memory pool.

[0041] In this solution, when a disconnection occurs to the tracking target during target tracking, obtain the mismatched tracking trajectory, and combine it with the trajectory memory pool to repair the disconnection of the tracking trajectory, which specifically includes:

[0042] Obtain the mismatched tracking trajectory, extract the last frame image where the trajectory mismatch occurs based on the mismatched tracking trajectory, define it as the disconnection repair start frame, and extract the frame timing features and multi-modal features of the disconnection repair start frame to obtain the disconnection repair frame feature information;

[0043] Obtain the trajectory memory pool, extract the scene target tracking trajectory after the frame timing features of the disconnection repair frame based on the disconnection frame feature information, define it as the initial repair tracking trajectory, and extract the motion state attributes of the disconnection repair frame through the multi-modal features of the disconnection repair start frame;

[0044] Extract the motion state attributes of each initial repair tracking trajectory, calculate the deviation from the motion state attributes of the disconnection repair frame, and select the initial repair tracking trajectories with deviations within the preset range as candidate repair tracking trajectories;

[0045] Extract the multi-modal features of each candidate repair trajectory, calculate the similarity with the disconnection repair frame feature information to obtain several candidate repair tracking trajectories with a similarity greater than the preset similarity threshold, and select the candidate repair tracking trajectory with the largest similarity value as the final repair tracking trajectory to connect with the mismatched tracking trajectory.

[0046] In this solution, when a disconnection occurs for the tracked target during target tracking, a mismatched tracking trajectory is obtained, and the disconnection of the tracking trajectory is repaired by combining with the trajectory memory pool. It further includes:

[0047] If the similarity values between the multi-modal features of all candidate repaired tracking trajectories and the disconnection repair frame feature information are all less than the preset similarity threshold, the spatial coordinate data of the scene tracking target in the disconnection repair frame is obtained as the origin to establish a polar coordinate system, and a predicted trajectory for the next k frames is generated based on the motion state attributes of the disconnection repair frame, and the predicted trajectory coordinates are calibrated in the established polar coordinate system;

[0048] According to the multi-modal features of the scene target associated with the starting frame of each candidate repaired tracking trajectory, the spatial coordinate data of the scene tracking target in the starting frame of each candidate repaired tracking trajectory is extracted and calibrated in the established polar coordinate system, which is defined as the point to be determined;

[0049] The predicted trajectory in the polar coordinates is segmented according to the temporal sequence features, and different confidence levels are set for different predicted trajectory segments based on the sequence order of the temporal sequence features. The Manhattan distance between each point to be determined and the predicted trajectory calibrated in the polar coordinate system at different time sequences is calculated in the constructed polar coordinate system;

[0050] According to the calculated Manhattan distance, the predicted trajectory segment with the smallest Manhattan distance is selected as the target predicted trajectory segment corresponding to the point to be determined, and weighted fusion is performed in combination with the confidence level corresponding to the target predicted trajectory segment to obtain the association degree between each point to be determined and the predicted trajectory;

[0051] The association degree between each point to be determined and the preset trajectory is judged against the preset association threshold. If there is no point to be determined greater than the preset association threshold, the target mismatched tracking trajectory is marked as an irreparable trajectory;

[0052] If there is a point to be determined greater than the preset association threshold, the point to be determined with the largest association degree is selected as the target determination point, and the candidate repaired tracking trajectory corresponding to the target determination point is defined as the target repaired tracking trajectory, and it is connected to the target mismatched tracking trajectory to complete the disconnection repair of the trajectory.

[0053] The second aspect of the present invention provides a target recognition and tracking system based on multi-source fusion imaging. The system includes: a memory and a processor. The memory contains a program for the target recognition and tracking method based on multi-source fusion imaging. When the program for the target recognition and tracking method based on multi-source fusion imaging is executed by the processor, the following steps are implemented:

[0054] Perform environmental monitoring on the target scene to obtain multi-modal environmental monitoring data and perform preprocessing, and perform image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data;

[0055] Perform multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, and introduce a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features;

[0056] Construct a cascaded target recognition model, input the cross-modal fusion features to recognize the targets in the current monitoring scene, and obtain scene target recognition information;

[0057] Based on the cross-modal fusion features and the scene target recognition information, track the scene recognition targets in the current monitoring scene to generate scene target tracking information and store it in a preset trajectory memory pool;

[0058] When a disconnection occurs during target tracking, obtain the mismatched tracking trajectory, and combine it with the trajectory memory pool to repair the disconnection of the tracking trajectory.

[0059] The present invention discloses a target recognition and tracking method and system based on multi-source fusion imaging, including: obtaining multi-modal environmental monitoring data and performing preprocessing, performing image enhancement on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data; performing multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, introducing a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features; constructing a cascaded target recognition model, inputting the cross-modal fusion features to recognize the targets in the current monitoring scene; tracking the scene recognition targets in the current monitoring scene to generate scene target tracking information and storing it in a preset trajectory memory pool; when a disconnection occurs during target tracking, obtain the mismatched tracking trajectory, and combine it with the trajectory memory pool to repair the disconnection of the tracking trajectory. Thereby improving the recognition accuracy and tracking reliability of the targets in the monitoring scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments or exemplary examples of the present invention, the following will briefly introduce the drawings required for use in the embodiments or exemplary descriptions. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings shown.

[0061] Figure 1 It is a flowchart of a target recognition and tracking method based on multi-source fusion imaging provided by an embodiment of the present invention;

[0062] Figure 2 It is a flowchart of a tracking disconnection repair method for target recognition and tracking provided by an embodiment of the present invention;

[0063] Figure 3Block diagram of an object recognition and tracking system based on multi-source fusion imaging provided by an embodiment of the present invention;

[0064] The realization, functional characteristics and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0065] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0066] Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0067] Figure 1 Flowchart of an object recognition and tracking method based on multi-source fusion imaging provided by an embodiment of the present invention;

[0068] As Figure 1 shown, the present invention provides a flowchart of an object recognition and tracking method based on multi-source fusion imaging, including:

[0069] S102, performing environmental monitoring on the target scene to obtain multi-modal environmental monitoring data and performing preprocessing, and performing image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data;

[0070] S104, performing multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, and introducing a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features;

[0071] S106, constructing a cascaded object recognition model, inputting the cross-modal fusion features to perform object recognition on the current monitoring scene, and obtaining scene object recognition information;

[0072] S108, tracking the scene recognition objects in the current monitoring scene based on the cross-modal fusion features and the scene object recognition information to generate scene object tracking information and storing it in a preset trajectory memory pool;

[0073] S110, when a tracking target disconnection occurs during object tracking, obtaining a mismatched tracking trajectory, and combining the trajectory memory pool to repair the tracking trajectory disconnection.

[0074] Further, in a preferred embodiment of the present invention, the environmental monitoring of the target scene to obtain multi-modal environmental monitoring data and perform preprocessing specifically includes:

[0075] Install a heterogeneous sensor array in the target scene, and use the installed heterogeneous sensor array to perform environmental monitoring on the target scene to obtain multi-modal environmental monitoring data, where the heterogeneous sensor array includes a high-resolution imaging camera, an infrared thermal imaging camera, and a millimeter-wave radar;

[0076] Based on the obtained multi-modal environmental monitoring data, perform spatial registration on the data obtained from different monitoring sources. Introduce the ORB feature extraction algorithm, extract the scale pyramid features from the initial visible light image, and eliminate the illumination difference through gray value normalization;

[0077] After completing the elimination of the illumination difference, extract the image region features of the initial visible light image after the illumination difference elimination, and perform threshold segmentation processing on the corresponding region of the initial infrared image through the extracted image region features;

[0078] Extract feature points from the initial visible light image and the initial infrared image respectively, perform preliminary matching on the extracted feature points through the Hamming distance to generate a set of matching points. After completing the preliminary matching, use the RANSAC algorithm to iteratively screen the inlier set;

[0079] Randomly select 4 pairs of feature points each time to calculate the affine transformation matrix, and calculate the reprojection error of all matching segments with respect to this affine transformation matrix. Select the points with a reprojection error less than the preset reprojection error and include them in the inlier set. After repeated iteration, retain the final affine transformation matrix and the corresponding inlier set to obtain the spatial registration result of the initial visible light image and the initial infrared image;

[0080] Extract the radar point cloud data from the multi-modal environmental monitoring data, convert the polar coordinates corresponding to the radar point cloud data into Cartesian coordinate system three-dimensional coordinates, and perform rigid body transformation projection onto the imaging plane of the visible light camera in combination with the external parameter calibration matrix of the sensor, and finally generate a two-dimensional point cloud heat map spatially aligned with the optical image;

[0081] Combine the spatial registration result of the initial visible light image and the initial infrared image and the two-dimensional point cloud heat map to obtain the preprocessed multi-modal environmental monitoring data.

[0082] It should be noted that in the target scenario, a heterogeneous sensor array composed of a high-resolution imaging camera, an infrared thermal imaging camera, and a millimeter-wave radar is deployed. Through the collaborative work of multiple sensors, multi-modal environmental monitoring data such as high-resolution image information, infrared thermal radiation distribution, and millimeter-wave radar point cloud of the scenario are collected in real time. Based on the obtained multi-source data, first, cross-modal spatial registration is performed on the visible light and infrared images: for the visible light image, the ORB feature extraction algorithm is used to construct a scale pyramid, and directional FAST corner points are detected at different resolution levels and the rotation-invariant BRIEF binary descriptor is calculated. At the same time, the influence of illumination intensity difference on feature stability is eliminated through gray value normalization; for the infrared image, adaptive threshold segmentation is performed according to the regional features (such as edge gradient distribution) of the visible light image where the illumination difference has been eliminated, and the thermal radiation contour corresponding to the visible light target area is extracted. Subsequently, ORB feature points are respectively extracted from the two-modal images, and the Hamming distance is used to measure the similarity of the binary descriptors to generate a preliminary matching point set, and then the registration parameters are iteratively optimized through the RANSAC algorithm: each time 4 pairs of feature points are randomly selected to calculate the affine transformation matrix, and based on this matrix, the reprojection error of all matching point pairs (i.e., the Euclidean distance between the projected coordinates and the measured coordinates) is calculated, and the points with errors less than the preset threshold are selected and included in the inlier set. After multiple rounds of iteration, the affine transformation matrix with the largest number of inliers and the smallest average error is selected as the final registration parameter to achieve high-precision spatial alignment of the visible light and infrared images.

[0083] Furthermore, for the millimeter-wave radar point cloud data, its original polar coordinates (distance, azimuth angle, elevation angle) are converted into Cartesian coordinate system three-dimensional coordinates, and a rigid body transformation is performed in combination with the sensor extrinsic calibration matrix (rotation matrix and translation vector pre-obtained through a multi-view calibration board), and the three-dimensional point cloud is projected onto the imaging plane coordinate system of the visible light camera. In the projection process, the perspective projection model is used to calculate the two-dimensional coordinates of each radar point on the image plane, and a gray value mapping is generated according to the point cloud reflection intensity to form a two-dimensional point cloud heat map that is spatially aligned with the visible light image. Finally, the visible light-infrared registered image and the radar point cloud heat map are integrated to construct a spatio-temporally consistent multi-modal environmental monitoring data set, providing a fusion data basis with cross-modal spatial alignment characteristics for subsequent target detection and tracking.

[0084] Further, in a preferred embodiment of the present invention, image enhancement is performed on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data, which specifically includes:

[0085] Obtain the preprocessed multi-modal environmental monitoring data, input it into the HSI model to convert the original RGB image to the HSI color space, and obtain the hue component, saturation component, and brightness component through the HSI color space;

[0086] The Retinex algorithm is introduced to decompose the luminance component into a reflection component and an illumination component, and the reflection component is processed in the number domain to enhance the dark area of the image. The separated saturation component is adaptively adjusted non-linearly, and the color concentration is enhanced through a dynamic range expansion function;

[0087] The Canny edge detection operator is used to extract the structural features of the whole image and form an edge map. The edge map is weighted and fused with the enhanced luminance component at the pixel level. After processing each component, the optimized H, S, and I channels are re-fused, and the RGB image is reconstructed through inverse color space conversion to obtain the initial enhanced image;

[0088] The Markov algorithm is introduced to reprocess the initial enhanced image. The initial enhanced image is imported into a convolutional autoencoder network for encoding and mapped to the latent space to generate an initial latent variable as the starting state of the Markov chain;

[0089] The state distribution is defined by the generated initial latent variable and the state transition probability is calculated. Candidate latent variables are generated according to the state transition probability, and the Metropolis criterion is used to judge whether to accept the target candidate latent variable;

[0090] Iterative optimization is performed until the stopping criterion is met, and then the final latent variable is output. The final latent variable is input into a pre-trained U-Net decoder network, and the high-resolution image is gradually reconstructed through deconvolution layers and skip connections to obtain the image-enhanced multi-modal environmental monitoring data.

[0091] It should be noted that first, the visible light RGB image is input into the HSI (Hue, Saturation, Intensity) color space conversion model, and the hue component, saturation component representing color attributes, and the brightness component representing illumination intensity are separated through a non-linear formula. For the brightness component, the Retinex algorithm is introduced for illumination correction: based on multi-scale Gaussian filtering, the brightness layer is decomposed into a reflection component (representing the inherent brightness of the target) and an illumination component (representing ambient illumination interference), a logarithmic transformation is applied to the reflection component to expand the dynamic range of the dark area, and at the same time, adaptive gamma correction is used to further suppress uneven illumination; for the saturation component, a piecewise non-linear stretching function is adopted, exponential enhancement is performed in the low saturation area to improve color discrimination, and linear mapping is maintained in the high saturation area to avoid over-saturation. Subsequently, the Canny edge detection operator is used to extract the full-image structure features and form an edge map: through Gaussian filtering to smooth noise, non-maximum suppression, and double-threshold hysteresis processing, a binary edge map with complete edge connection is generated. The edge map is pixel-level weighted fused with the enhanced brightness component, and the fusion weight is dynamically adjusted according to the edge density to enhance texture details while retaining illumination consistency. After independent optimization of the H, S, and I components, the color space is reconstructed through the reverse HSI to RGB conversion formula to generate an initial enhanced image.

[0092] Furthermore, a Markov algorithm-based approach is introduced to improve the image quality. The initial enhanced image is input into a pre-trained convolutional autoencoder. The encoder network extracts multi-scale features and compresses them into the latent space to generate an initial latent variable as the initial state of the Markov chain. The state transition process is defined as a random walk in the latent space, the acceptance probability of the candidate latent variable is calculated, and whether to accept the candidate state is determined according to the Metropolis criterion (based on the energy function difference and probability threshold). After multiple iterative optimizations, the latent variable with the minimum energy function is selected and input into the U-Net decoder network. Through deconvolution layers, it is gradually upsampled and combined with skip connections to fuse the low-level features in the encoding stage to reconstruct a high-resolution image. This process enhances the detail sharpness while suppressing noise through probabilistic exploration in the latent space and deterministic decoding, and finally outputs multi-modal environmental monitoring data with balanced colors and clear edges, providing high-quality cross-modal inputs for subsequent target recognition and tracking.

[0093] It is worth mentioning that the energy function is expressed as:

[0094]

[0095] where, is to map the latent variable to the image space to generate the enhanced image, is the original image, is the regularization coefficient, is the image gradient.

[0096] Furthermore, in a preferred embodiment of the present invention, for the multi-modal feature extraction of the image-enhanced multi-modal environmental monitoring data, a cross-attention mechanism is introduced to fuse the extracted multi-modal features to construct cross-modal fusion features, which specifically includes:

[0097] Obtain the image-enhanced multi-modal environmental monitoring data, and input the image-enhanced multi-modal environmental monitoring data into a multi-channel feature extraction network for multi-modal feature extraction. The multi-channel feature extraction network includes a first feature extraction channel, a second feature extraction channel, and a third feature extraction channel;

[0098] Import the visible light image into the first feature extraction channel to extract multi-scale context features based on the hybrid dilated convolutional layer, capture context information under different receptive fields according to the parallel convolution branches, and then compress it into a multi-dimensional global vector through global average pooling to obtain the first feature extraction information;

[0099] Import the infrared image into the second feature extraction channel for spatial feature extraction, input the extracted spatial features into the SE attention module to perform global average pooling to generate a channel description vector, use the channel description vector to obtain a channel weight matrix and multiply it with the original feature map to extract a local thermal feature vector, and obtain the second feature extraction information;

[0100] Import the radar point cloud data into the third feature extraction channel to extract three-dimensional spatial features, select k key points as the initial point set through farthest point sampling, and capture the geometric contour and motion trajectory of the target object in combination with the local aggregation mechanism to generate the third feature extraction information;

[0101] Introduce a cross-attention mechanism to perform feature cross-modal association based on the first feature extraction information, the second feature extraction information, and the third feature extraction information. Use the visible light image feature as the query, and the infrared image feature and the radar point cloud feature as the key values, and calculate the spatial correlation weight through multi-head attention to generate the cross-modal fusion feature.

[0102] It should be noted that after obtaining the multi-modal environmental monitoring data after image enhancement, it is input into a multi-channel feature extraction network composed of a first feature extraction channel (visible light), a second feature extraction channel (infrared), and a third feature extraction channel (radar point cloud) to extract the deep semantic features of each modality respectively. After the visible light image is input into the first feature extraction channel, convolution kernels with different dilation rates (such as 1, 3, and 5) are fused through a hybrid dilated convolution layer to capture multi-scale context information while maintaining the resolution. Among them, the parallel convolution branches use 1×1, 3×3, and 5×5 convolution kernels to synchronously extract local detail and global structure features. Subsequently, the feature map is compressed into a multi-dimensional vector representing global semantics through global average pooling, generating the first feature extraction information containing target contour, texture, and illumination information. For the infrared image, first, the spatial convolution network (SCN) is used to extract the thermal radiation distribution features in the spatial dimension, and then it is input into the SE attention module: global average pooling is performed on the feature map to generate a channel description vector, the channel weight matrix is calculated through a fully connected layer and the Sigmoid activation function, and it is multiplied with the original feature map channel by channel to strengthen the thermal response of the target area and suppress background noise, finally extracting the second feature extraction information focusing on local high-temperature or low-temperature targets. For the millimeter-wave radar point cloud data, farthest point sampling is used to select k key points from the original point cloud as the initial point set, and the local neighborhood features of the point cloud are constructed based on the Set Abstraction layer in the local aggregation PointNet++: the coordinate normalization of the neighborhood point set (points within a radius r) of each key point is performed, and the coordinates, reflection intensity, and velocity information of the neighborhood points are aggregated through a multi-layer perceptron (MLP) to generate the third feature extraction information describing the three-dimensional geometric contour and motion trajectory of the target object. Finally, a cross-attention mechanism is introduced to achieve multi-modal feature fusion: using the visible light feature vector as the query, the infrared thermal feature vector and the radar point cloud feature vector as the key and value, the spatial correlation weights between the visible light feature and the features of the other modalities are calculated through multi-head self-attention, and the infrared and radar features are weighted and aggregated after Softmax normalization to generate a cross-modal joint feature integrating the target appearance (visible light), thermal radiation (infrared), and three-dimensional motion (radar). It realizes the adaptive enhancement of the contribution of key modalities in complex scenarios, such as increasing the weights of infrared and radar features in haze weather, ensuring the robustness and environmental adaptability of multi-modal fusion.

[0103] Furthermore, in a preferred embodiment of the present invention, the cascade target recognition model is constructed, and the cross-modal fusion feature is input to perform target recognition on the current monitoring scene to obtain scene target recognition information, which specifically includes:

[0104] Construct a cascade target recognition model, which is constructed with a Bi-LSTM and a multi-scale detection head as the framework, obtain the cross-modal fusion feature and input it into the cascade target recognition model to analyze the current scene;

[0105] Perform temporal context modeling using a Bi-LSTM network based on the input cross-modal fusion features. Based on two stacked LSTM cells, capture the spatio-temporal context features of the objects in the target scene through forward and backward passes respectively, and output spatio-temporal enhanced features;

[0106] Input the spatio-temporal enhanced features into a multi-scale detection head for object parsing. Execute multi-level feature abstraction through a lightweight GhostNet module to obtain multiple groups of low-rank feature maps, and use a cross-stage local connection module to extract local detail features;

[0107] Perform depthwise separable convolution compression on the obtained multiple groups of low-rank feature maps to obtain the channel dimension, perform channel concatenation with the local detail features to obtain concatenated fusion features, introduce spatial pyramid pooling and input the concatenated fusion features, extract cross-receptive field features through parallel multi-scale max pooling and splice them with the original feature map to obtain a three-level feature pyramid;

[0108] Based on the obtained three-level feature pyramid, perform scene object recognition on the target scene to generate a number of candidate recognition frames, and perform non-maximum suppression and mask threshold filtering on the candidate recognition frames to obtain scene object recognition information.

[0109] It should be noted that a cascaded object recognition model is constructed based on the design of a model architecture with a bidirectional long short-term memory network (Bi-LSTM) and a multi-scale detection head framework, and the cross-modal fusion features are input into the model for scene analysis. The Bi-LSTM network performs forward and backward propagation respectively through two stacked LSTM cells to perform temporal modeling on the input feature sequence: the forward LSTM captures the motion trajectory features of the target object from the initial state to the current frame, and the backward LSTM learns the context dependence in reverse. Through the gating mechanism (input gate, forget gate, output gate), the cell state is dynamically updated, fusing the historical and future information in the spatio-temporal dimension to generate a spatio-temporal enhanced feature vector. This feature vector encodes the joint representation of the dynamic behavior patterns (such as motion speed, direction change) and static attributes (such as shape, size) of the objects in the scene.

[0110] Subsequently, the spatio-temporal enhanced features are input into the multi-scale detection head for object parsing, and multiple groups of low-rank feature maps are generated through the lightweight GhostNet module. GhostNet adopts a linear transformation and channel splitting strategy to decompose the original feature map into backbone features and "ghost" features. The backbone features are extracted through conventional convolutions, and the ghost features are generated through inexpensive linear operations (such as per-channel convolutions). After splicing the two, a multi-level feature abstraction is formed, which retains the multi-scale characteristics of the object while reducing the computational cost. Then, the feature map is processed in stages through the Cross-Stage Partial Connection (CSP) module: the feature map is divided into two parts. One part extracts local detail features (such as edges and corners) through depthwise convolutions, and the other part is directly passed and fused with the result of the previous part to suppress gradient redundancy and enhance the detail expression ability. For the multiple groups of low-rank feature maps output by GhostNet, depthwise separable convolutions are used for channel dimension compression. By separating the feature learning of the spatial and channel dimensions through per-channel convolutions and pointwise convolutions, the number of parameters is reduced and the computational efficiency is improved. The compressed features are concatenated with the local detail features extracted by the CSP module to form a cascaded fusion feature. Subsequently, spatial pyramid pooling is introduced. The fusion feature is input into parallel multi-scale max pooling layers (such as 4×4, 8×8, 16×16 grids) to extract global context information under different receptive fields, and the multi-scale pooling results are concatenated with the original feature map along the channel dimension to construct a three-level feature pyramid containing fine-grained details, medium-scale structures, and macroscopic semantics. Finally, based on the three-level feature pyramid, an anchor box mechanism is used to generate candidate recognition boxes covering different scales and aspect ratios, and the class probability, bounding box offset, and mask confidence of each anchor box are predicted through convolutional layers. Non-maximum suppression is applied to the candidate boxes with high overlap rates, and they are sorted based on the intersection-over-union threshold and class scores to eliminate redundant detection results. At the same time, low-confidence predictions are removed by combining mask threshold filtering (confidence > 0.7), and finally, the position, class, and motion state information of the scene objects are output. This enhances the robustness of subsequent dynamic object tracking. The multi-scale detection head balances computational efficiency and recognition accuracy, improving the real-time object perception ability and accuracy of multi-modal data in complex environments.

[0111] Further, in a preferred embodiment of the present invention, the tracking of the scene recognition target in the current monitoring scene based on the cross-modal fusion feature and the scene object recognition information to generate scene object tracking information and store it in a preset trajectory memory pool specifically includes:

[0112] Obtain the cross-modal fusion feature and the scene object recognition information, extract the motion state feature of the scene recognition target based on the cross-modal fusion feature, and import the extracted motion state feature into an extended Kalman filter to construct a state vector;

[0113] Predict the position distribution of the scene recognition target in the next frame of image based on the constructed state vector to obtain the target prediction box, calculate the Mahalanobis distance between the target prediction box and the target detection box in combination with the scene target recognition information, and generate a motion cost matrix;

[0114] Obtain the appearance state features of the scene recognition target through the cross-modal fusion features, calculate the cosine similarity with the target appearance embedding in the adjacent frame of image, generate an appearance cost matrix and perform weighted fusion with the motion cost matrix, generate a comprehensive cost matrix as the input of the Hungarian algorithm to solve the optimal matching pair, and generate scene target tracking information according to the obtained optimal matching pair and store it in the preset trajectory memory pool.

[0115] It should be noted that first, the motion state features of the target (including dynamic parameters such as position, speed, and acceleration) are extracted based on the cross-modal fusion features and used as the state vector to input into the Extended Kalman Filter (EKF). The EKF predicts the position distribution of the target in the next frame through the non-linear state transition model, calculates the mean and covariance of the predicted state in combination with the process noise covariance matrix (characterizing the uncertainty of the motion model), and generates the probability distribution of the target prediction box. At the same time, the Mahalanobis distance between the prediction box and the detection box is calculated using the scene target recognition information of the current frame, and the Euclidean distance is normalized by the covariance matrix to quantify the matching degree between the two in the motion state space, and a motion cost matrix is constructed to characterize the motion consistency of the target in time series. To enhance the matching robustness, the appearance state features of the target (such as color histogram, texture descriptor, or depth embedding vector) are further extracted from the cross-modal fusion features, and the cosine similarity is calculated with the appearance embedding of the target in the adjacent frame to generate an appearance cost matrix reflecting the appearance consistency. The motion cost matrix (weight coefficient α) and the appearance cost matrix (weight coefficient β) are linearly weighted and fused to generate a comprehensive cost matrix (total cost = α × motion cost + β × appearance cost), where α and β are adaptively adjusted according to the scene dynamic characteristics (such as increasing α in high-speed motion scenes and increasing β in scenes with frequent occlusions). This comprehensive cost matrix is used as the input of the Hungarian algorithm, and the optimal detection box and trajectory association pair are obtained by constructing a bipartite graph and solving the minimum weight matching, excluding false matches and repeated tracking. Finally, the successfully matched trajectories are updated to the trajectory memory pool, recording the target ID, position history, and motion state. At the same time, new trajectories are initialized for the unmatched detection boxes, continuously outputting continuous tracking information of the scene target to solve problems such as occlusion, deformation, and cross-modal feature drift.

[0116] Further, in a preferred embodiment of the present invention, when a tracking target disconnection occurs during target tracking, obtain the mismatched tracking trajectory, and combine it with the trajectory memory pool to repair the tracking trajectory disconnection, specifically including:

[0117] Obtain the mismatch tracking trajectory, extract the last frame image with trajectory mismatch based on the mismatch tracking trajectory, define it as the disconnection repair start frame, extract the frame timing features and multi-modal features of the disconnection repair start frame to obtain the disconnection repair frame feature information;

[0118] Obtain the trajectory memory pool, extract the scene target tracking trajectory after the frame timing features of the disconnection repair frame based on the disconnection frame feature information, define it as the initial repair tracking trajectory, and extract the motion state attributes of the disconnection repair frame through the multi-modal features of the disconnection repair start frame;

[0119] Extract the motion state attributes of each initial repair tracking trajectory, calculate the deviation from the motion state attributes of the disconnection repair frame, and select the initial repair tracking trajectories with deviations within the preset range as candidate repair tracking trajectories;

[0120] Extract the multi-modal features of each candidate repair trajectory, calculate the similarity with the disconnection repair frame feature information to obtain several candidate repair tracking trajectories with a similarity greater than the preset similarity threshold, and select the candidate repair tracking trajectory with the largest similarity value as the final repair tracking trajectory to connect with the mismatch tracking trajectory.

[0121] It should be noted that after detecting the mismatch tracking trajectory, first locate the last frame before the disconnection of the trajectory (i.e., the disconnection repair start frame) based on the trajectory memory pool, extract the timing features (such as timestamp, motion speed history) and multi-modal features (including visible light appearance embedding, infrared thermal radiation distribution, radar point cloud geometric features) of this frame to form the disconnection repair frame feature information. Subsequently, retrieve all active tracking trajectories in the trajectory memory pool after the timestamp of the disconnection repair start frame as the initial repair tracking trajectories, and extract the motion state attributes (such as position, velocity vector, acceleration covariance matrix) of these trajectories at the disconnection moment. Extrapolate the predicted motion states of each initial repair tracking trajectory from the disconnection moment to the current moment through the Extended Kalman Filter (EKF), and calculate the deviation from the actual motion state of the disconnection repair frame: use the Mahalanobis distance to measure the statistical difference between the predicted state and the measured state, and filter out the trajectories with a Mahalanobis distance less than the preset threshold as candidate repair tracking trajectories. Further extract the multi-modal features of the candidate trajectories (such as visible light appearance embedding vector, infrared thermal radiation histogram, radar point cloud geometric descriptor), and calculate the cross-modal similarity with the disconnection repair frame features. Filter out the candidate trajectories with a similarity greater than the threshold, and select the one with the highest similarity as the final repair trajectory. Connect the mismatch trajectory and the repair trajectory through a trajectory interpolation algorithm (such as cubic spline interpolation), update the target ID, motion state and multi-modal features in the trajectory memory pool, and restore the tracking interruption caused by occlusion or detection failure to ensure the spatio-temporal continuity of the target motion trajectory.

[0122] Furthermore, the statistical difference between the predicted state and the measured state is measured using the Mahalanobis distance, and the specific calculation formula is as follows:

[0123]

[0124] where is the predicted state mean value, is the observation mean value of the disconnection frame, is the inverse matrix of the covariance matrix, is the residual vector between the predicted state and the observed state.

[0125] Figure 2 FIG. is a flowchart of a tracking disconnection repair method for target recognition and tracking provided by an embodiment of the present invention;

[0126] As Figure 2 shown, the present invention provides a flowchart of a tracking disconnection repair method for target recognition and tracking, including:

[0127] S202, if the similarity values between the multi-modal features of all candidate repair tracking trajectories and the feature information of the disconnection repair frame are all less than a preset similarity threshold, then obtain the spatial coordinate data of the scene tracking target in the disconnection repair frame as the origin to establish a polar coordinate system, and generate the predicted trajectories of the next k frames based on the motion state attributes of the disconnection repair frame, and calibrate the predicted trajectory coordinates in the established polar coordinate system;

[0128] S204, extract the spatial coordinate data of the scene tracking target in the starting frame of each candidate repair tracking trajectory according to the multi-modal features of the scene target associated with the starting frame of each candidate repair tracking trajectory, and calibrate it in the established polar coordinate system, which is defined as the point to be determined;

[0129] S206, segment the predicted trajectories in the polar coordinates according to the temporal sequence features and set the confidence levels of different predicted trajectory segments based on the order of the temporal sequence features, and calculate the Manhattan distances between each point to be determined and the predicted trajectories calibrated in the polar coordinate system at different time sequences;

[0130] S208, select the predicted trajectory segment with the smallest Manhattan distance as the target predicted trajectory segment corresponding to each point to be determined according to the calculated Manhattan distances, and perform weighted fusion in combination with the confidence level corresponding to the target predicted trajectory segment to obtain the association degree between each point to be determined and the predicted trajectory;

[0131] S210, judge the association degree between each point to be determined and the preset trajectory and the preset association threshold. If there is no point to be determined greater than the preset association threshold, then mark the target mismatched tracking trajectory as an irreparable trajectory;

[0132] S212, if there is a to-be-determined point greater than the preset association threshold, then select the to-be-determined point with the highest degree of association as the target determination point, define the candidate repair tracking trajectory corresponding to the target determination point as the target repair tracking trajectory, and connect it with the target mismatch tracking trajectory to complete the disconnection repair of the trajectory.

[0133] It should be noted that during the trajectory repair process, if the multi-modal feature similarity of all candidate repair tracking trajectories is lower than the preset threshold, then start the alternative repair strategy based on the motion state and spatial relationship. First, construct a polar coordinate system with the spatial coordinates of the target in the starting frame of disconnection repair (coordinates in the Cartesian coordinate system) as the origin, define the motion direction of the target (direction of the velocity vector) as the polar axis, and the radial distance r from the origin and the angle θ with the polar axis as coordinate parameters. Based on the motion state attributes (such as velocity v, acceleration a) of the disconnection repair frame, extrapolate the predicted trajectory of the next k frames through the Extended Kalman Filter (EKF) or the uniform motion model, and calibrate the coordinate sequence of the predicted points at time intervals in the polar coordinate system. At the same time, extract the spatial coordinates of the target in the starting frame of each candidate repair tracking trajectory, and convert them into to-be-determined points in the same polar coordinate system. The predicted trajectory is divided into several segments in chronological order (each 1 frame is a segment), and each segment is assigned a decreasing confidence weight (for example, the weight of the first segment is 0.8, the second segment is 0.7, and the weights of subsequent segments decrease gradually) to reflect the cumulative effect of prediction uncertainty. For each to-be-determined point, calculate its Manhattan distance from each segment of the predicted trajectory: convert the polar coordinates to a local Cartesian grid (the grid resolution matches the target size), calculate the sum of the Manhattan distances from it to all grid points in the predicted segment, and take the minimum value as the spatial proximity measure between this segment and the to-be-determined point. Select the segment of the predicted trajectory corresponding to the minimum Manhattan distance for each to-be-determined point, and calculate the degree of association in combination with the confidence weight of this segment. If all degrees of association are lower than the preset threshold, then determine that the mismatched trajectory cannot be repaired and mark it as the termination state; if there is a to-be-determined point greater than the preset association threshold, then select the to-be-determined point with the highest degree of association, connect its corresponding candidate repair trajectory with the mismatched trajectory through cubic spline interpolation, update the motion history and multi-modal features in the trajectory memory pool, and complete the disconnection repair. Ensure the continuity of the trajectory by giving priority to motion consistency, and improve the accuracy and continuity of target tracking in complex scenarios where the target appearance changes significantly after occlusion.

[0134] Furthermore, the specific formula for the degree of association is as follows:

[0135]

[0136] where is the confidence weight of the th segment of the predicted trajectory, is the minimum Manhattan distance between the point i to be determined and the j-th segment, is the preset maximum correlation distance.

[0137] Figure 3 A target recognition and tracking system 3 based on multi-source fusion imaging provided by an embodiment of the present invention, the system includes: a memory 31 and a processor 32. The memory 31 contains a program of a target recognition and tracking method based on multi-source fusion imaging. When the program of the target recognition and tracking method based on multi-source fusion imaging is executed by the processor 32, the following steps are implemented:

[0138] Perform environmental monitoring on the target scene to obtain multi-modal environmental monitoring data and perform preprocessing. Based on the preprocessed multi-modal environmental monitoring data, perform image enhancement to obtain image-enhanced multi-modal environmental monitoring data;

[0139] Perform multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, and introduce a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features;

[0140] Construct a cascaded target recognition model, input the cross-modal fusion features to perform target recognition on the current monitoring scene, and obtain scene target recognition information;

[0141] Based on the cross-modal fusion features and the scene target recognition information, track the scene recognition targets in the current monitoring scene to generate scene target tracking information and store it in a preset trajectory memory pool;

[0142] When a tracking target disconnection occurs during target tracking, obtain the mismatched tracking trajectory, and combine the trajectory memory pool to repair the tracking trajectory disconnection.

[0143] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical, or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in the embodiments of the present invention may be all integrated into a processing unit, or each unit may be separately taken as a unit alone, or two or more units may be integrated into one unit; the above-mentioned integrated unit may be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0146] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks and other various media that can store program codes.

[0147] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. And the foregoing storage medium includes: removable storage devices, ROM, RAM, magnetic disks, or optical disks and other various media that can store program codes.

[0148] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A target recognition and tracking method based on multi-source fusion imaging, characterized in that, Including: Performing environmental monitoring on the target scene to obtain multi-modal environmental monitoring data and preprocessing it, and performing image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data; Performing multi-modal feature extraction on the image-enhanced multi-modal environmental monitoring data, and introducing a cross-attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features; Constructing a cascaded target recognition model, inputting the cross-modal fusion features to perform target recognition on the current monitoring scene, and obtaining scene target recognition information; Based on the cross-modal fusion features and the scene target recognition information, tracking the scene recognition targets in the current monitoring scene to generate scene target tracking information and storing it in a preset trajectory memory pool; When a tracking target disconnection occurs during target tracking, obtaining a mismatched tracking trajectory and combining it with the trajectory memory pool to repair the tracking trajectory disconnection; Among them, the performing image enhancement based on the preprocessed multi-modal environmental monitoring data to obtain image-enhanced multi-modal environmental monitoring data specifically includes: Obtaining the preprocessed multi-modal environmental monitoring data, inputting it into the HSI model to convert the original RGB image to the HSI color space, and obtaining the hue component, saturation component, and brightness component through the HSI color space; Introducing the Retinex algorithm to decompose the brightness component into a reflection component and an illumination component, performing digital domain processing on the reflection component to enhance the dark area of the image, performing non-linear adaptive adjustment on the separated saturation component, and enhancing the color concentration through a dynamic range expansion function; Using the Canny edge detection operator to extract the full-image structure features and form an edge map, performing pixel-level weighted fusion of the edge map and the enhanced brightness component, after processing each component, re-fusing the optimized H, S, and I channels, and reconstructing the RGB image through inverse color space conversion to obtain the initial enhanced image; Introducing the Markov algorithm to reprocess the initial enhanced image, importing the initial enhanced image into a convolutional autoencoder network for encoding and mapping it to the latent space, and generating an initial latent variable as the starting state of the Markov chain; Defining a state distribution through the generated initial latent variable and calculating the state transition probability, generating candidate latent variables according to the state transition probability, and using the Metropolis criterion to judge whether to accept the target candidate latent variable; Performing iterative optimization until the stop criterion is met, then outputting the final latent variable, inputting the final latent variable into a pre-trained U-Net decoder network, and gradually reconstructing a high-resolution image through deconvolution layers and skip connections to obtain image-enhanced multi-modal environmental monitoring data.

2. The object recognition and tracking method based on multi-source fusion imaging according to claim 1, characterized in that, The performing environmental monitoring on the target scene to obtain multi-modal environmental monitoring data and preprocessing it specifically includes: Installing a heterogeneous sensor array in the target scene, and performing environmental monitoring on the target scene through the installed heterogeneous sensor array to obtain multi-modal environmental monitoring data, where the heterogeneous sensor array includes a high-resolution imaging camera, an infrared thermal imaging camera, and a millimeter-wave radar; Based on the acquired multi-modal environmental monitoring data, spatial registration is performed on the data obtained from different monitoring sources. The ORB feature extraction algorithm is introduced to extract scale pyramid features from the initial visible light image, and illumination difference elimination is carried out through gray value normalization. After completing the illumination difference elimination, the image region features of the initial visible light image for illumination difference elimination are extracted, and the initial infrared image is subjected to threshold segmentation processing in the corresponding region through the extracted image region features. Feature points are extracted from the initial visible light image and the initial infrared image respectively. The extracted feature points are preliminarily matched through the Hamming distance to generate a set of matching points. After completing the preliminary matching, the RANSAC algorithm is used to iteratively screen the inlier set. Each time, 4 pairs of feature points are randomly selected to calculate the affine transformation matrix, and the reprojection error of all matching segments with respect to this affine transformation matrix is calculated. Points with a reprojection error less than the preset value are included in the inlier set. After repeated iterations, the final affine transformation matrix and the corresponding inlier set are retained to obtain the spatial registration result of the initial visible light image and the initial infrared image. Radar point cloud data is extracted through multi-modal environmental monitoring data. The polar coordinates corresponding to the radar point cloud data are converted into three-dimensional Cartesian coordinates, and rigid body transformation is performed in combination with the external parameter calibration matrix of the sensor and projected onto the imaging plane of the visible light camera. Finally, a two-dimensional point cloud heat map spatially aligned with the optical image is generated. Combined with the spatial registration result of the initial visible light image and the initial infrared image and the two-dimensional point cloud heat map, the preprocessed multi-modal environmental monitoring data is obtained.

3. A target recognition and tracking method based on multi-source fusion imaging according to claim 1, characterized in that For the image-enhanced multi-modal environmental monitoring data, multi-modal feature extraction is performed. A cross-attention mechanism is introduced to fuse the extracted multi-modal features to construct cross-modal fusion features, which specifically includes: The image-enhanced multi-modal environmental monitoring data is acquired and input into a multi-channel feature extraction network for multi-modal feature extraction. The multi-channel feature extraction network includes a first feature extraction channel, a second feature extraction channel, and a third feature extraction channel. The visible light image is imported into the first feature extraction channel to extract multi-scale context features based on the hybrid dilated convolutional layer. The context information under different receptive fields is captured through the parallel convolution branches, and then compressed into a multi-dimensional global vector through global average pooling to obtain the first feature extraction information. The infrared image is imported into the second feature extraction channel for spatial feature extraction. The extracted spatial features are input into the SE attention module to perform global average pooling to generate a channel description vector. The channel weight matrix is obtained using the channel description vector and multiplied with the original feature map to extract the local thermal feature vector, obtaining the second feature extraction information. The radar point cloud data is imported into the third feature extraction channel to extract three-dimensional spatial features. k key points are selected as the initial point set through farthest point sampling, and the geometric contour and motion trajectory of the target object are captured in combination with the local aggregation mechanism to generate the third feature extraction information. Introduce the cross-attention mechanism to perform feature cross-modal association based on the first feature extraction information, the second feature extraction information, and the third feature extraction information. Use the visible light image feature as the query, and the infrared image feature and the radar point cloud feature as the key-value. Calculate the spatial correlation weight through multi-head attention to generate the cross-modal fusion feature.

4. An object recognition and tracking method based on multi-source fusion imaging according to claim 1, characterized in that, Construct a cascaded object recognition model, input the cross-modal fusion feature to perform object recognition on the current monitoring scene, and obtain the scene object recognition information, specifically including: Construct a cascaded object recognition model, which is constructed with Bi-LSTM and a multi-scale detection head as the framework. Obtain the cross-modal fusion feature and input it into the cascaded object recognition model to analyze the current scene; Use the Bi-LSTM network to perform temporal context modeling according to the input cross-modal fusion feature. Based on the stacked two-layer LSTM units, capture the spatio-temporal context features of the objects in the target scene by forward and backward propagation respectively, and output the spatio-temporal enhanced feature; Input the spatio-temporal enhanced feature into the multi-scale detection head for object parsing. Execute multi-level feature abstraction through the lightweight GhostNet module to obtain multiple groups of low-rank feature maps, and use the cross-stage local connection module to extract local detail features; Perform depthwise separable convolution compression on the obtained multiple groups of low-rank feature maps to obtain the channel dimension, perform channel concatenation with the local detail features to obtain the cascaded fusion feature. Introduce spatial pyramid pooling and input the cascaded fusion feature. Extract the cross-receptive field features through parallel multi-scale max pooling and splice them with the original feature map to obtain a three-level feature pyramid; Perform scene object recognition on the target scene based on the obtained three-level feature pyramid to generate several candidate recognition frames, and perform non-maximum suppression and mask threshold filtering on the candidate recognition frames to obtain the scene object recognition information.

5. A target recognition and tracking method based on multi-source fusion imaging according to claim 1, characterized in that, Based on the cross-modal fusion feature and the scene object recognition information, track the scene recognition target in the current monitoring scene to generate the scene object tracking information and store it in the preset trajectory memory pool, specifically including: Obtain the cross-modal fusion feature and the scene object recognition information. Extract the motion state feature of the scene recognition target based on the cross-modal fusion feature, and import the extracted running state feature into the extended Kalman filter to construct the state vector; Predict the position distribution of the scene recognition target in the next frame image based on the constructed state vector to obtain the target prediction box. Calculate the Mahalanobis distance between the target prediction box and the target detection box in combination with the scene object recognition information to generate the motion cost matrix; Obtain the appearance state feature of the scene recognition target through the cross-modal fusion feature, calculate the cosine similarity with the target appearance embedding in the adjacent frame image, generate the appearance cost matrix and perform weighted fusion with the motion cost matrix, generate the comprehensive cost matrix as the input of the Hungarian algorithm to solve the optimal matching pair, and generate the scene object tracking information according to the obtained optimal matching pair and store it in the preset trajectory memory pool.

6. A target recognition and tracking method based on multi-source fusion imaging according to claim 1, characterized in that When there is a disconnection situation of the tracking target during object tracking, obtain the mismatched tracking trajectory, and combine it with the trajectory memory pool to repair the disconnection of the tracking trajectory, specifically including: Obtain the mismatch tracking trajectory, extract the last frame image where trajectory mismatch occurs based on the mismatch tracking trajectory, define it as the disconnection repair start frame, extract the frame timing features and multi-modal features of the disconnection repair start frame to obtain the disconnection repair frame feature information; Obtain the trajectory memory pool, extract the scene target tracking trajectory after the frame timing features of the disconnection repair frame based on the disconnection repair frame feature information, define it as the initial repair tracking trajectory, and extract the motion state attributes of the disconnection repair frame through the multi-modal features of the disconnection repair start frame; Extract the motion state attributes of each initial repair tracking trajectory, calculate the deviation from the motion state attributes of the disconnection repair frame, and select the initial repair tracking trajectories with deviations within the preset range as candidate repair tracking trajectories; Extract the multi-modal features of each candidate repair trajectory, calculate the similarity with the disconnection repair frame feature information to obtain several candidate repair tracking trajectories with a similarity greater than the preset similarity threshold, and select the candidate repair tracking trajectory with the largest similarity value as the final repair tracking trajectory to connect with the mismatch tracking trajectory.

7. A target recognition and tracking method based on multi-source fusion imaging according to claim 6, characterized in that, When there is a disconnection situation of the tracking target during target tracking, obtain the mismatch tracking trajectory and combine the trajectory memory pool to repair the disconnection of the tracking trajectory, and further include: If the similarity values of the multi-modal features of all candidate repair tracking trajectories and the disconnection repair frame feature information are less than the preset similarity threshold, obtain the spatial coordinate data of the scene tracking target in the disconnection repair frame as the origin to establish a polar coordinate system, and generate the predicted trajectories for the next k frames based on the motion state attributes of the disconnection repair frame, and calibrate the predicted trajectory coordinates in the established polar coordinate system; Extract the spatial coordinate data of the scene tracking target in the start frame of each candidate repair tracking trajectory according to the multi-modal features of the scene target associated with the start frame of each candidate repair tracking trajectory, and calibrate it in the established polar coordinate system, defined as the point to be determined; Segment the predicted trajectory in the polar coordinates according to the timing features and set the confidence levels of different predicted trajectory segments based on the sequence of the timing features, and calculate the Manhattan distance between each point to be determined and the predicted trajectory calibrated in the polar coordinate system at different time sequences; Select the predicted trajectory segment with the smallest Manhattan distance as the target predicted trajectory segment corresponding to the point to be determined according to the calculated Manhattan distance, and perform weighted fusion in combination with the confidence level corresponding to the target predicted trajectory segment to obtain the association degree between each point to be determined and the predicted trajectory; Judge the association degree between each point to be determined and the preset trajectory and the preset association threshold. If there is no point to be determined greater than the preset association threshold, mark the target mismatch tracking trajectory as an irreparable trajectory; If there is a point to be determined greater than the preset association threshold, select the point to be determined with the largest association degree as the target determination point, define the candidate repair tracking trajectory corresponding to the target determination point as the target repair tracking trajectory, and connect it with the target mismatch tracking trajectory to complete the disconnection repair of the trajectory.

8. An object recognition and tracking system based on multi-source fusion imaging, characterized in that, The system includes: a memory and a processor. The memory contains a program for the target recognition and tracking method based on multi-source fusion imaging. When the program for the target recognition and tracking method based on multi-source fusion imaging is executed by the processor, the steps of the target recognition and tracking method based on multi-source fusion imaging as described in any one of claims 1-7 are implemented.

Images