An adaptive parameter filter for multi-modal data fusion for pedestrian detection

Abstract

The application provides an adaptive parameter filter for multi-modal data fusion of pedestrian detection, which is used for performing the following steps: S1: acquiring multi-modal data and carrying out denoising pretreatment; S2: performing semantic association on the pretreated multi-modal data to obtain a double-branch three-dimensional pedestrian detection frame set; S3: acquiring laser radar branch observation and depth camera branch observation; S4: processing the laser radar branch observation and the depth camera branch observation to obtain multi-dimensional data; S5: constructing and extracting a multi-dimensional data input time sequence attention long short-term memory network in a time sequence window, and outputting an adaptation probability and a noise adjustment factor of each motion model; S6: based on the noise adjustment factor and a noise matrix, obtaining each motion model estimation through state prediction and update, and then performing weighted fusion according to the adaptation probability to obtain an optimal state estimation and a tracking trajectory of the pedestrian. The application can improve the target detection and tracking precision of a mobile robot in a complex environment.

Description

Technical Field

[0001] This specification relates to the field of intelligent sensing and multimodal sensor fusion technology, and in particular to an adaptive parameter filter for multimodal data fusion for pedestrian detection. Background Technology

[0002] With the continuous expansion of applications such as autonomous driving, intelligent security, and service robots, pedestrian detection and tracking technology, as a core component of environmental perception systems, has received widespread attention. Traditional single-sensor solutions (such as relying solely on LiDAR or visual cameras) suffer from significant blind spots and insufficient robustness. For example, while LiDAR can provide high-precision 3D spatial information, it lacks semantic recognition capabilities and struggles to effectively distinguish "pedestrians" from other static or dynamic objects. RGB cameras, while possessing strong semantic understanding capabilities, are susceptible to changes in lighting and occlusion, leading to decreased positioning accuracy, especially at long distances or in textureless areas.

[0003] To overcome the limitations of single sensors, multimodal fusion sensing technology has gradually become a research hotspot in recent years. By fusing high-precision spatial information from lidar, near-range depth data from depth cameras, and semantic information from visual images, not only can the accuracy of target detection be improved, but the robustness of the system in complex environments such as occlusion and changes in lighting can also be enhanced. However, most existing multimodal fusion methods remain at the data-level or feature-level fusion level and still face many challenges.

[0004] Existing methods face three main challenges: First, the filtering parameters lack adaptability. The core parameters of traditional tracking algorithms rely on manual presets and remain fixed, making it difficult to keep up with real-time changes in pedestrian movement patterns and environmental noise in dynamic scenes, leading to decreased tracking accuracy and trajectory jitter. Second, they lack robustness in complex scenes. Most methods employ simple weighted or serial update strategies, allocating the observation contributions of each branch with fixed weights. If a sensor temporarily fails due to occlusion or data loss, invalid data is still forcibly included in the fusion, and the failure of preceding branches directly undermines the basis for subsequent updates, easily causing tracking interruptions. Finally, they fail to fully exploit the guiding role of temporal features in parameter adjustment, mostly relying on single-frame observation data for updates, ignoring the temporal continuity of motion states. When faced with sudden movements, parameter adjustment exhibits a significant lag, further exacerbating trajectory disconnection and increasing the interruption rate. Summary of the Invention

[0005] This specification provides an adaptive parameter filter for multimodal data fusion in pedestrian detection, in order to solve the problem of insufficient target detection accuracy of mobile robots in complex environments in the prior art.

[0006] The following technical solution is adopted in this specification: An adaptive parameter filter for multimodal data fusion in pedestrian detection, used to perform the following steps: S1: Acquire multimodal data of the region to be detected, and perform noise reduction data preprocessing on the multimodal data respectively; S2: Perform semantic association on the denoised preprocessed multimodal data to obtain a set of dual-branch 3D pedestrian detection boxes; S3: Perform time synchronization and spatial coordinate transformation on the dual-branch pedestrian 3D detection box to obtain the dual-branch observation sequence in the bird's-eye view coordinate system. The dual-branch observation sequence includes lidar branch observation and depth camera branch observation. S4: Use Kalman filtering to sequentially fuse the lidar branch observations and depth camera branch observations to obtain an intermediate state estimation sequence, and calculate the state estimation difference sequence between adjacent frames; S5: Construct a temporal feature window, extract the observation residuals of the lidar branch, the observation residuals of the depth camera branch, the difference sequence, and the corresponding innovation sequences calculated based on each pre-set motion model within the temporal window, concatenate them sequentially, and input them into a pre-trained temporal attention long short-term memory network. Output the adaptation probability and noise adjustment factor of each motion model, and dynamically update the noise matrix according to the noise adjustment factor. S6: Based on the updated noise matrix, perform unscented Kalman filtering on each motion model, obtain the motion model estimate through state prediction and update, and then fuse them according to the corresponding adaptation probability to output the optimal state estimate and tracking trajectory of the pedestrian.

[0007] Based on the above technical means, by constructing a temporal feature window, the observation residual, state difference, and multi-motion model information sequence are extracted. Then, a temporal attention long short-term memory network (LSTM) is used to model and identify the pedestrian's motion pattern. The adaptation probability and noise adjustment factor of each motion model are output in real time. Based on the noise adjustment factor, the process noise covariance matrix and observation noise covariance matrix of the Kalman filter are dynamically updated, so that the filter can automatically adjust the parameters according to the fluctuation of environmental noise and the change of pedestrian motion state. This avoids the problem of decreased tracking accuracy and trajectory jitter caused by parameter mismatch, and improves the filter's adaptability to dynamic scenes.

[0008] This invention employs a dual-branch parallel detection architecture. When a branch temporarily fails due to occlusion, changes in lighting, or data loss, LSTM can dynamically adjust the noise adjustment factor and the fusion weights of each model. This allows the system to automatically reduce the contribution of the failed branch, focus on the information of the effective branch, and increase the weights of the model that adapts to the current motion mode. This avoids the problem of a sharp increase in fusion result deviation and trajectory interruption caused by the failure of a single branch, and achieves stable tracking in complex environments.

[0009] Furthermore, the acquisition of multimodal data in S1 specifically involves using a lidar, a depth camera, and an RGB camera to acquire 3D point cloud data, RGB image data, and depth image data of the area to be detected, respectively.

[0010] Furthermore, the noise reduction preprocessing in S1 includes: The three-dimensional point cloud data is denoised and downsampled using voxel mesh to obtain the first point cloud data; The RGB image data is subjected to distortion correction and scale normalization. The depth image data is subjected to hole filling and noise smoothing processing, and then converted into three-dimensional point cloud data, which is the second point cloud data.

[0011] Based on the above technical means, by performing corresponding denoising preprocessing on 3D point cloud data, RGB image data and depth image data respectively, it is possible to significantly reduce the amount of point cloud data and reduce the computational load of subsequent plane segmentation and clustering tasks, while preserving the overall spatial structure features of the point cloud to the maximum extent.

[0012] Furthermore, step S2 specifically includes the following steps: S21: Convert the first point cloud data and the second point cloud data to the RGB image coordinate system; S22: Perform planar clustering and segmentation on the transformed first point cloud data and second point cloud data respectively to generate multiple lidar-side 3D candidate boxes and multiple depth camera-side 3D candidate boxes; A pre-trained detection network is used to output two-dimensional detection boxes based on two-dimensional image data; S23: Obtain the correlation between the 3D candidate boxes on the LiDAR side and the 3D candidate boxes on the depth camera side and the 2D detection boxes respectively, and filter out multiple 3D candidate boxes on the LiDAR side and multiple 3D candidate boxes on the depth camera side that meet the preset conditions, and set them into a dual-branch 3D pedestrian detection box set.

[0013] Based on the above technical means, on the one hand, the candidate boxes on the lidar side and the depth camera side are independently associated with the 2D detection boxes. Even if the point cloud quality of one branch is poor (such as sparse point cloud of lidar at a long distance and hole of depth camera at a close distance), the other branch can still provide effective pedestrian detection boxes through its own association results, thus ensuring the integrity and reliability of the dual-branch 3D pedestrian detection box set. Furthermore, S3 specifically includes the following steps: S31: Set up data cache queues for LiDAR, depth camera and RGB camera respectively to store the raw data and corresponding timestamps in the most recent time period; S32: Using the timestamp of the two-dimensional image data as a reference, linear interpolation is used to time-align the lidar point cloud and the depth point cloud to generate a multimodal data sequence with unified timestamps; S33: The center coordinates of the dual-branch 3D pedestrian detection box are transformed to the chassis coordinate system through sensor extrinsic parameters, ignoring height information, to obtain the lidar branch observation and depth camera branch observation in the bird's-eye view coordinate system.

[0014] Based on the above technical means, the pedestrian tracking problem in three-dimensional space is simplified into a tracking problem in two-dimensional plane (BEV) by dimensionality reduction processing, which reduces the dimension of the state vector while still preserving the core position and velocity information of the pedestrian's movement on the ground.

[0015] Furthermore, the noise adjustment factor in S5 includes the process noise adjustment factor, the lidar observation noise adjustment factor, and the depth camera observation noise adjustment factor.

[0016] Furthermore, the dynamic update of the noise matrix in S5 specifically involves: multiplying the preset basic process noise covariance matrix by the process noise adjustment factor to obtain the updated process noise covariance matrix, and multiplying the preset basic observation noise covariance matrix by the corresponding observation noise adjustment factor to obtain the updated observation noise covariance matrix.

[0017] Furthermore, S6 specifically includes the following steps: S61: Generate multiple Sigma points based on the current state estimate and covariance matrix, and substitute each Sigma point into the state transition function of the motion model to obtain the predicted Sigma points. Then, calculate the predicted state mean and predicted covariance based on the predicted Sigma points with weights. S62: Project the predicted Sigma points onto the observation space to obtain the observation prediction values, and calculate the observation covariance matrix and the state-observation cross-covariance matrix accordingly. S63: Solve for the Kalman gain based on the observation covariance matrix and the state-observation cross-covariance matrix, and update the predicted state mean and predicted covariance using the current observation values ​​to obtain the state estimate and covariance of the motion model. S64: Based on the adaptation probability corresponding to each motion model, the state estimates of all motion models are weighted and fused to output the optimal state estimate and tracking trajectory of the pedestrian.

[0018] Based on the aforementioned techniques, the state estimates of all motion models are weighted and fused according to the adaptation probabilities corresponding to each motion model. When the pedestrian's motion pattern changes (e.g., switching from uniform straight-line motion to turning), the adaptation probabilities output by the LSTM are adjusted accordingly. The weighted fusion result of the Unscented Kalman Filter (UKF) naturally tilts towards the higher probability model, eliminating the need for explicit mode switching logic. This achieves smooth and seamless adaptive tracking, avoids trajectory jitter caused by hard switching, and increases the accuracy of target detection and tracking in complex environments.

[0019] Furthermore, the intermediate state estimates in S4 are obtained as follows: S41: Using the intermediate state estimate from the previous time step, the prior state estimate for the current time step is obtained through the prediction step of Kalman filtering; then, the Kalman gain is calculated using the observation of the lidar branch, and the posterior state estimate is updated to obtain a new one. S42: Based on the aforementioned posterior state estimate, the Kalman gain is recalculated using depth camera branch observations, and the intermediate state estimate for the current time moment is updated again.

[0020] Furthermore, the motion models pre-set in S5 specifically include a uniform linear model, a uniform acceleration model, and a non-linear turning model.

[0021] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: 1. By constructing a temporal feature window, observation residuals, state differences, and multi-motion model information sequences are extracted. A temporal attention long short-term memory network (LSTM) is used to model and identify pedestrian motion patterns. The adaptation probability and noise adjustment factor of each motion model are output in real time. Based on the noise adjustment factor, the process noise covariance matrix and observation noise covariance matrix of the Kalman filter are dynamically updated, enabling the filter to automatically adjust parameters according to environmental noise fluctuations and changes in pedestrian motion state. This avoids the problem of decreased tracking accuracy and trajectory jitter caused by parameter mismatch, and improves the filter's adaptability to dynamic scenes.

[0022] 2. This invention adopts a dual-branch parallel detection architecture. When a branch temporarily fails due to occlusion, changes in lighting, or data loss, LSTM can dynamically adjust the noise adjustment factor and the fusion weights of each model. This allows the system to automatically reduce the contribution of the failed branch, focus on the information of the effective branch, and increase the weights of the model that adapts to the current motion mode. This avoids the problem of a sharp increase in the deviation of the fusion result and trajectory interruption caused by the failure of a single branch, and achieves stable tracking in complex environments. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall process of this embodiment; Figure 2 This is a schematic diagram of the specific process of step S2 in this embodiment; Figure 3 This is a schematic diagram of the specific process of step S3 in this embodiment; Figure 4 This is a schematic diagram of the specific process of step S4 in this embodiment; Figure 5 This is a schematic diagram of the specific process of step S6 in this embodiment; Figure 6 This is a schematic diagram of the LSTM network structure in this embodiment; Figure 7 This is a schematic diagram of the relevant verification results in this embodiment; Figure 8 This is the prediction result of the noise scaling factor on the test set in this embodiment; Figure 9 This is a schematic diagram of the changes in observation data from the dual branches in this embodiment; Figure 10 This is a schematic diagram of the dynamic adjustment of the LSTM network in this embodiment; Figure 11 This is a schematic diagram illustrating the accuracy of experimental results using Time Attention LSTM-KF-UKF versus other frameworks in this embodiment. Figure 12 This diagram illustrates the fusion accuracy of experimental results using Time Attention LSTM-KF-UKF and other frameworks in this embodiment.

[0024] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The same or similar reference numerals correspond to the same or similar components. The terms describing positional relationships in the drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0026] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0028] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0029] In the embodiments of this application, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium.

[0030] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0031] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings. Example 1

[0032] like Figure 1 As shown, this embodiment provides an adaptive parameter filter for multimodal data fusion in pedestrian detection, including the following steps: Step S1: Acquire multimodal data of the region to be detected, and perform noise reduction data preprocessing on the multimodal data respectively; Specifically, the denoising data preprocessing for multimodal data involves: preprocessing the original depth map output by the depth camera; mapping the pixel coordinates of the depth map to the binocular camera coordinate system based on the depth camera's intrinsic parameters; and generating a depth point cloud in the binocular coordinate system. ; then Statistical filtering (SS) is performed for noise reduction, and voxel grid downsampling is used to reduce the data volume while retaining the effective target point cloud (second point cloud data); this is applied to the point cloud information acquired by the rotating mechanical lidar. Preprocessing is performed, and voxel lattice filtering is used for data optimization. For each voxel unit, the mean coordinate of all points within the unit is calculated, and the center point corresponding to this mean is used to replace all original points within the voxel (first point cloud data). This can significantly reduce the amount of point cloud data and reduce the computational load of subsequent planar segmentation and clustering tasks, while preserving the overall spatial structure features of the point cloud to the maximum extent. like Figure 1 As shown, in this embodiment, step S2: semantic association is performed on the multimodal data that has undergone denoising preprocessing to obtain a set of dual-branch 3D pedestrian detection boxes.

[0033] like Figure 2 As shown, in this embodiment, S21: convert the first point cloud data and the second point cloud data to the RGB image coordinate system; Establish a general planar model ,in It is the unit normal vector of the plane. , Let these be the coordinates of a point on the plane. This is the distance from the plane to the origin.

[0034] Set minimum number of plane points Set the threshold for the distance from a point to a plane. Set the maximum number of planes , which represents the maximum number of planes fitted to the indoor scene; Randomly select 3 non-collinear points from either the first point cloud data or the second point cloud data. Calculate the plane normal vector ; Substitution calculate To obtain a planar model ; S22: Perform planar clustering and segmentation on the transformed first point cloud data and second point cloud data respectively to generate multiple lidar-side 3D candidate boxes and multiple depth camera-side 3D candidate boxes; Calculate all points in the first point cloud data or the second point cloud data Distance to the current plane Statistical satisfaction Points in the plane, repeated iteratively Next, the plane with the most points in the plane is selected as the optimal plane for this round; Determine if satisfied For the effective plane, record the parameters. And remove these internal points from either the first point cloud data or the second point cloud data; Repeat the above steps to perform parallel extraction on either the first point cloud data or the second point cloud data until the number of extracted planes reaches a certain threshold. The number of interior points of the largest plane in either the first point cloud data or the second point cloud data. ; Traverse all non-planar points ,calculate ,in for Euclidean distance between two points The first point cloud data set or the second point cloud data set; In label P, satisfying point As the core point, among which The minimum number of points is set. Randomly select an unclassified core point , and Unclassified points within the neighborhood are added to the cluster until no new points are added, forming a complete cluster. Repeat this process until all core points have been categorized, then remove the remaining points. Clusters; Generate cluster set Each cluster Corresponding to a potential target; For each cluster Generate the minimum axis-aligned bounding box and calculate the centroid position of the cluster. As the center of the bounding box,

[0035] Size filtering is performed on the smallest axis-aligned bounding box that does not conform to pedestrian characteristics to obtain the bounding box size. ; Calculate the confidence score for the candidate boxes. ,in It is the number of points within the cluster. For point cloud density, The preset maximum point cloud density; Generate a set of 3D candidate bounding boxes from either the LiDAR side or the depth camera side, specifically: , ; This includes the center coordinates. ,size Confidence score; like Figure 2 As shown, in this embodiment, S22: A pre-trained detection network is used to output a two-dimensional detection box based on RGB image data; In this preferred embodiment, the Zhang Zhengyou calibration method based on a checkerboard target is used to obtain the participation distortion coefficients of the RGB camera. By acquiring checkerboard images in multiple poses and detecting the inner corner points of the images, the intrinsic parameters K and distortion coefficients of the RGB camera are obtained. ,in , For focal length, Principal coordinates Radial distortion coefficient, The tangential distortion coefficient; Based on the aforementioned intrinsic parameter K and distortion coefficient D, correction is performed on the RGB image data acquired by the RGB camera to obtain a preprocessed RGB image. The specific correction formula is as follows:

[0036] in The coordinates of a point in the RGB image data. These are the coordinates of a point after preprocessing. The open-source model YOLOv11 was selected as the pre-trained detection network. The corrected RGB image was input into the pre-trained detection network, which output 2D detection boxes. Category confidence Target center point ; To eliminate duplicate detection boxes of targets in 2D detection results, a non-maximum suppression algorithm is used to optimize the detection results. The cross-union ratio (CUI) between detection boxes is calculated using the following formula: ,reserve For multiple detection boxes of the same target, the detection box with the highest confidence is retained, and the set of 2D detection boxes is obtained by integrating and filtering. Each element corresponds to the coordinates of the Kth detection box, the category label, and the confidence score. S23: Obtain the correlation between the 3D candidate boxes on the LiDAR side and the 3D candidate boxes on the depth camera side and the 2D detection boxes respectively, and filter out multiple 3D candidate boxes on the LiDAR side and multiple 3D candidate boxes on the depth camera side that meet the conditions, and set them into a set of dual-branch 3D pedestrian detection boxes.

[0037] In this embodiment, the non-planar point cloud obtained from the lidar side is extracted as follows:

[0038] The RGB camera coordinate system is calculated as follows: , in The extrinsic matrix for laser radar to RGB camera; Convert the original depth map into a 3D point cloud in the binocular camera coordinate system. Calculate the coordinates of the RGB camera. ,in The extrinsic matrix from a binocular camera to an RGB camera ; For points projected onto the coordinate system of the RGB camera by the dual branches ,calculate count satisfy ,in Focal length Principal point coordinates; The output 2D semantic detection box Based on this, spatial matching is performed on the dual-branch projection points, and the projected point cloud is selected within the RGB camera's view. Preserve point cloud information that falls within the pedestrian detection frame. ; The correlation degree on the lidar side is calculated as follows: , in, Indicates the correlation degree on the lidar side. Let k be the number of point clouds in the k-th 2D detection box. The number of point clouds belonging to the k-th 3D bounding box; like Assign 3D candidate boxes to the LiDAR side Semantic tags and indicated with Association successful; Using the same association method, if Assign 3D candidate bounding boxes to the depth point cloud. Semantic tags and indicated with Association successful; 3D bounding boxes with "pedestrian" semantic labels from both the LiDAR and depth camera sides were selected to obtain the 3D objects detected by both branches, as follows: , in, This represents the set of 3D pedestrian detection boxes for two branches. This represents the 3D bounding box of the pedestrian from the LiDAR side. Represents a 3D bounding box of a pedestrian from the depth camera side, under ideal conditions. .

[0039] The position of the dual-branch 3D detection box is projected from the sensor coordinate system to the center of the robot chassis, as follows:

[0040] in The position of the radar-monitored target in the ground coordinate system, The position of the target in the ground coordinate system detected by the depth camera; In the chassis coordinate system, ignoring the height Z, we obtain the bird's-eye view (BEV) projection of the 3D frame of the LiDAR side. That is, the BEV projection of the LiDAR branch BEV detection frame and the depth camera side 3D frame. That is, the depth camera branch BEV detection frame.

[0041] By leveraging the high semantic recognition capability of RGB images, pedestrian labels are assigned to LiDAR point clouds and depth point clouds that lack semantic information, thereby effectively eliminating non-pedestrian targets (such as vehicles, roadblocks, trees, etc.) and significantly reducing the false detection rate.

[0042] like Figure 1 As shown, in this embodiment, S3: the time synchronization and spatial coordinate transformation of the dual-branch pedestrian 3D detection box are performed to obtain the dual-branch observation sequence in the bird's-eye view coordinate system. The dual-branch observation sequence includes lidar branch observation and depth camera branch observation. Specifically, such as Figure 3 As shown, in this embodiment, S31: Set up data cache queues for the LiDAR, depth camera and RGB camera respectively to store the raw data and corresponding timestamps within a recent time period; S32: Using the timestamp of RGB image data as a reference, linear interpolation is used to time-align the lidar point cloud and depth point cloud to generate a multimodal data sequence with unified timestamps; In this embodiment, S33: The center coordinates of the dual-branch 3D pedestrian detection box are transformed to the chassis coordinate system through sensor extrinsic parameters, ignoring height information, to obtain the lidar branch observation and depth camera branch observation in the bird's-eye view coordinate system.

[0043] In this embodiment, the specific implementation process of S3 is as follows: The BEV detection timestamps obtained from dual-branch detection are synchronized to the same reference clock source. An independent buffer queue is set up for each sensor to store data within the most recent T seconds. Each frame of data carries a calibrated timestamp. , .

[0044] Filter for the closest match in the cache queue. The frames, as the observation data for data fusion at this time, include This is the current sensor fusion moment, after time synchronization, the length-aligned lidar BEV observation sequence ( ) and depth camera BEV observation sequence ( ); like Figure 1 As shown, in this embodiment, S4: Kalman filtering is used to sequentially fuse the lidar branch observations and the depth camera branch observations to obtain an intermediate state estimation sequence.

[0045] Using a basic Kalman filter for multi-sensor information fusion, based on the BEV state estimate from the previous time step (hereinafter referred to as the intermediate estimate), the intermediate estimate of the current state is predicted as follows:

[0046] in, This represents the BEV state estimate at the previous moment. Represents the state transition matrix. This represents an intermediate estimate of the current state. To calculate the dual-observation update, we first use lidar observations for the update. The lidar Kalman gain is calculated as follows:

[0047] in, Indicates the Kalman gain of the lidar. This represents the prior covariance of the lidar in its current state. Represents the observation matrix. LiDAR observation noise; in, , This represents the observation noise of the basic lidar. This represents the lidar observation noise adjustment factor; The intermediate state of the lidar observations is updated as follows: ; in, This indicates the intermediate state of the lidar observation at the current moment.

[0048] The results are then updated a second time using depth camera observations, and the Kalman gain is calculated as follows: , in Indicates the Kalman gain of the depth camera. This indicates the noise level observed by the depth camera; in, , This represents the noise level observed by the base depth camera. This represents the depth camera observation noise adjustment factor; The intermediate state of the BEV after dual-branch fusion is updated as follows: ; like Figure 1 As shown, in this embodiment, a temporal feature window is constructed, and the observation residuals of the lidar branch, the observation residuals of the depth camera branch, the intermediate state estimation sequence, and the corresponding innovation sequences calculated based on each pre-set motion model within the temporal window are sequentially concatenated and input into a pre-trained temporal attention long short-term memory network. The adaptation probability and noise adjustment factor of each motion model are output, and the noise matrix is ​​dynamically updated according to the noise adjustment factor. In this embodiment, the specific implementation process of S5 is as follows: Build length is The temporal feature window of a frame includes: F1-LiDAR observation and filtering estimation bias: ; F2-Depth camera observation and filtering estimation bias: ; F3 - Estimated state evolution inter-frame difference sequence: ; F4 - The news sequence of three motion models: , in For the first The model's prediction bias; The above sequences are concatenated in order to form a feature vector. Input to an LSTM network (2 layers, 64 hidden units), learn long-term dependencies between features through a gating mechanism, and output L hidden states. The LSTM network structure is as follows: Figure 6 As shown; Hide the state of each frame Input to a fully connected layer, map to obtain the corresponding score. Attention weights for each hidden state are calculated using the softmax function. The formula is: , in, A larger weight value indicates that the features of the corresponding frame contribute more to the result; in, , which represents the attention score.

[0049] The hidden states are weighted and summed based on attention weights to obtain the aggregate feature H, as shown in the formula: ; The aggregated feature H is input into the output layer, and the output consists of two types of core parameters: Output model probability The adaptation probabilities of the three pedestrian motion models satisfy the following conditions. ; Output noise adjustment factor ,in This is the process noise amplification factor. This represents the noise amplification factor of the lidar observation. This represents the noise amplification factor for depth camera observations. Based on noise adjustment factor Dynamically adjust the preset base noise parameters; process noise adjustment: Observation noise ; In this preferred embodiment, considering the diversity of pedestrian movements, three differentiated motion models are defined to cover typical motion scenarios, specifically including: Model 1 (Uniform Linear Motion Model): Applicable to scenarios where pedestrians move at a uniform speed along a straight line. The state space vector is expressed as follows: The state transition matrix is: ; Model 2 (Uniform Acceleration Model): Applicable to scenarios involving pedestrian acceleration or deceleration. The state equation incorporates acceleration parameters, and the state space vector is expressed as follows: The state transition matrix is: ; Model 3 (Nonlinear Turning Model): Applicable to pedestrian curve turning scenarios, it introduces a steering angular velocity parameter, uses state equations to characterize the dynamic changes in the heading angle, and expresses the state space vector as follows: The state transition function is: .

[0050] like Figure 1 As shown, in this embodiment, S6: Based on the updated noise matrix, perform unscented Kalman filtering on each motion model, obtain the motion model estimate through state prediction and update, and then fuse them according to the corresponding adaptation probability to output the optimal state estimate and tracking trajectory of the pedestrian.

[0051] like Figure 5 As shown, in this embodiment, S61: Generate multiple Sigma points based on the current state estimate and covariance matrix, and substitute each Sigma point into the state transition function of the motion model to obtain the predicted Sigma points, and calculate the predicted state mean and predicted covariance based on the predicted Sigma points. The three motion models described above all use the pedestrian's position and velocity in the BEV view as the core state variables. Based on the differences in motion characteristics, state transition logic is designed to form their own unique state equations. ( (These correspond to three different types of models).

[0052] Perform UT transformation for each model Generate 2n+1 sigma points: , in , Indicates the sigma point dispersion. Indicates the Sigma point scaling parameter; Substituting each sigma point into the state equation of model j yields the predicted sigma points: ; Calculate the mean of the predicted state: ,in, Indicates the weight of the Sigma point; Calculate the predicted covariance: , in , , , Indicates the weight of the Sigma point; Calculate the observation prediction: Substitute the predicted sigma point into the observation equation to obtain the observation prediction value of model j: , in For the observation equation, To observe noise; Calculate the observed mean: ; Calculate the observed covariance: ; Calculate the state-observation cross-covariance: ; Calculate the Kalman gain: ; Status Update: ,remember ; Covariance update: ; According to model probability The update results from the three models are combined: , Fusion covariance estimation: ; in For optimal state estimation, take the first two columns of states. As pedestrian tracking results under the BEV view For covariance estimation; In other embodiments, the method of acquiring multimodal data can be achieved by programmatically constructing a simulation system to generate training data in batches that conforms to the characteristics of real-world scenarios. The core objective is to provide standardized samples for the Time Attention LSTM model, thereby reducing the cost of collecting data from real-world environments, as detailed below: Based on the motion characteristics of pedestrians, and according to the parameter ranges of the three types of motion models specified in Patent Example 1, a continuous trajectory with model switching is generated: Uniform linear model ( ): Suitable for pedestrians walking in a straight line, with a linear speed range of 0.5~1.5m / s, a movement direction of 0~360°, and no acceleration or turning angular velocity.

[0053] Uniform acceleration model ( ): Adapted for pedestrian acceleration / deceleration scenarios, initial velocity 0.3~1.0m / s, acceleration .

[0054] Nonlinear turning model ( ): Adapted for pedestrian curvilinear motion scenarios, linear velocity 0.5~1.5m / s, turning angular velocity .

[0055] Each model segment lasts for 50 to 200 frames, and motion continuity is ensured through state mapping during switching. Setting the basic observation noise of the lidar Depth camera basic observation noise ; Randomly generated noise scaling factor , , ; Computational lidar observation Depth camera observation:

[0056] Constructing 68-dimensional features: Observation residuals (20 dimensions): bias between dual-branch observations and intermediate filtering estimates, lidar residuals. Depth camera residual .

[0057] State evolution difference (18 dimensions): Inter-frame changes in the intermediate filtered estimate. .

[0058] Model innovation (30 dimensions): Prediction bias of the three types of motion models. Each of the 10 frames × 1 dimension.

[0059] Generate 100 independent trajectories (2000 frames each), each trajectory contains N samples, specifically each sample includes: a 68-dimensional feature vector sequence, a model true probability sequence, and a noise scaling factor sequence.

[0060] The Time Attention LSTM model is abstracted into an architecture of "temporal feature extraction - attention aggregation - dual-task output", as follows: LSTM feature extraction layer: 2 stacked LSTM layers, with an input dimension of 7 and a hidden layer dimension of 64, capturing temporal dependencies within 10 frames, where the features of each frame are 2-dimensional observation residuals + 2-dimensional state differences + 3-dimensional model innovation sequence; Temporal Attention Layer: Attention scores for the hidden states in each frame are calculated using a fully connected layer, normalized with Softmax, and then weighted and aggregated to highlight key frame features such as model switching points. The aggregation formula is as follows: ,in For attention weights, .

[0061] Model probability output branch: The fully connected layer outputs 3-dimensional probabilities, which are then normalized using Softmax. The corresponding model fit probabilities are three types. Noise factor output branch: The fully connected layer outputs a 3D factor, which is activated by ReLU and offset to ensure non-negativity, resulting in... .

[0062] To ensure the prediction accuracy of both the model probability and the noise adjustment factor, we use a multi-task loss function to jointly train the network: ; KL divergence loss is used to optimize the matching degree between the probability distribution of multiple models and the real motion pattern. The calculation formula is: ;

[0063] in, The mean squared error loss is used to optimize the prediction accuracy of the noise adjustment factor, and its calculation formula is as follows: ; The example employs an alternating optimization strategy to decouple the two-branch training, thereby stabilizing the training process and ensuring a balanced learning of model probabilities and noise factors. Fixed noise fully connected layer branch parameters , As a constant, the total loss is calculated as follows: At this point, the deviation between the model probability and the actual motion pattern is minimized; Then fix the model probability branch parameters , As a constant, the total loss is calculated as follows: Training the model improves the regression accuracy of noise factors.

[0064] To verify the ability of Time Attention LSTM to recognize motion patterns, this embodiment performs 10 random independent samplings on the simulated pedestrian motion scenario characterized by "straight line-uniform speed-turning" to construct a test sample set. For detailed verification results, please refer to [link to relevant documentation]. Figure 7 Table 1, as shown in the table, intuitively illustrates the performance of this sub-network in both motion pattern matching and dynamic noise adaptation.

[0065] The average KL divergence is only 0.005126, and the MAE probabilities of the three motion models are all less than 0.005, indicating that the subnetwork can accurately identify different motion modes of pedestrians, such as uniform speed, uniform acceleration, and nonlinear turning. The output model fit probability is high. The results closely match the actual motion state. This provides a reliable weighting basis for subsequent multi-model UKF fusion, avoiding tracking accuracy loss due to weight errors. Type III noise adjustment factors of All values ​​are greater than 0.9, with the maximum MAE being only 0.066098. This indicates that the subnetwork can dynamically capture the changing patterns of process noise, lidar observation noise, and depth camera observation noise, and the output factors can accurately update the core filtering parameters. , , ; like Figure 8 As shown, this embodiment is preferred. Ten frames are randomly selected from 2000 frames in the test set to intuitively verify the probability prediction results of the TimeAttention LSTM model. "Successful prediction" is defined as a high degree of consistency between the core distribution features of the predicted probability and the true probability. Sampling results show that such successful samples account for 90% or more. Even for samples that do not fully meet the "high degree of consistency in core distribution features" standard, the model still possesses a certain probability of correct model selection. This characteristic ensures the robustness of the subsequent fusion process from the outset.

[0066] To evaluate the accuracy and goodness of fit of the noise factor prediction, the coefficient of determination was selected. As a core indicator, its core function is to measure the proportion of the variance of the predicted value that is explained by the actual value. The definition of this indicator is: , in, The mean of the true values ​​of the noise factor, specifically... The closer the value is to 1, the better the fit between the predicted and actual values. This indicates that the model's prediction performance for noise factors is inferior to that of a simple model that directly uses the mean of the true values.

[0067] like Figure 8 As shown, a quantitative evaluation of the prediction results for the noise scaling factor on the test set is conducted, and the results are as follows: All indicators are close to 1.

[0068] Specifically, the prediction model for The prediction effect is optimal, for and The predictive performance is also quite good; all three show significant linear correlation with the ideal distribution and excellent fitting accuracy, which can accurately capture the changing patterns of the real factors and have high reliability in practical applications, providing solid theoretical and data support for the effectiveness of the model.

[0069] This embodiment deploys the complete algorithm flow described above. This part verifies the performance advantages of the proposed Time Attention LSTM-KF-UKF fusion method by processing the collected multimodal sensor data offline. The core objectives include: comparing the trajectory estimation accuracy of single-modal sensors, traditional methods, and the proposed method; verifying the robustness of the proposed method in scenarios with missing sensor data; and evaluating the role of dynamic parameter adjustment of temporal attention LSTM and multi-model UKF optimization in improving fusion performance.

[0070] Simulation data source: To make pedestrian walking postures and social mechanics logic more consistent with real-world situations, a pedestrian social mechanics plugin was used in the Gazebo simulator to generate raw simulation data, enabling dynamic obstacle avoidance of the surrounding environment and pedestrians.

[0071] Ground Truth: In a simulation environment, a pedestrian walks on a pre-set trajectory on the ground. The ground truth of the trajectory has a certain geometric shape and is processed by key point interpolation to become the real trajectory. Traditional single-mode (Lidar-Bev): radar branch Bev projection trajectory.

[0072] Traditional single-mode (Depth-Bev): Bev projection trajectory of depth camera.

[0073] IMM-KF: Using an interactive multi-model Kalman filter to fuse trajectories from dual-branch observations.

[0074] IMM-UKF: Trajectory Fusion Results of Two-Branch Observations Using Interactive Multi-Model Unscented Kalman Filter Time Attention LSTM-KF: The result of fusing information from two branches using dynamic noise amplification coefficients updated by Time Attention LSTM.

[0075] Time Attention LSTM-KF-UKF: Using Dynamic Noise and model probability Results of multi-model UKF fusion of dual-branch data.

[0076] The root mean square error of Euclidean distance (RMSE) is used as the core evaluation index to quantify the deviation between the trajectory estimate and the true value. The formula is as follows: , Where N is the total number of frames in the trajectory. The estimated position of the i-th frame This represents the true location of the i-th frame. A smaller RMSE indicates higher trajectory estimation accuracy.

[0077] In this embodiment, the pedestrian is controlled to perform circular motion with a constant radius and an indefinite angular velocity. The experimental results are as follows: Figure 9 As shown, the dual-branch observation data exhibits significant instability before initial fusion, not only due to random lag but also due to observation noise, resulting in uneven distribution of observation points.

[0078] In scenarios where LiDAR branch observations are missing in frames 50-100, Time Attention LSTM dynamically adjusts the fusion noise coefficient. , This allows the initial fusion results to place greater emphasis on the observation information from the depth camera branch; simultaneously, the perception system accurately identifies the current motion state, significantly increasing the fusion weight of Model 3 (the nonlinear turning model) and reducing the proportion of other models, such as... Figure 10 As shown, the model can still achieve stable probability prediction even when single-branch observations are missing. This capability is due to the diverse design of the training data, which includes the simulation generation of real-world scenarios such as temporary occlusion of single-branch paths and missing data, providing sufficient support for model generalization.

[0079] Leveraging the strong adaptability of UKF to nonlinear motion, the Time Attention LSTM-KF-UKF framework performs secondary optimization on the dynamic fusion results of dual-branch observations, significantly improving the smoothness of the final output trajectory and the uniformity of the spatial distribution of sampling points. Comparison results are shown below. Figure 11 and Figure 12 As shown, the proposed method improves accuracy by 55.93% compared to the IMM-KF baseline method, by 43.03% compared to the IMM-UKF baseline method, and by 10.15% compared to the LSTM-KF fusion method, fully demonstrating the synergistic effect of nonlinear optimization and adaptive fusion strategies.

[0080] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. An adaptive parameter filter for multimodal data fusion in pedestrian detection, characterized in that, Used to perform the following steps: S1: Acquire multimodal data of the region to be detected, and perform noise reduction data preprocessing on the multimodal data respectively; S2: Perform semantic association on the denoised preprocessed multimodal data to obtain a set of dual-branch 3D pedestrian detection boxes; S3: Perform time synchronization and spatial coordinate transformation on the dual-branch pedestrian 3D detection box to obtain the dual-branch observation sequence in the bird's-eye view coordinate system. The dual-branch observation sequence includes lidar branch observation and depth camera branch observation. S4: The lidar branch observations and depth camera branch observations are fused sequentially using Kalman filtering to obtain the intermediate state estimation sequence; S5: Construct a temporal feature window, extract the observation residuals of the lidar branch, the observation residuals of the depth camera branch, the intermediate state estimation sequence, and the corresponding innovation sequences calculated based on each pre-set motion model within the temporal window, concatenate them sequentially, and input them into a pre-trained temporal attention long short-term memory network. Output the adaptation probability and noise adjustment factor of each motion model, and dynamically update the noise matrix according to the noise adjustment factor. S6: Based on the updated noise matrix, perform unscented Kalman filtering on each motion model, obtain the motion model estimate through state prediction and update, and then fuse them according to the corresponding adaptation probability to output the optimal state estimate and tracking trajectory of the pedestrian.

2. The adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 1, characterized in that, The acquisition of multimodal data in S1 specifically involves using a lidar, a depth camera, and an RGB camera to acquire 3D point cloud data, RGB image data, and depth image data of the area to be detected, respectively.

3. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 2, characterized in that, The noise reduction preprocessing in S1 includes: The three-dimensional point cloud data is denoised and downsampled using voxel mesh to obtain the first point cloud data; The RGB image data is subjected to distortion correction and scale normalization. The depth image data is subjected to hole filling and noise smoothing processing, and then converted into three-dimensional point cloud data, which is the second point cloud data.

4. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 3, characterized in that, S2 specifically includes the following steps: S21: Convert the first point cloud data and the second point cloud data to the RGB image coordinate system; S22: Perform planar clustering and segmentation on the transformed first point cloud data and second point cloud data respectively to generate multiple lidar-side 3D candidate boxes and multiple depth camera-side 3D candidate boxes; A pre-trained detection network is used to output 2D detection boxes based on RGB image data; S23: Obtain the correlation between the 3D candidate boxes on the LiDAR side and the 3D candidate boxes on the depth camera side and the 2D detection boxes respectively, and filter out multiple 3D candidate boxes on the LiDAR side and multiple 3D candidate boxes on the depth camera side that meet the conditions, and set them into a set of dual-branch 3D pedestrian detection boxes.

5. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 2, characterized in that, S3 specifically includes the following steps: S31: Set up data cache queues for LiDAR, depth camera and RGB camera respectively to store the raw data and corresponding timestamps within a recent time period; S32: Using the timestamp of RGB image data as a reference, linear interpolation is used to time-align the lidar point cloud and depth point cloud to generate a multimodal data sequence with unified timestamps; S33: The center coordinates of the dual-branch 3D pedestrian detection box are transformed to the chassis coordinate system through sensor extrinsic parameters, ignoring height information, to obtain the lidar branch observation and depth camera branch observation in the bird's-eye view coordinate system.

6. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 1, characterized in that, The noise adjustment factors in S5 include process noise adjustment factors, lidar observation noise adjustment factors, and depth camera observation noise adjustment factors.

7. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 6, characterized in that, The dynamic update of the noise matrix in S5 specifically involves: multiplying the preset basic process noise covariance matrix by the process noise adjustment factor to obtain the updated process noise covariance matrix, and multiplying the preset basic observation noise covariance matrix by the corresponding observation noise adjustment factor to obtain the updated observation noise covariance matrix.

8. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 7, characterized in that, S6 specifically includes the following steps: S61: Generate multiple Sigma points based on the current state estimate and covariance matrix, and substitute each Sigma point into the state transition function of the motion model to obtain the predicted Sigma points. Then, calculate the predicted state mean and predicted covariance based on the predicted Sigma points with weights. S62: Project the predicted Sigma points onto the observation space to obtain the observation prediction values, and calculate the observation covariance matrix and the state-observation cross-covariance matrix accordingly. S63: Solve for the Kalman gain based on the observation covariance matrix and the state-observation cross-covariance matrix, and update the predicted state mean and predicted covariance using the current observation values ​​to obtain the state estimate and covariance of the motion model. S64: Based on the adaptation probability corresponding to each motion model, the state estimates of all motion models are weighted and fused to output the optimal state estimate and tracking trajectory of the pedestrian.

9. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 1, characterized in that, The intermediate state estimates in S4 are obtained as follows: S41: Using the intermediate state estimate from the previous time step, the prior state estimate for the current time step is obtained through the prediction step of Kalman filtering; then, the Kalman gain is calculated using the observation of the lidar branch, and the posterior state estimate is updated to obtain a new one. S42: Based on the aforementioned posterior state estimate, the Kalman gain is recalculated using depth camera branch observations, and the intermediate state estimate for the current time moment is updated again.

10. An adaptive parameter filter for multimodal data fusion in pedestrian detection according to claim 1, characterized in that, The motion models pre-defined in S5 specifically include uniform linear motion model, uniform acceleration model, and non-linear turning model.

Images