Three-dimensional space moving target detection method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of moving target detection, and discloses a three-dimensional space moving target detection method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a multi-modal sensor data sequence, carrying out the point cloud data fusion of the multi-modal sensor data sequence, and obtaining a fused point cloud data sequence; separating a moving target pixel region and a static background region from the fused point cloud data sequence, and performing time sequence optimization to obtain a dense point cloud sequence including a timestamp; performing motion target pose estimation according to the dense point cloud sequence including the timestamp to obtain a pose estimation sequence; and performing moving target matching according to the pose estimation sequence to obtain a tracking moving target. Through the implementation of the invention, the target deformation and the high-speed motion scene in the moving target identification can be effectively processed, and the dynamic characteristics of the target can be more comprehensively described, so that the moving target matching can be more accurately carried out, and the tracked moving target can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of moving target detection, and particularly to a three-dimensional space moving target detection method, an electronic device and a storage medium. BACKGROUND

[0002] With the development of science and technology, three-dimensional space target detection as a key technology in the field of computer vision plays a pivotal role in many frontier fields, for example, realizing high-precision identification and trajectory tracking of dynamic targets such as unmanned aerial vehicles and aircrafts.

[0003] In the traditional field of air target tracking, mainly relying on single sensor systems (such as pure vision or radar) and static three-dimensional reconstruction technologies such as SFM (Structure from Motion) and SLAM (Simultaneous Localization and Mapping) to realize target perception and positioning.

[0004] Among them, the pure vision system in the single sensor system estimates the target pose through multi-view geometric algorithms (such as perspective n-point problem and triangulation), but is limited by light changes and occlusions, and has a high miss rate in low light and backlight scenes; although the millimeter wave radar has all-weather detection capability, the sparseness of the radar point cloud leads to the loss of target details (such as the inability to distinguish the rotors of an unmanned aerial vehicle). Therefore, the single sensor system cannot effectively cope with complex scenes such as occlusion and target intersection. At the same time, the static three-dimensional reconstruction technology has high reconstruction error for high-speed moving targets, and cannot handle target deformation and motion blur, resulting in model tearing or missing, and dynamic segmentation in complex backgrounds is easily disturbed by light changes, rain and fog, with low accuracy.

[0005] In summary, how to improve the accuracy of three-dimensional space target detection has become a problem to be solved. SUMMARY

[0006] The present application provides a three-dimensional space moving target detection method, an electronic device and a storage medium, which mainly aims at the problem of low accuracy of three-dimensional space target detection.

[0007] To achieve the above-mentioned purpose, the three-dimensional space moving target detection method provided by the present application comprises: acquiring a multi-modal sensor data sequence, fusing point cloud data of the multi-modal sensor data sequence to obtain a fused point cloud data sequence; separating a moving target pixel region and a static background region from the fused point cloud data sequence; respectively performing time series optimization on the moving target pixel region and the static background region to obtain a dense point cloud sequence including a timestamp; performing moving target pose estimation according to the dense point cloud sequence including the timestamp to obtain a pose estimation sequence; and performing moving target matching according to the pose estimation sequence to obtain a tracked moving target.

[0008] The application further provides a three-dimensional space moving target detection system, comprising: a data fusion module, configured to acquire a multi-modal sensor data sequence, perform point cloud data fusion on the multi-modal sensor data sequence, and obtain a fused point cloud data sequence; a region separation module, configured to separate a moving target pixel region and a static background region from the fused point cloud data sequence; a time sequence optimization module, configured to perform time sequence optimization on the moving target pixel region and the static background region respectively, and obtain a dense point cloud sequence comprising a timestamp; a pose estimation module, configured to perform moving target pose estimation according to the dense point cloud sequence comprising the timestamp, and obtain a pose estimation sequence; and a target matching module, configured to perform moving target matching according to the pose estimation sequence, and obtain a tracked moving target.

[0009] The application further provides an electronic device, comprising: a memory in communication connection with at least one processor; wherein the processor is configured to execute a computer program stored on the memory; the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the three-dimensional space moving target detection method.

[0010] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the three-dimensional space moving target detection method of any one of the above.

[0011] The embodiment of the application can obtain a fused point cloud data sequence by performing point cloud data fusion on a multi-modal sensor data sequence, can form a standardized input under a unified space-time reference, eliminate errors caused by sensor heterogeneity, and improve the accuracy of subsequent moving target recognition; can obtain a more accurate moving target pixel region by separating a moving target pixel region and a static background region from the fused point cloud data sequence; can effectively process target deformation and high-speed motion scenes in moving target recognition, and further improve the accuracy of moving target recognition by performing time sequence optimization on the moving target pixel region and the static background region to obtain a dense point cloud sequence comprising a timestamp; and can more accurately perform moving target matching to obtain a tracked moving target by performing moving target pose estimation according to the dense point cloud sequence comprising the timestamp to obtain a pose estimation sequence. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A flowchart of a three-dimensional space moving target detection method provided by an embodiment of the application;

[0013] Figure 2 A flowchart of point cloud data fusion on a multi-modal sensor data sequence provided by an embodiment of the application;

[0014] Figure 3 A flowchart of a motion target pose estimation method provided by an embodiment of the present application is shown in FIG. 1.

[0015] Figure 4 A function module diagram of a three-dimensional space motion target detection system provided by an embodiment of the present application is shown in FIG. 2.

[0016] Figure 5 A structural diagram of an electronic device for implementing a three-dimensional space motion target detection method provided by an embodiment of the present application is shown in FIG. 3.

[0017] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0019] An embodiment of the present application provides a three-dimensional space motion target detection method. The execution subject of the three-dimensional space motion target detection method includes but is not limited to at least one of the electronic devices that can be configured to execute the method provided by the present application, such as a server and a terminal. In other words, the three-dimensional space motion target detection method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc.

[0020] Referring to FIG. 1, a flowchart of a three-dimensional space motion target detection method provided by an embodiment of the present application is shown. In this embodiment, the three-dimensional space motion target detection method includes: Figure 1 S1, acquiring a multi-modal sensor data sequence, performing point cloud data fusion on the multi-modal sensor data sequence to obtain a fused point cloud data sequence.

[0021]

[0022] ​In the embodiment of the present application, the multi-modal sensor data sequence is a plurality of time frames of corresponding sensor data collected by a plurality of modal sensors, wherein the multi-modal sensor high frame rate industrial camera (≥ 30 fps) captures image data composed of target RGB / infrared images, radar point cloud data obtained by millimeter wave radar or laser radar, and inertial measurement data including acceleration, angular velocity and latitude and longitude coordinate data collected by inertial measurement unit GPS (Global Positioning System) at high frequency, etc. The image data collected by the industrial camera, the sparse point cloud data collected by the radar and the acceleration, angular velocity and latitude and longitude coordinate data are collected to obtain multi-modal sensor data. The multi-modal sensor data sequence can be obtained by collecting multi-modal sensor data of multiple time frames in time sequence. The motion of the target in space can be analyzed according to the change of the time frame through the multi-modal sensor data sequence, so as to match the moving target.

[0023] Further, the point cloud data fusion is to form standardized input under the unified space-time reference from different sensors, eliminate the error caused by sensor heterogeneity, and improve the accuracy of subsequent point cloud reconstruction.

[0024] In the embodiment of the present application, referring to Figure 2 The point cloud data fusion of the multi-modal sensor data sequence is obtained to obtain a fused point cloud data sequence, comprising:

[0025] S21, respectively, data preprocessing is performed on the multi-modal sensor data in the multi-modal sensor data sequence to obtain a preprocessed data sequence;

[0026] S22, the radar point cloud data and the image data in the preprocessed data sequence are registered to obtain registered data;

[0027] S23, the inertial measurement data in the preprocessed data sequence is aligned by cubic spline interpolation to obtain aligned data;

[0028] S24, data fusion is performed on the registered data and the aligned data to obtain a fused point cloud data sequence.

[0029] Specifically, the multi-modal sensor data is the data collected by the multi-modal sensor corresponding to each time frame, and the multi-modal sensor data sequence is processed in time sequence according to the time frame to obtain a plurality of time frames corresponding to the fused point cloud data sequence. The data preprocessing includes image de-warping and histogram equalization processing of the image in the multi-modal sensor data sequence; the radar point cloud data is down-sampled and the statistical outliers are removed by voxel filtering (resolution 1cm 3 ) to obtain a preprocessed data sequence.

[0030] The image de-distortion is a process of correcting image distortion caused by factors such as a lens, and specifically, a series of images of a known geometric pattern (such as a checkerboard) can be pre-shot by using a camera, the intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (position and posture of the camera in the world coordinate system) of the camera and distortion coefficients can be calculated by analyzing the positions of feature points in the images, after the distortion parameters are obtained, each pixel point in the image is geometrically transformed by using the parameters, the distorted image is mapped to the corrected image coordinate system, and a pre-processed image is obtained.

[0031] Further, the image histogram equalization is to calculate the number of pixels of each gray value in the image to obtain a histogram of the image, calculate a cumulative distribution function according to the histogram, the cumulative distribution function represents the sum of probabilities less than or equal to the gray level, and the gray level of each pixel in the original image is mapped and transformed by the cumulative distribution function to obtain a new gray level, thereby generating an equalized image.

[0032] In the embodiment of the application, the point cloud registration is to calculate a camera-radar extrinsic parameter matrix by using an ICP (Iterative Closest Point) algorithm, so as to project the radar point cloud to an image coordinate system, register the pre-processed radar point cloud data and the data in the image, and obtain registration data.

[0033] In detail, the inertial measurement data in the pre-processed data sequence is aligned by cubic spline interpolation to obtain aligned data, including: extracting the time stamp and measurement data of each data point in the inertial measurement data; sorting the measurement data according to the time stamp to obtain an array sequence corresponding to the time stamp and the measurement data; selecting an interpolation node according to the array sequence, and constructing a cubic polynomial coefficient of the interpolation node; and performing data interpolation based on the cubic polynomial coefficient to obtain the aligned data.

[0034] In the embodiment of the application, the time stamp and corresponding measurement values such as acceleration and angular velocity of each data point are extracted from the inertial measurement data to obtain the time stamp and measurement data of each data, which are arranged in order of time stamp to form a plurality of time-measurement value pairs, for example, there are a plurality of data pairs such as (time stamp, acceleration) for acceleration data, and there are also data pairs such as (time stamp, angular velocity), and an array sequence sorted by time is obtained.

[0035] Further, the time range of the entire time stamp is taken as an interpolation interval, and the interpolation nodes in the interpolation interval are selected according to the preset interval time points, or can be unevenly distributed according to the change characteristics and density of the data in the array sequence.

[0036] Further, based on the selected interpolation nodes, the mathematical method of cubic spline interpolation is used to calculate the cubic polynomial coefficients on each interpolation interval segment. Specifically, a three-diagonal equation set is constructed, the second derivative values of each segment curve at the nodes are obtained by solving the equation set, then the coefficients of each cubic polynomial are determined by combining the preset interpolation conditions and boundary conditions, and then the data value at the time stamp of the interpolation node is calculated by substituting the corresponding cubic polynomial into the interpolation function, including acceleration and angular velocity, to obtain the aligned data.

[0037] In the embodiment of the present application, the registration data and the aligned data can be converted to a unified coordinate system, and then data fusion is performed using a data fusion method based on filtering, a feature matching method, etc., to obtain a fused point cloud data sequence, and then the multi-modal sensor data can be standardized under a unified space-time reference to eliminate errors caused by sensor heterogeneity and improve the accuracy of subsequent motion target recognition.

[0038] S2, separating the motion target pixel region and the static background region from the fused point cloud data sequence.

[0039] In the embodiment of the present application, the motion target pixel region is the pixel region corresponding to the inter-frame motion region in the fused point cloud data sequence, and the static background region is the scene part that remains static or changes very slowly relative to the dynamic target or moving object in the fused point cloud data sequence, and then the static background region is obtained by separating the motion target from the fused point cloud data sequence and then modeling the static background.

[0040] Specifically, separating the motion target pixel region and the static background region from the fused point cloud data sequence includes: converting the fused point cloud data sequence into a Gaussian distribution data sequence, densifying the Gaussian distribution data sequence to obtain a dense point cloud data sequence; denoising and point cloud registration processing the dense point cloud data sequence to obtain registered point cloud data; calculating the optical flow field between the registered point cloud data, and selecting motion candidate points from the target point cloud data sequence according to the optical flow field; performing connected domain analysis and morphological operation on the motion candidate points to obtain an initial motion region; and separating the motion target pixel region and the static background region from the registered point cloud data according to the initial motion region using a pre-constructed fine segmentation neural network.

[0041] In the embodiment of the present application, the covariance matrix can be calculated according to the mean of the point cloud data points in each fused point cloud data in the fused point cloud data sequence, and then the Gaussian distribution random number generation function can be used to generate data conforming to the distribution according to the calculated mean and covariance matrix, to obtain a Gaussian distribution data sequence.

[0042] In detail, Gaussian compaction is a method of making sparse point cloud data in a Gaussian distribution data sequence denser and smoother by modeling it with a Gaussian distribution, which can more accurately and densely represent the geometric and appearance information in the scene.

[0043] In this embodiment of the invention, a Gaussian distributed data sequence is densified to obtain a dense point cloud data sequence; the process includes: performing rasterization rendering on the Gaussian distributed data in the Gaussian distributed data sequence to obtain a rendered image; calculating the rendering loss value between the rendered image and a preset real image; identifying the region to be densified in the 3D Gaussian distributed data based on the rendering loss value; and performing Gaussian densification iterative processing on the region to be densified to obtain a dense point cloud data sequence.

[0044] In this embodiment of the invention, the rasterization rendering process transforms the position, covariance matrix, opacity and other parameters of the Gaussian data in the Gaussian distribution data into the camera coordinate system through camera extrinsic parameters (translation and rotation matrix W), then maps it to the image plane through the projection matrix K, then divides the mapped image into blocks, filters out the Gaussian points in the view frustum, and sorts them by depth.

[0045] In detail, the correct occlusion relationship between Gaussian data is ensured by depth sorting, and then the color and opacity of multiple Gaussian data are mixed using the alpha synthesis formula to gradually superimpose the final synthesized color onto the frame buffer to complete the image synthesis and obtain the rendered image.

[0046] The alpha synthesis formula is a scalar between 0 and 1, representing the transparency (or opacity) of an object. It is a key parameter used to control color blending and transparency. The alpha synthesis formula can measure the "visibility" of the color and opacity of multiple Gaussian data when they are blended and superimposed on the target image. This ensures that occlusion relationships are handled correctly during subsequent rendering, making the rendered image conform to the real scene.

[0047] Furthermore, the preset real image is an image actually captured or recorded during moving target matching. It is obtained from the real-world scene and reflects the true appearance of the moving target matching. It can be a predefined reference image. By comparing the rendered image with the real image, the quality of the rendering result can be evaluated, and the parameters of the Gaussian distribution can be adjusted according to the differences to improve the accuracy of 3D scene reconstruction.

[0048] In this embodiment of the invention, the loss value includes L1 loss and a weighted loss of structural similarity (Differentiable SSIM Loss). The L1 loss is obtained by calculating the absolute error pixel by pixel for the rendered image and the real image and averaging it. The structural similarity loss is an indicator that measures the structural similarity between two images and is calculated from three dimensions: brightness (L), contrast (C), and structure (S).

[0049] In detail, the regions where the rendering loss value of 3D Gaussian distribution data is greater than a preset threshold are designated as regions to be densified. In the regions that need densification, the number of Gaussians is increased by splitting existing Gaussians or inserting new Gaussians at appropriate positions. The regions to be densified are subjected to Gaussian densification iterative processing until the loss is less than the preset threshold, resulting in a dense point cloud data sequence.

[0050] In this embodiment of the invention, by densifying the Gaussian distributed data sequence, the Gaussian distributed data can transition from sparse to medium density, more meticulously depicting the geometric shape and appearance features of moving targets in the fused point cloud data, obtaining a more accurate 3D reconstruction model of the target, and achieving more accurate moving target matching.

[0051] Furthermore, the Gaussian distributed data sequence is filtered and denoised. The denoised point cloud data is then input into the ICP Iterative Closest Point (ICP) point cloud registration model to register point cloud data from different frames to the same coordinate system, thereby ensuring the accuracy of subsequent optical flow field calculations.

[0052] In detail, geometric features of the registered point cloud data, such as XYZ axis coordinates and normal vectors, are extracted and a pre-constructed point cloud optical flow network model is used. This model is then used to calculate the optical flow field information between two frames of point clouds. Based on the calculated optical flow field, the optical flow vector magnitude of each point is calculated. An appropriate magnitude threshold is set, and points with optical flow vector magnitudes greater than the threshold are selected as motion candidate points. Connectivity analysis and morphological operations are then performed on these motion candidate points to obtain the initial motion region.

[0053] Optical flow refers to the two-dimensional vector field formed by the apparent motion of the image's grayscale pattern. The motion of pixels is determined by analyzing the grayscale changes and positional differences between adjacent frames. The pre-built point cloud optical flow network model can be a pre-trained convolutional neural network incorporating the Huber loss function.

[0054] Furthermore, the optical flow vector magnitude refers to the size of the optical flow vector, that is, the length of the optical flow vector in two-dimensional space, which is used to represent the speed at which a pixel moves between adjacent frames.

[0055] In this embodiment of the invention, the initial motion region may include all regions in the entire scene where motion occurs, not just the moving target itself. For example, when the camera moves, all pixels in the entire scene may be considered as inter-frame motion regions, even if some of these pixels belong to the static background. The moving target pixel region, on the other hand, only includes the pixels occupied by the actually moving object. For instance, in surveillance video, if an airplane moves in the scene, the moving target pixel region is the set of pixels occupied by the airplane. Therefore, by using the moving target pixel region and the static background region, moving targets in point cloud data sequences can be separated more accurately.

[0056] Specifically, the pre-built fine segmentation neural network is a MotionNet neural network (motion network neural network) based on a convolutional neural network (CNN) architecture, with the addition of an attention mechanism and a context information module. Through the fine segmentation neural network, the features of the motion region can be learned better, thereby accurately segmenting the running pixel region in the initial motion region and obtaining a more accurate motion target pixel region.

[0057] S3. Perform temporal optimization on the moving target pixel region and the static background region respectively to obtain a dense point cloud sequence including timestamps.

[0058] In this embodiment of the invention, temporal optimization involves reconstructing Poisson surfaces for both the moving target pixel region and the static background region to generate continuous surfaces, thereby obtaining corresponding three-dimensional point cloud data. The three-dimensional point cloud data is then sorted according to the time frames of the acquisition to obtain a dense point cloud sequence with timestamps.

[0059] In detail, temporal optimization is performed on the moving target pixel region and the static background region respectively to obtain a dense point cloud sequence including timestamps. This includes: predicting the target displacement of the moving target pixel region to obtain the target displacement; performing motion blur compensation on the moving target pixel region based on the target displacement to obtain the optimized region; converting the optimized region into 3D point cloud data, performing Laplacian smoothing on the point cloud data to obtain smoothed point cloud data; reconstructing the smoothed point cloud data using a Poisson surface to obtain a reconstructed surface, fusing the reconstructed surface with the 3D point cloud data to obtain reconstructed point cloud data; combining the reconstructed point cloud data and the corresponding 3D point cloud data of the static background region to obtain temporally optimized point cloud data; and sorting the temporally optimized point cloud data according to timestamps to obtain a dense point cloud sequence including timestamps.

[0060] In this embodiment of the invention, displacement prediction is performed by using Kalman filtering to predict the motion trend and velocity information of a moving target in an image sequence composed of moving target pixel regions. Specifically, Kalman filtering determines the initial state by sequentially detecting the initial pixel coordinates of the moving target in the moving target pixel region corresponding to the time frame, and initializes the velocity of the moving target. By constructing a motion model (state equation) and an observation model (based on image detection results) for the pixel coordinates, the real-time pixel position of the target is estimated, and the future displacement is extrapolated based on the latest state to obtain the displacement of the moving target in the moving target pixel region as the time frame progresses.

[0061] Furthermore, motion blur compensation utilizes Kalman filtering to predict the target displacement, calculates the target's motion trajectory, establishes a motion blur model based on the motion trajectory and exposure time, and then uses inverse filtering, Wiener filtering, or deep learning-based image deblurring algorithms to perform motion blur compensation on the pixel region of the moving target, resulting in a clear optimized region.

[0062] In detail, the pixels in the optimization area can be converted into 3D point cloud data by collecting the intrinsic and extrinsic parameters of the camera. Then, the point cloud data can be converted into a triangular mesh model. Laplacian smoothing is performed on the triangular mesh to reduce high-frequency noise and irregularities in the mesh. Laplacian smoothing updates the position of each vertex to the average of the positions of its neighboring vertices.

[0063] In this embodiment of the invention, a continuous surface is generated by reconstructing a Poisson surface. Three-dimensional point cloud data is mapped onto the surface based on methods such as projection and interpolation, or the surface is used as an implicit representation of the point cloud. New reconstructed point cloud data is generated from the surface through sampling and other methods. The reconstructed point cloud data and the three-dimensional point cloud data corresponding to the static background area are collected to obtain temporally optimized point cloud data. Then, the temporally optimized point cloud data is sorted according to the time frame corresponding to the moving target pixel area and the static background area to obtain a dense point cloud sequence including the timestamp.

[0064] In detail, by performing temporal optimization on the pixel regions of moving targets and static background regions, a dense point cloud sequence including timestamps is obtained, which can effectively handle target deformation and high-speed motion scenes in moving target recognition, and further improve the accuracy of moving target recognition.

[0065] S4. Perform pose estimation of the moving target based on the dense point cloud sequence including timestamps to obtain the pose estimation sequence.

[0066] In this embodiment of the invention, the pose estimation sequence is the position and attitude of a moving target in space at different time frames, which can be predicted by the target's pose at the next moment (the state vector includes position, velocity and acceleration).

[0067] For details, please refer to Figure 3 As shown, the pose estimation of a moving target is performed based on a dense point cloud sequence including timestamps, resulting in a pose estimation sequence, including:

[0068] S31. Extract the geometric features of the first time-stamp dense point cloud in the dense point cloud sequence including timestamps;

[0069] S32. Align the dense point cloud of the first timestamp with the preset template point cloud according to the geometric features to obtain the initial alignment result;

[0070] S33. Perform Gaussian mixture optimization on the initial alignment result to obtain the first pose parameter corresponding to the first time-stamp dense point cloud.

[0071] S34. Based on the first pose parameter, perform motion prediction on the dense point cloud sequence including the timestamp to obtain the pose estimation sequence.

[0072] In this embodiment of the invention, the dense point cloud data represented by the first timestamp in the dense point cloud sequence including timestamps is taken as the first timestamp dense point cloud, and the geometric features (which may include centroid accuracy ±0.05m, three-dimensional oriented bounding box) and appearance features (e.g., hue-saturation-brightness histogram) of the first timestamp dense point cloud are extracted.

[0073] In detail, the point cloud alignment between the first time-stamp dense point cloud and the preset template point cloud based on geometric features is performed using the ICP (Iterative Closest Point) algorithm to obtain an initial alignment result. This initial alignment result is then used as the initial corresponding point pair, and the pose parameters are refined using the optimization method of a Gaussian Mixture Model (GMM) to obtain the first pose parameter corresponding to the first time-stamp dense point cloud.

[0074] Specifically, the Gaussian Mixture Model (GMM) is initialized using the coordinates of the point cloud in the initial corresponding point pair as the mean of the Gaussian distribution and the covariance matrix based on the distance distribution between the point pairs. The mixing coefficients can be initially set to equal values ​​or preliminarily estimated based on the reliability of ICP matching. The mean of the Gaussian distribution, the initialized covariance matrix, and the initialized mixing coefficients are used as the initialization parameters of the GMM. By initializing the parameters of the GMM and constructing the objective function based on the probabilistic characteristics of the GMM, the objective function is usually the log-likelihood function. By maximizing the log-likelihood function, the joint optimization of the GMM parameters and pose parameters can be achieved, thus obtaining the first pose parameter corresponding to the first timestamp dense point cloud.

[0075] Furthermore, motion prediction of the moving target can be performed using Kalman filtering, thereby predicting the pose at the next moment (the state vector includes position, velocity, and acceleration), and obtaining a pose estimation sequence.

[0076] In detail, in a continuous dense point cloud sequence including timestamps, Kalman filtering can predict the subsequent pose based on the target's initial position and velocity, and update the current pose based on the new pose to obtain the pose estimation sequence of the moving target.

[0077] In this embodiment of the invention, motion prediction can update the pose state of a moving target, such as its position and velocity. The pose estimation sequence records the pose changes of the moving target over a period of time, thus providing a more comprehensive characterization of the target's dynamic features, including its direction of motion, velocity, acceleration, and rotation. For example, when identifying a drone, the pose estimation sequence can reveal whether it is accelerating upwards, cruising smoothly, or rapidly descending, allowing for more accurate moving target matching and identification.

[0078] S5. Match moving targets based on the pose estimation sequence to obtain the tracked moving targets.

[0079] In this embodiment of the invention, moving target matching involves matching the pose estimation sequence of a moving target in a multimodal sensor data sequence with an existing target motion trajectory to obtain the tracked moving target in space.

[0080] In this embodiment of the invention, matching a moving target based on a pose estimation sequence to obtain a tracked moving target includes: obtaining the centroid of the moving target based on the pose estimation sequence, and calculating the Euclidean distance in centroid space between the centroid of the moving target and a preset centroid trajectory; determining the three-dimensional directed bounding box of the moving target based on the pose estimation sequence, and calculating the intersection-union ratio (IU) of the three-dimensional directed bounding box and the three-dimensional directed bounding box of the preset bounding box sequence; obtaining the color space histogram corresponding to the pose estimation sequence, and calculating the one-step similarity between the color space histogram and the preset histogram sequence; constructing a matching cost function based on the centroid space Euclidean distance, the IU of the three-dimensional directed bounding box, and the one-step similarity; and identifying the tracked moving target corresponding to the pose estimation sequence based on the matching cost function.

[0081] In this embodiment of the invention, in the pose estimation sequence, the target pose at each time step typically includes position (e.g., coordinates) and orientation (e.g., rotation matrix or Euler angles) in three-dimensional space. This information can be used to determine the centroid position of the target in each time frame. The corresponding centroid positions in the pose estimation sequence at the same time frame are obtained and the Euclidean distance is calculated to obtain the centroid space Euclidean distance. The preset centroid trajectory is a tracking trajectory composed of the existing centroid positions of multiple known moving targets in each time frame before moving target matching.

[0082] Furthermore, the 3D directed bounding box is a 3D OBB (Oriented Bounding Box) bounding box, which describes the occupied area of ​​a moving target in 3D space. The center coordinates of the bounding box can be obtained by using the position in pose estimation as the center coordinates, and then determining the rotation relationship between the local coordinate system of the moving target and the global coordinate system based on the attitude information to obtain the axis of the 3D directed bounding box. The occupied area is determined based on the center coordinates, the axis, and the preset length of the moving target in each axis, thus obtaining the 3D directed bounding box.

[0083] In detail, the intersection-union ratio of a 3D directed bounding box is the ratio of the intersection volume to the union volume among bounding boxes corresponding to multiple known moving targets in each time frame, and can be calculated by calculating the overlapping parts between the bounding boxes.

[0084] In this embodiment of the invention, a single-step similarity can be obtained by comparing the histograms of hue (H), saturation (S), and luminance (V) in the pixel region of the moving target corresponding to the pose estimation sequence with the similarity of multiple known moving targets in each time frame.

[0085] Furthermore, by using preset weights to perform a weighted summation of the Euclidean distance in the centroid space, the intersection-union ratio of the three-dimensional directed bounding boxes, and the single-step similarity, a matching cost function is obtained, which can measure the similarity between the moving target and multiple known moving targets. The minimum value of the matching cost function of multiple known moving targets is selected as the tracking moving target, and the tracking matching of the moving target can be performed to obtain the tracking moving target.

[0086] like Figure 4 The diagram shown is a functional block diagram of a three-dimensional spatial moving target detection system provided in an embodiment of the present invention.

[0087] The three-dimensional moving target detection system 400 of the present invention can be installed in an electronic device. Depending on the functions implemented, the three-dimensional moving target detection system 400 may include a data fusion module 401, a region separation module 402, a temporal optimization module 403, a pose estimation module 404, and a target matching module 405. A module, also known as a unit, refers to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0088] In this embodiment, the functions of each module / unit are as follows:

[0089] The data fusion module 401 is used to acquire multimodal sensor data sequences, perform point cloud data fusion on the multimodal sensor data sequences, and obtain fused point cloud data sequences.

[0090] The region separation module 402 is used to separate the moving target pixel region and the static background region from the fused point cloud data sequence;

[0091] The timing optimization module 403 is used to perform timing optimization on the moving target pixel region and the static background region respectively, to obtain a dense point cloud sequence including timestamps;

[0092] The pose estimation module 404 is used to estimate the pose of a moving target based on a dense point cloud sequence including timestamps, and to obtain a pose estimation sequence.

[0093] The target matching module 405 is used to match moving targets based on the pose estimation sequence to obtain the tracked moving target.

[0094] In detail, each module in the three-dimensional spatial moving target detection system 400 in this embodiment of the invention adopts the same characteristics as described above during use. Figures 1 to 3 The method uses the same techniques as the three-dimensional moving target detection method in China and can produce the same technical effects, so it will not be elaborated here.

[0095] like Figure 5 The diagram shown is a structural schematic of an electronic device for implementing a three-dimensional spatial moving target detection method according to an embodiment of the present invention.

[0096] The electronic device 500 may include a processor 501, a memory 502, a communication bus 503, and a communication interface 504. It may also include a computer program stored in the memory and capable of running on the processor, such as a three-dimensional spatial moving target detection method program.

[0097] In some embodiments, the processor 501 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0098] The memory 502 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 502 may be an internal storage unit of an electronic device, such as the portable hard drive of the electronic device.

[0099] The communication bus 503 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 502 and at least one processor 501.

[0100] Communication interface 504 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface.

[0101] Figure 5 The image only shows electronic devices with components; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 500, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0102] Specifically, the processor's implementation method for the above instructions can be found in the description of the relevant steps in the corresponding embodiments in the accompanying drawings, and will not be repeated here.

[0103] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0106] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0107] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0108] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting moving targets in three-dimensional space, characterized in that, The method includes: A multimodal sensor data sequence is acquired, and point cloud data fusion is performed on the multimodal sensor data sequence to obtain a fused point cloud data sequence. Separate the moving target pixel region and the static background region from the fused point cloud data sequence; Temporal optimization is performed on the moving target pixel region and the static background region respectively to obtain a dense point cloud sequence including timestamps; Based on the dense point cloud sequence including timestamps, the pose of the moving target is estimated to obtain a pose estimation sequence; Based on the pose estimation sequence, moving target matching is performed to obtain the tracked moving target.

2. The three-dimensional moving target detection method as described in claim 1, characterized in that, The step of fusing point cloud data from the multimodal sensor data sequence to obtain a fused point cloud data sequence includes: The multimodal sensor data in the multimodal sensor data sequence are preprocessed to obtain a preprocessed data sequence. The radar point cloud data and image data in the preprocessed data sequence are registered to obtain registration data. The inertial measurement data in the preprocessed data sequence is aligned using cubic spline interpolation to obtain aligned data; The registration data and the alignment data are fused to obtain a fused point cloud data sequence.

3. The three-dimensional spatial moving target detection method as described in claim 2, characterized in that, The step of performing cubic spline interpolation alignment on the inertial measurement data in the preprocessed data sequence to obtain aligned data includes: Extract the timestamp and measurement data for each data point in the inertial measurement data; The measurement data is sorted according to the timestamp to obtain an array sequence of the timestamp and the measurement data. Interpolation nodes are selected based on the array sequence, and the cubic polynomial coefficients of the interpolation nodes are constructed. Data interpolation is performed based on the coefficients of the cubic polynomial to obtain aligned data.

4. The three-dimensional spatial moving target detection method as described in claim 1, characterized in that, The step of separating the moving target pixel region and the static background region from the fused point cloud data sequence includes: The fused point cloud data sequence is converted into a Gaussian distribution data sequence, and the Gaussian distribution data sequence is densified to obtain a dense point cloud data sequence. The dense point cloud data sequence is subjected to denoising and point cloud registration processing to obtain registered point cloud data; Calculate the optical flow field between the registered point cloud data, and select motion candidate points from the target point cloud data sequence based on the optical flow field; Connectivity analysis and morphological operations are performed on the motion candidate points to obtain the initial motion region; Using a pre-constructed fine segmentation neural network, the moving target pixel region and the static background region are separated from the registered point cloud data based on the initial motion region.

5. The three-dimensional spatial moving target detection method as described in claim 4, characterized in that, The densification process of the Gaussian distributed data sequence to obtain a dense point cloud data sequence includes: Rasterization rendering is performed on the Gaussian distributed data in the Gaussian distributed data sequence to obtain the rendered image; Calculate the rendering loss value between the rendered image and the preset real image; Identify the regions to be densified in the 3D Gaussian distribution data based on the rendering loss value; The region to be densified is subjected to Gaussian densification iteration processing to obtain a dense point cloud data sequence.

6. The three-dimensional spatial moving target detection method as described in claim 1, characterized in that, The step of performing temporal optimization on the moving target pixel region and the static background region respectively to obtain a dense point cloud sequence including timestamps includes: Target displacement is predicted for the moving target pixel region to obtain the target displacement; Motion blur compensation is performed on the moving target pixel region based on the target displacement to obtain an optimized region; The optimized region is converted into three-dimensional point cloud data, and the point cloud data is smoothed using Laplacian smoothing to obtain smoothed point cloud data. Poisson surface reconstruction is performed on the smooth point cloud data to obtain a reconstructed surface. The reconstructed surface is then fused with the three-dimensional point cloud data to obtain reconstructed point cloud data. By combining the reconstructed point cloud data and the 3D point cloud data corresponding to the static background area, time-series optimized point cloud data is obtained. The time-series optimized point cloud data is sorted according to timestamps to obtain a dense point cloud sequence including timestamps.

7. The three-dimensional spatial moving target detection method as described in claim 1, characterized in that, The step of estimating the pose of a moving target based on the dense point cloud sequence including timestamps to obtain a pose estimation sequence includes: Extract the geometric features of the first time-stamped dense point cloud from the dense point cloud sequence including timestamps; Based on the geometric features, the first timestamp dense point cloud is aligned with the preset template point cloud to obtain an initial alignment result; The initial alignment result is optimized by Gaussian mixture to obtain the first pose parameter corresponding to the dense point cloud of the first timestamp. Motion prediction is performed on the dense point cloud sequence including timestamps based on the first pose parameters to obtain a pose estimation sequence.

8. The three-dimensional spatial moving target detection method as described in claim 1, characterized in that, The step of matching moving targets based on the pose estimation sequence to obtain the tracked moving targets includes: The centroid of the moving target is obtained based on the pose estimation sequence, and the Euclidean distance between the centroid of the moving target and the preset centroid trajectory is calculated. The three-dimensional directed bounding box of the moving target is determined based on the pose estimation sequence, and the intersection-union ratio of the three-dimensional directed bounding box and the three-dimensional directed bounding box of the preset bounding box sequence is calculated. Obtain the color space histogram corresponding to the pose estimation sequence, and calculate the single-step similarity between the color space histogram and the preset histogram sequence; A matching cost function is constructed based on the Euclidean distance in the centroid space, the intersection-union ratio of the three-dimensional directed bounding boxes, and the single-step similarity. The matching cost function is used to identify the tracking moving target corresponding to the pose estimation sequence.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The processor is used to execute computer programs stored in the memory; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the three-dimensional moving target detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional spatial moving target detection method according to any one of claims 1 to 8.

Images