Method and apparatus for generating millimeter wave radar data based on video in a mobile scenario

Abstract

The application discloses a kind of methods and devices for generating millimeter wave radar data based on video under mobile scene, belong to millimeter wave radar mobile sensing field, including two components of human body grid extraction and calibration and reflection and noise signal estimation.Human body grid extraction and calibration component is used to accurately estimate the real position of camera and human, including blur and jitter filter, human body grid extraction module, camera pose estimation module and human position calibration module;Reflection and noise signal estimation component is used to simulate the multipath reflection and attenuation of radar signal and background noise, including depth prediction model, reflection model, space-time noise estimation module and U-Net network module.The application generates a large number of realistic and convertible dynamic millimeter wave radar data, including Doppler data and point cloud data, using rich dynamic video data, effectively solves the problem of insufficient millimeter wave radar data in mobile scene, and can be applied in action recognition, target detection and other applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of millimeter wave radar mobile perception, and particularly relates to a method and device for generating millimeter wave radar data based on video in a mobile scenario. BACKGROUND

[0002] Millimeter wave radar as a new and promising perception modality has attracted more and more attention. Millimeter wave radar has good penetration ability to obstacles, has the ability to cope with bad conditions, and at the same time does not expose the user's appearance features, supports robust and privacy-protected human perception.

[0003] Based on these characteristics, there are two typical applications of millimeter wave radar. One is action recognition, for example: (1) using mobile robots to recognize human actions to assist the elderly living alone, (2) allowing drones to follow users according to recognized human actions; the other is target detection, for example: (1) performing emergency rescue tasks according to the personnel detected by mobile robots in a fire, (2) mobile robots detect illegal intruders in buildings at night and perform intelligent alarm operations.

[0004] However, existing dynamic radar datasets are usually customized for specific tasks, and the corresponding scale is small, which makes it difficult to tap the potential of deep learning models and cannot achieve high generalization and robustness. At the same time, collecting and labeling radar data is also a very time-consuming and laborious task, which also limits the large-scale expansion of millimeter wave radar datasets.

[0005] In order to expand the millimeter wave radar dataset, some methods have currently used motion capture data, depth camera data and video data to generate other types of data, such as inertial measurement unit data, sound data and millimeter wave radar data. However, the millimeter wave radar data generated based on motion capture data is usually sparse and rough. The millimeter wave radar data generated based on depth camera data usually lacks some actions in the scene and cannot cover human daily behaviors. The method of using a generative adversarial network to enhance real millimeter wave radar data can easily confuse some similar actions. Recently, a method of generating millimeter wave radar data based on video has appeared, but this method still has the following limitations: (1) the existing method relies on depth maps and cannot accurately estimate the relative position of the person and the camera; at the same time, dynamic video data can appear motion blur and jitter, which further limits the extraction accuracy of the human mesh, resulting in serious false superposition of the generated intensity signals; (2) the existing method fails when there is motion in the background or the sensor itself is moving. Therefore, it is still a challenge to use large-scale dynamic video data to solve the problem of lack of dynamic millimeter wave radar data. SUMMARY

[0006] The present application aims at the problem of lack of millimeter wave radar sensing data in the current mobile scene, and the purpose is to design a method and device for generating millimeter wave radar data based on video in a mobile scene, which generates dynamic millimeter wave radar data in the corresponding scene by taking dynamic video data as input, effectively expands the millimeter wave radar data set, reduces the collection cost of radar data, and supports the further expansion and application of millimeter wave radar in the field of human perception in a mobile scene.

[0007] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0008] In one aspect, the present application provides a method for generating millimeter wave radar data based on video in a mobile scene, comprising the following steps:

[0009] S1, using a blur and jitter filter to filter noise of dynamic video data, based on the processed video, using a human mesh extraction algorithm to extract human mesh data relative to the camera coordinate system;

[0010] S2, using a camera pose estimation algorithm to estimate the camera self-motion state parameters by taking the background and the obtained camera motion true value as input, calibrating the human position based on the camera self-motion state parameters and the local human network data, and outputting the global human mesh data;

[0011] S3, using a depth prediction model to estimate the depth information of the human body in the dynamic video data, based on the depth information of the human body, using the global human mesh data to calculate the radar cross section of the human body vertex, and using the previous frame position of each vertex to calculate the radial velocity of the human body surface; using a reflection model to take the depth information and the radar cross section as input to simulate the multipath reflection and attenuation of the radar signal, and output the intensity map;

[0012] S4, using a spatio-temporal noise estimation algorithm to estimate the reflection signal and background noise of the dynamic video data from the time and space dimensions, outputting the background reflection noise; splicing the intensity map, the background reflection noise and the human body surface radial velocity to obtain the coarse-grained radar data, and using a U-Net network to fit the realistic millimeter wave radar data.

[0013] Further, in step S1, the blur and jitter filter uses a video restoration converter with parallel frame prediction and long-term time series dependence modeling capability to filter video noise, and uses a method combining a neighborhood motion perception local model and a regularization model to solve the optimal distortion transformation of different motion states.

[0014] Further, in step S1, the human mesh data includes the posture and shape of the human body and the relative depth, wherein the posture and shape of the human body are composed of a plurality of human body surface three-dimensional point clouds, and the relative depth describes the distance of each human body from the camera.

[0015] Further, in step S2, the camera pose estimation algorithm eliminates the human in the video, and uses the background and the true value between the continuous frames as input to obtain accurate camera ego-motion state parameters.

[0016] Further, in step S2, the human position calibration process is as follows: based on the camera ego-motion state parameters, the real position of the camera relative to the human is derived; the translation and rotation estimation sequence of the human mesh data represents the camera ego-motion result; then the camera ego-motion parameters and the position calibration are integrated to complete the calibration of the human position in the continuous frames; finally, the human position calibration calibrates the position of the local human mesh data, infers the position information of the human in the real world coordinate system, and generates global human mesh data.

[0017] Further, in step S3, the depth prediction model averages the depth values of different parts of the human body to determine the depth information of the user.

[0018] Further, in step S3, the reflection model adds multiple virtual targets and sets corresponding attenuation coefficients to simulate the real signal attenuation phenomenon.

[0019] Further, in step S4, the space-time noise estimation algorithm estimates the background noise from the time and space dimensions; wherein, for the spatial part, an instance segmentation network is used to segment the background and the human in a single frame image, and then the human in the video is eliminated to retain the background for the following noise simulation; for the time part, an optical flow network is used to obtain the displacement vector between the continuous frames, so as to calculate the speed of any target in the frame; next, the speed is mapped to the segmentation result obtained in the spatial part to obtain the representation form of the background in space-time, so as to simulate the noise characteristics of the background and generate the background reflection noise.

[0020] Further, in step S4, the U-Net network is composed of a feature extraction network and a feature fusion network, wherein the feature extraction network is composed of convolution and downsampling, and the convolution structure used is uniformly a 3x3 convolution kernel.

[0021] On the other hand, the application also provides a device for generating millimeter wave radar data based on video in a mobile scene, which comprises the following components to realize the method for generating millimeter wave radar data based on video in a mobile scene according to any one of the above:

[0022] A blur and jitter filter is used to filter noise from dynamic video data.

[0023] A human mesh extraction module extracts human mesh data relative to the camera coordinate system based on the processed video.

[0024] a camera pose estimation module configured to estimate camera ego-motion state parameters by taking the background and the obtained camera motion ground truth as inputs;

[0025] a human position calibration module configured to calibrate the human position based on the camera ego-motion state parameters and the local human network data, and output global human mesh data;

[0026] a depth prediction model configured to estimate the depth information of the human in the dynamic video data;

[0027] a reflection model configured to calculate the radar cross section of the human vertex by taking the global human mesh data as input based on the depth information of the human, and calculate the radial velocity of the human surface by taking the previous frame position of each vertex, and simulate the multipath reflection and attenuation of the radar signal by taking the depth information and the radar cross section as input, and output an intensity map;

[0028] a space-time noise estimation module configured to estimate the reflection signal and the background noise of the dynamic video data from the time and space dimensions, and output background reflection noise; and splice the intensity map, the background reflection noise and the radial velocity of the human surface to obtain coarse-grained radar data;

[0029] a U-Net network module configured to fit the coarse-grained radar data to realistic millimeter wave radar data.

[0030] Compared with the prior art, the method and device for generating millimeter wave radar data based on video in a mobile scene have the following beneficial effects:

[0031] 1. The method and device for generating millimeter wave radar data based on video in a mobile scene can generate a large amount of realistic and convertible dynamic millimeter wave radar data, including Doppler data and point cloud data, by using rich dynamic video data, thereby effectively solving the problem of insufficient millimeter wave radar data in a mobile scene and being applicable to action recognition, target detection and other applications.

[0032] 2. The camera pose estimation algorithm takes the background and the obtained camera motion ground truth as input to estimate the camera ego-motion, and then calibrates the real position of the human based on the camera ego-motion parameters, and integrates the ego-motion parameters and the position calibration, thereby solving the problem that the relative position of the human between consecutive frames in a dynamic scene will be deviated, and further solving the problems of inaccurate camera ego-motion estimation and human mesh data.

[0033] 3. The depth prediction model is used to obtain the depth information of the human, and the reflection model takes the depth information and the radar cross section as input to simulate the propagation characteristics of the radar signal in the transmission and reception process, thereby solving the problem that it is difficult to simulate the reflection characteristics of the background and the human in a mobile scene. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0035] Figure 1 The system architecture diagram of the method for generating millimeter wave radar data based on video in a mobile scenario provided by the embodiments of the present application. DETAILED DESCRIPTION

[0036] In order to better understand the technical solutions, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described examples are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0037] The method and device for generating millimeter wave radar data based on video in a mobile scenario provided by the present application have a system architecture as shown in Figure 1 The device includes two components: a human mesh extraction and calibration component and a reflection and noise signal estimation component. The human mesh extraction and calibration component is used to accurately estimate the real positions of the camera and the human, and specifically includes a blur and jitter filter, a human mesh extraction module, a camera pose estimation module, and a human position calibration module. The reflection and noise signal estimation component is used to simulate the multipath reflection and attenuation of radar signals and background noise, and specifically includes a depth prediction model, a reflection model, a spatio-temporal noise estimation module, and a U-Net network module.

[0038] The method for generating millimeter wave radar data based on video in a mobile scenario specifically includes the following steps:

[0039] 1. Blur and jitter filtering and human mesh extraction

[0040] This step uses a blur and jitter filter to filter noise from dynamic video data. Based on the processed video, a human mesh extraction module is used to extract human mesh data relative to the camera coordinate system.

[0041] 1) Blur and jitter filtering

[0042] Since the existing method cannot well solve the jitter problem existing in the video without fully utilizing the inter-frame motion, the blur and jitter filter of the present application adopts a video restoration converter with parallel frame prediction and long-term time sequence dependent modeling capability to perform video noise filtering, and simultaneously adopts a method of combining a neighborhood motion perception local model and a regularization model to solve the optimal distortion transformation of different motion states to solve the influence of video jitter.

[0043] 2) human mesh extraction

[0044] Based on the processed video, the present application adopts a latest human mesh extraction module to perform mesh extraction on the human body in the video, and outputs human mesh data relative to the camera coordinate system. The human mesh data includes the posture and shape of the human body and the relative depth, wherein the posture and shape of the human body are composed of a plurality of human surface three-dimensional point clouds, and the relative depth describes the distance of each human from the camera.

[0045] 2) camera pose estimation and human position calibration

[0046] Considering that the relative positions of the human in the dynamic scene between the continuous frames will exist deviation, the camera pose estimation module is used to estimate the camera self-motion state parameters, and then the human position calibration module is used to calibrate the human position. The process of this step is that the camera pose estimation module uses a camera pose estimation algorithm to estimate the camera self-motion state parameters by taking the background and the obtained camera motion true value as input, calibrates the human position based on the camera self-motion state parameters and the local human network data, and outputs the global human mesh data.

[0047] 1) camera pose estimation

[0048] The human mesh data obtained above is not the real position information of the human in the real scene, but the position information in the camera coordinate system. Therefore, the present application designs a camera pose estimation module to estimate the motion state of the camera. The human mesh extraction model can accurately obtain the human mesh, which can provide a true value (including a rotation and translation matrix) for training the camera pose estimation. Since the human has mobility between the continuous frames, it cannot be accurately matched between the frames, which leads to the fact that an optimal camera self-motion parameter cannot be output when training the camera pose estimation. Therefore, the human in the video is removed, and the background and the true value between the continuous frames are used as input to obtain the accurate camera self-motion state parameters.

[0049] The camera pose estimation algorithm removes the human in the video, and uses the background and the true value between the continuous frames as input to obtain the accurate camera self-motion state parameters.

[0050] 2) human position calibration

[0051] The process of human position calibration is: based on the camera self-motion state parameters, the real position of the camera relative to the human is obtained; the translation and rotation estimation sequence table of the human body grid data represents the camera self-motion result; then the camera self-motion parameters and position calibration are integrated to complete the calibration of the human position in the continuous frame; finally, the human position calibration calibrates the position of the local human body grid data, infers the position information of the human in the real world coordinate system, and generates global human body grid data.

[0052] The specific human position calibration formula is as follows:

[0053]

[0054] Wherein, respectively represent the t-th calibration frame and the calibration result, respectively represent the rotation and translation vectors in the current iteration process.

[0055] 3. Depth prediction and reflection signal simulation

[0056] This step uses a depth prediction model to estimate the depth information of the human body in the dynamic video data, based on the depth information of the human body, uses the global human body grid data to calculate the radar cross section of the human body vertex, and calculates the radial velocity of the human body surface through the previous frame position of each vertex; a reflection model is used to simulate the multipath reflection and attenuation of the radar signal using the depth information and the radar cross section as input, and outputs the intensity map.

[0057] 1) Depth prediction

[0058] Given an input video, a depth prediction model is used to obtain the depth information of the human body. The invention does not require the depth of any object other than the human body, and the average value of the depth values of different parts of the human body is used to determine the depth information of the user, which further reduces the error. The actual measurement error is within 5cm.

[0059] 2) Reflection signal simulation

[0060] By obtaining the depth information, the global human body grid data can be used to calculate the cross section of each vertex relative to the radar. The radial velocity is calculated by the previous frame position of each vertex. In order to simulate the multipath reflection and attenuation of the radar signal, the invention designs a reflection model, which takes the depth information and the radar cross section as input to simulate the propagation characteristics of the radar signal during transmission and reception. In order to simulate a more realistic scene, some virtual targets are added when designing the reflection model, such as: walls, tables, floors, etc. In addition, since each reflection has significant energy loss, the invention sets up a corresponding attenuation coefficient to simulate the real signal attenuation phenomenon.

[0061] 4. Noise signal estimation

[0062] This step uses a space-time noise estimation module to estimate the reflection signal and background noise of the dynamic video data from the time and space dimensions, outputs the background reflection noise; splices the intensity map, the background reflection noise and the human body surface radial velocity to obtain coarse-grained radar data; since this data is coarse-grained and has no continuity, a U-Net network module adopts a U-Net network to fit out realistic millimeter wave radar data.

[0063] 1) Space-time noise estimation

[0064] When the background or the sensor is in a state of motion, the background also produces Doppler shift. If the background noise is ignored, it will seriously affect the quality of the generated data. In order to solve this problem, the present application designs a space-time noise estimation module to estimate the background noise from the time and space dimensions. The video can be divided into spatial and temporal two channels. The spatial part represents the scene and target information in the video in the form of a single frame, and the temporal part conveys the motion information of the camera and the target in the form of cross-frame motion.

[0065] For the spatial part, the present application uses an instance segmentation network to segment the background and the human in the single frame image, and then removes the human in the video and retains the background for the subsequent noise simulation.

[0066] For the temporal part, the present application uses an optical flow network to obtain the displacement vector between consecutive frames, so as to calculate the speed of any target in the frame; next, the speed is mapped to the segmentation result obtained by the spatial part to obtain the representation form of the background in space-time, so as to simulate the noise characteristics of the background and generate the background reflection noise.

[0067] The method for obtaining the displacement vector d between consecutive frames using the optical flow network is as follows:

[0068] d i (u, v) represents the displacement vector of the pixel point (u, v) in the i-th frame, which will move to the corresponding pixel point in the i+1-th frame. The horizontal component and the vertical component of the displacement vector field and (high intensity corresponds to positive value, low intensity corresponds to negative value) can be regarded as an image channel. For any frame i, L frames of images after it are collected, and L optical flow frames can be obtained from the L+1 frames, and there are 2L input channels in total as the input of the optical flow network. Assuming that the width and height of a video are w and h respectively, then the input of the optical flow network For any frame i, it can be constructed as follows:

[0069]

[0070]

[0071] For any point (u, v), its corresponding displacement vector on the sequence of L frames can be obtained, so that the speed of any target in the frame can be calculated.

[0072] For the problem of obstacle occlusion in video data, the same strategy is adopted, that is, the occluded part is simulated with noise instead of directly using the reflection signal generated based on the human body grid.

[0073] 2) U-Net network

[0074] Based on the background reflection noise and the radial velocity of the human body surface, rough radar data can be obtained. This data can be converted into Doppler signals by stacking, and in addition, corresponding 3D point cloud data can be generated. Through these ways, the data generated by the application can meet the data needs of various application programs. However, such data is coarse and has no continuity. In order to solve this problem, the application adopts a U-Net framework to generate realistic radar data, which is composed of a feature extraction network and a feature fusion network. The feature extraction network is composed of convolution and down-sampling, and the convolution structure used is uniformly 3x3 convolution kernel. In the training process, the Adam optimizer is used to train 1000 times with a variable learning rate.

[0075] The application designs a millimeter wave radar data generation method for mobile scenarios, which converts nearly infinite dynamic video data into realistic millimeter wave radar data. Experimental results show that the data generated by the application has an average absolute error (MAE) of 0.06 / 0.07 (standard deviation of 0.03 / 0.03) compared with real data in indoor / outdoor scenarios; in terms of action recognition, the model trained using the generated radar data (accuracy of 80.85% / 80.27%) and the model trained using real radar data (accuracy of 87.22% / 86.82%) have similar performance in indoor / outdoor scenarios. If a small amount of real user data is added to the generated radar data set, the recognition accuracy can jump to 94.74% / 94.16% in indoor / outdoor scenarios. Similarly, the target detection task also has similar performance trends, and the model trained using the generated radar data (accuracy of 79.54% / 78.49%) and the model trained using real radar data (accuracy of 86.25% / 85.46%) have similar performance in indoor / outdoor scenarios. Similarly, if a small amount of real user data is used to enhance the generated radar data set, the detection accuracy will jump to 91.88% / 91.34% in indoor / outdoor scenarios.

[0076] The above merely describes the preferred embodiments and principles of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application by those skilled in the art according to the provided ideas should be considered as falling within the protection scope of the present application.

Claims

1. A method for generating millimeter wave radar data based on video in a mobile scenario, characterized in that, The method comprises the following steps: S1, noise filtering of dynamic video data is performed using a blur and jitter filter, based on the processed video, human mesh data relative to the camera coordinate system is extracted using a human mesh extraction algorithm; S2, camera pose estimation algorithm is used to estimate camera ego-motion state parameters by taking the background and the obtained camera motion true value as input, human position calibration is performed based on the camera ego-motion state parameters and the local human network data, and global human mesh data is output; S3, a depth prediction model is used to estimate the depth information of the human body in the dynamic video data, based on the depth information of the human body, the radar cross section of the human body vertex is calculated using the global human mesh data, the radial velocity of the human body surface is calculated through the previous frame position of each vertex, and the depth information and the radar cross section are taken as input to simulate the multipath reflection and attenuation of the radar signal using a reflection model, and an intensity map is output; S4, a space-time noise estimation algorithm is used to estimate the reflection signal and background noise of the dynamic video data from the time and space dimensions, and background reflection noise is output; The intensity map, the background reflection noise and the human body surface radial velocity are spliced to obtain coarse-grained radar data, and a U-Net network is used to fit realistic millimeter wave radar data; the space-time noise estimation algorithm estimates the background noise from the time and space dimensions; wherein, for the spatial part, an instance segmentation network is used to segment the background and the human in a single frame image, and then the human in the video is removed and the background is retained for the subsequent noise simulation; for the time part, an optical flow network is used to obtain the displacement vector between the continuous frames, so as to calculate the speed of any target in the frame; Next, the speed is mapped to the segmentation result obtained from the spatial part to obtain the background representation in space-time, so as to simulate the noise characteristics of the background and generate the background reflection noise.

2. The method for generating millimeter wave radar data based on video in a mobile scenario according to claim 1, characterized in that, In step S1, the blur and jitter filter uses a video restoration converter with parallel frame prediction and long-term temporal dependence modeling capability to perform video noise filtering, and a method combining a neighborhood motion-aware local model and a regularization model is used to solve the optimal distortion transformation of different motion states.

3. The method for generating millimeter wave radar data based on video in a mobile scenario according to claim 1, characterized in that, In step S1, the human mesh data includes the posture and shape of the human body and the relative depth, wherein the posture and shape of the human body are composed of a plurality of human body surface three-dimensional point clouds, and the relative depth describes the distance of each human from the camera.

4. The method for generating millimeter wave radar data based on video in a mobile scenario according to claim 1, characterized in that, In step S2, the camera pose estimation algorithm removes the human in the video, and uses the background and the true value between the continuous frames as input to obtain accurate camera ego-motion state parameters.

5. The method for generating millimeter wave radar data based on video in mobile scenarios according to claim 1, characterized in that, In step S2, the process of human position calibration is as follows: based on the camera ego-motion state parameters, the real position of the camera relative to the human is obtained; the translation and rotation estimation sequence of the human mesh data represents the camera ego-motion result; then the camera ego-motion parameters and the position calibration are integrated to complete the calibration of the human position in the continuous frames; finally, the human position calibration calibrates the position of the local human mesh data, infers the position information of the human in the real world coordinate system, and generates global human mesh data.

6. The method for generating millimeter wave radar data based on video in mobile scenarios according to claim 1, characterized in that, In step S3, the depth prediction model averages the depth values of different parts of the human body to determine the depth information of the user.

7. The method for generating millimeter wave radar data based on video in mobile scenarios according to claim 1, characterized in that, In step S3, the reflection model adds multiple virtual targets and sets corresponding attenuation coefficients to simulate the real signal attenuation phenomenon.

8. The method for generating millimeter wave radar data based on video in mobile scenarios according to claim 1, characterized in that, In step S4, the U-Net network is composed of a feature extraction network and a feature fusion network, wherein the feature extraction network is composed of convolution and down-sampling, and the convolution structure used is uniformly a 3×3 convolution kernel.

9. An apparatus for generating millimeter wave radar data based on video in a mobile scenario, the apparatus comprising: The following modules are included to implement the method for generating millimeter wave radar data based on video in a mobile scenario according to any one of claims 1-8: a blur and jitter filter for noise filtering of dynamic video data; a human body grid extraction module for extracting human body grid data relative to a camera coordinate system based on the processed video; a camera pose estimation module for estimating camera ego-motion state parameters by taking the background and obtained camera motion ground truth as input; a human body position calibration module for calibrating human body positions based on camera ego-motion state parameters and local human body network data, and outputting global human body grid data; a depth prediction model for estimating the depth information of the human body in the dynamic video data; a reflection model for calculating the radar cross section of the human body vertex based on the depth information of the human body, and calculating the radial velocity of the human body surface through the previous frame position of each vertex; taking the depth information and the radar cross section as input to simulate the multipath reflection and attenuation of the radar signal, and outputting an intensity map; a space-time noise estimation module for estimating the reflection signal and background noise of the dynamic video data from the time and space dimensions, and outputting background reflection noise; splicing the intensity map, the background reflection noise and the radial velocity of the human body surface to obtain coarse-grained radar data; a U-Net network module for fitting realistic millimeter wave radar data from the coarse-grained radar data.

Images