A millimeter wave radar video fusion method based on micro-Doppler effect

Abstract

The present invention discloses a millimeter wave radar video fusion method based on micro-Doppler effect, which specifically relates to the field of multi-sensor fusion technology, and includes the following steps: step 10: collecting echo signals; step 20: signal classification; step 30: extracting feature vectors; step 40: scatter sequence sorting; step 50: posture recognition of convolutional neural network; step 60: building a posture recognition system; step 70: generating target information. The present invention combines it with human posture recognition of video surveillance, so as to achieve matching fusion of millimeter wave radar and video camera screen through the association matching of human posture, thereby greatly improving the matching efficiency of millimeter wave radar and video, and automatically filtering false alarms caused by animals.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor fusion, and more specifically, to a millimeter wave radar video fusion method based on micro-Doppler effect. Background Art

[0002] At present, through the analysis of the micro-Doppler effect of millimeter-wave radar, it is possible to recognize the posture of human movement, mainly by extracting the characteristics of millimeter-wave echo signals and performing neural network recognition algorithms; similarly, video cameras currently have mature technology, which can obtain images or video data of human movement through cameras, and obtain characteristic data of human movement by processing the images as the basis for human posture recognition.

[0003] In the perimeter warning system, the combination of millimeter-wave radar and video surveillance can greatly improve the warning capability of the perimeter warning system. Current research mainly associates radar with video images through the movement trajectory of pedestrians. However, it is easy to make mistakes in the target association problem under complex conditions such as dense distribution of targets and cross-motion. To this end, we propose a millimeter-wave radar video fusion method based on the micro-Doppler effect to solve the above problems.

[0004] Invention content

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a millimeter-wave radar video fusion method based on the micro-Doppler effect, which realizes the matching fusion of the millimeter-wave radar and the video camera screen through the correlation matching of the human body posture, which can greatly improve the matching efficiency of the millimeter-wave radar and the video, so as to solve the problems raised in the above-mentioned background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A millimeter wave radar video fusion method based on micro-Doppler effect specifically includes the following steps:

[0008] Step 10: Collect echo signals. First, set up millimeter-wave radar and video surveillance cameras in an open area to collect human echo signals. During the signal collection process, minimize the impact of the environment. Each action is sampled at least 50 times.

[0009] Step 20: signal classification, classifying the human motion echo signal in step 10 to generate a plurality of different motion echo signals, each different echo signal is represented by a different original waveform, and the original waveforms of any two actions are compared to distinguish different human postures;

[0010] Step 30: Extracting feature vectors, performing signal processing on different original waveforms generated in step 20, and extracting feature vectors in the signals;

[0011] Step 40: Sorting the scattered point sequence, corresponding different parts of the human body to different scattered points, and recording the distance between the scattered points of each part of the human body and the radar at the same time, generating a human body scattered point sequence, recording at different time points, generating multiple scattered point sequences that are different over time, and generating a scattered point sequence that is sorted over time.

[0012] Step 50: posture recognition of a convolutional neural network, generating multi-layer target features of a human body posture through convolution calculation of a neural network and a multi-layer network structure; recognizing the generated target features, and generating a human body posture corresponding to the target features;

[0013] Step 60: Building a posture recognition system. According to the millimeter wave radar human posture recognition theory generated in step 50, building a posture recognition system;

[0014] Step 70: Generate target information. When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously and transmit the generated data to the human posture recognition system, which is finally integrated with the video information of the surveillance camera and then associated with the surveillance image to obtain detailed information of the target.

[0015] In a preferred embodiment, in step 20, different human body postures are distinguished by analyzing the amplitude changes of the original waveform of the human body movement.

[0016] In a preferred embodiment, the specific construction steps of step 50 are as follows:

[0017] Step 501: Collect human body movements through the millimeter wave radar installed in step 10 to generate raw data;

[0018] Step 502: sorting the original data generated in step 501 to generate serialized original data;

[0019] Step 503: extracting a data set corresponding to the human body posture features corresponding to the original data generated in step 502;

[0020] Step 504: Divide the data set generated in step 503 into two parts, one of which is used as a training set and the other is used as a test set;

[0021] Step 505: Perform deep learning on the training set data generated in step 504 using a convolutional neural network; identify the characteristics of human body posture and judge the training quality on the test set data generated in step 504;

[0022] Step 506: modify the feature weight according to the result of human body posture feature recognition in step 505.

[0023] In a preferred embodiment, the classification method of the training set and the test set in step 504 is as follows: the measured millimeter wave radar data is classified according to a ratio of 3:1, namely, a training set and a test set.

[0024] In a preferred embodiment, the loss function is used to judge the training quality in step 505, and the judgment method is as follows: if the loss function converges to a value lower than 0.0006, the training is successful and the learning process ends; otherwise, the weights are modified and the next judgment is performed.

[0025] In a preferred embodiment, the specific operation steps of step 70 are as follows:

[0026] Step 701: When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously, the millimeter wave radar collects data and analyzes the target's distance, speed, direction and other information;

[0027] Step 702: The human body posture recognition system generated in step 60 receives the measurement data generated by the millimeter wave radar, and generates the posture and threat information of the target through system identification.

[0028] Step 703: The information such as the posture and threat of the target generated in step 702 is integrated with the video information generated by the surveillance camera in step 701, and the information is associated with the surveillance image to generate detailed information of the target.

[0029] Technical effects and advantages of the present invention:

[0030] 1. The present invention proposes a neural network human posture recognition based on the micro-Doppler effect of millimeter-wave radar, which is combined with the human posture recognition of video surveillance, so as to achieve the matching fusion of millimeter-wave radar and video camera images through the correlation matching of human posture, thereby greatly improving the matching efficiency of millimeter-wave radar and video, and automatically filtering out false alarms caused by animals;

[0031] 2. The present invention solves many drawbacks of the current multi-sensor fusion technology using target tracks, which are listed as follows: 1. Target association problems in complex situations such as dense distribution and cross-movement of targets in the defense zone; 2. Low utilization rate of the target's dynamic information or position information; 3. The problem that the association accuracy cannot be improved or optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1This is a flow chart of step 50 in the millimeter wave radar video fusion method based on micro-Doppler effect proposed by the present invention.

[0033] Figure 2 This is a schematic diagram of the principle of millimeter wave radar video monitoring in the millimeter wave radar video fusion method based on micro-Doppler effect proposed in the present invention.

[0034] Figure 3 This is a schematic diagram of the neural network construction in the millimeter wave radar video fusion method based on micro-Doppler effect proposed in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Embodiment 1:

[0038] As attached Figure 1-3 A millimeter wave radar video fusion method based on micro-Doppler effect is shown, which specifically includes the following steps:

[0039] Step 10: Collect echo signals. First, set up millimeter-wave radar and video surveillance cameras in an open area to collect human echo signals. During the signal collection process, minimize the impact of the environment. Each action is sampled at least 50 times.

[0040] Step 20: signal classification, classifying the human motion echo signal in step 10 to generate a plurality of different motion echo signals, each different echo signal is represented by a different original waveform, and the original waveforms of any two actions are compared to distinguish different human postures;

[0041] Step 30: Extracting feature vectors, performing signal processing on different original waveforms generated in step 20, and extracting feature vectors in the signals;

[0042] Step 40: Sorting the scattered point sequence, corresponding different parts of the human body to different scattered points, and recording the distance between the scattered points of each part of the human body and the radar at the same time, generating a human body scattered point sequence, recording at different time points, generating multiple scattered point sequences that are different over time, and generating a scattered point sequence that is sorted over time, so that not only the distance information between the scattered points of each part of the human body and the radar can be obtained, but also the change of the scattered point sequence in the change of human posture can be obtained. This can not only improve the accuracy of human posture recognition, but also recognize more complex postures of the human body and improve the ability of human posture recognition.

[0043] Step 50: Convolutional neural network posture recognition, generating multi-layer target features of human posture through convolution calculation of neural network and multi-layer network structure; identifying the generated target features, generating human posture corresponding to the target features, compared with traditional human posture feature extraction, no manual feature extraction is required, but machine autonomous learning, which can greatly improve the feature extraction efficiency, and also greatly improve the accuracy of human posture recognition;

[0044] Step 60: Building a posture recognition system. According to the millimeter wave radar human posture recognition theory generated in step 50, building a posture recognition system;

[0045] Step 70: Generate target information. When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously and transmit the generated data to the human posture recognition system, which is finally integrated with the video information of the surveillance camera and then associated with the surveillance image to obtain detailed information of the target.

[0046] Furthermore, in step 20, different human body postures are distinguished by analyzing the amplitude changes of the original waveform of the human body movement.

[0047] Reference Figure 1 and Figure 3 The specific construction steps of step 50 are as follows:

[0048] Step 501: Collect human body movements through the millimeter wave radar installed in step 10 to generate raw data;

[0049] Step 502: sorting the original data generated in step 501 to generate serialized original data;

[0050] Step 503: extracting a data set corresponding to the human body posture features corresponding to the original data generated in step 502;

[0051] Step 504: Divide the data set generated in step 503 into two parts, one of which is used as a training set and the other is used as a test set;

[0052] Step 505: Perform deep learning on the training set data generated in step 504 using a convolutional neural network; identify the characteristics of human body posture and judge the training quality on the test set data generated in step 504;

[0053] Step 506: modify the feature weight according to the result of human body posture feature recognition in step 505.

[0054] Specifically, the classification method of the training set and the test set in step 504 is as follows: the measured millimeter wave radar data is classified according to a ratio of 3:1, namely, a training set and a test set.

[0055] Furthermore, in step 505, the loss function is used to judge the training quality, and the judgment method is as follows: if the loss function converges to a value lower than 0.0006, the training is successful and the learning process ends; otherwise, the weights are modified and the next judgment is performed.

[0056] It is worth noting that the specific operation steps of step 70 are as follows:

[0057] Step 701: When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously, the millimeter wave radar collects data and analyzes the target's distance, speed, direction and other information;

[0058] Step 702: The human body posture recognition system generated in step 60 receives the measurement data generated by the millimeter wave radar, and generates the posture and threat information of the target through system identification.

[0059] Step 703: The information such as the posture and threat of the target generated in step 702 is integrated with the video information generated by the surveillance camera in step 701, and the information is associated with the surveillance image to generate detailed information of the target.

[0060] Embodiment 2:

[0061] Combined with Figure 2 The specific implementation steps of this embodiment are described as follows:

[0062] Step 10: Set up a 4-meter monitoring pole in the center of the fenced area or in a location with a wide field of view. Install a video surveillance system on the top of the pole and a millimeter-wave radar directly below it.

[0063] Step 20. Turn on the millimeter-wave radar and video surveillance and start recording data. In the initial state, we need to train the neural network and train personnel to perform various postures in the perimeter defense area. The millimeter-wave radar and video surveillance record these data.

[0064] Step 30: Match the data collected by the millimeter-wave radar with the posture one by one, and start training the BP neural network using Matlab, with 70% of the samples as the training set, 15% as the validation set, and 15% as the test set.

[0065] Step 40, start training. After the training is completed, we observe the training effect. In the detailed analysis of the training process, the neural network will iterate and use the gradient descent method to correct the weights. After multiple iterations of the neural network, the neural network will converge, that is, the training is completed.

[0066] Step 50. We need to save the trained neural network for next use without having to train it again. When using it, you only need to download the trained neural network and import the measurement data. The neural network will give the result, that is, the human body posture, based on the previous training.

[0067] Step 60: For the video signal input by the surveillance camera, we also use the neural network to recognize the human posture, but combine the two parts, the spatial part describing the stillness and the temporal part describing the human movement, to form a two-stream convolutional neural network for human posture recognition. This method can improve the success rate of human posture recognition. However, it will also increase the difficulty and time length of neural network training.

[0068] Step 70, step 50 and step 60 realize the process of human body posture recognition by millimeter wave radar and video camera. Finally, we associate the targets with the same posture and display the precise target distance, direction, speed, posture and other information measured by the millimeter wave radar in the video screen. The security personnel can obtain enough information about the target according to the screen display.

[0069] Finally: The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A millimeter wave radar video fusion method based on micro-Doppler effect, characterized by: The specific steps include: Step 10: Collect echo signals. First, set up millimeter-wave radar and video surveillance cameras in an open area to collect human echo signals. During the signal collection process, minimize the impact of the environment. Each action is sampled at least 50 times. Step 20: signal classification, classifying the human motion echo signal in step 10 to generate a plurality of different motion echo signals, each different echo signal is represented by a different original waveform, and the original waveforms of any two actions are compared to distinguish different human postures; Step 30: Extracting feature vectors, performing signal processing on different original waveforms generated in step 20, and extracting feature vectors in the signals; Step 40: Sorting the scattered point sequence, corresponding different parts of the human body to different scattered points, and recording the distance between the scattered points of various parts of the human body and the radar at the same time, generating a human body scattered point sequence, recording at different time points, generating multiple scattered point sequences that are different over time, and generating a scattered point sequence that is sorted over time; Step 50: posture recognition of convolutional neural network, generating multi-layer target features of human posture through convolution calculation of neural network and multi-layer network structure; Identify the generated target features and generate a human body posture corresponding to the target features; Step 60: Building a posture recognition system. According to the millimeter wave radar human posture recognition theory generated in step 50, building a posture recognition system; Step 70: Generate target information. When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously and transmit the generated data to the human posture recognition system, which is finally integrated with the video information of the surveillance camera and then associated with the surveillance image to obtain detailed information of the target.

2. The millimeter wave radar video fusion method based on micro-Doppler effect according to claim 1, characterized in that: In step 20, different human body postures are distinguished by analyzing the amplitude changes of the original waveform of the human body movement.

3. The millimeter wave radar video fusion method based on micro-Doppler effect according to claim 1, characterized in that: The specific construction steps of step 50 are as follows: Step 501: Collect human body movements through the millimeter wave radar installed in step 10 to generate raw data; Step 502: sorting the original data generated in step 501 to generate serialized original data; Step 503: extracting a data set corresponding to the human body posture features corresponding to the original data generated in step 502; Step 504: Divide the data set generated in step 503 into two parts, one of which is used as a training set and the other is used as a test set; Step 505: Perform deep learning on the training set data generated in step 504 using a convolutional neural network; For the test set data generated in step 504, the characteristics of human posture are identified and the training quality is judged; Step 506: modify the feature weight according to the result of human body posture feature recognition in step 505.

4. The millimeter wave radar video fusion method based on micro-Doppler effect according to claim 3 is characterized in that: The classification method of the training set and the test set in step 504 is as follows: the measured millimeter wave radar data is classified according to a ratio of 3:1, and they are respectively a training set and a test set.

5. The millimeter wave radar video fusion method based on micro-Doppler effect according to claim 4 is characterized in that: In step 505, the loss function is used to judge the training quality, and the judgment method is as follows: if the loss function converges to a value lower than 0.0006, the training is successful and the learning process ends; otherwise, the weights are modified and the next judgment is performed.

6. The millimeter wave radar video fusion method based on micro-Doppler effect according to claim 1, characterized in that: The specific operation steps of step 70 are as follows: Step 701: When the target enters the warning range of the perimeter warning system, the millimeter wave radar and the surveillance camera work simultaneously, the millimeter wave radar collects data and analyzes the distance, speed, and direction information of the target; Step 702: The human posture recognition system generated in step 60 receives the measurement data generated by the millimeter wave radar, and generates the posture and threat information of the target through system judgment; Step 703: The posture and threat information of the target generated in step 702 are integrated with the video information generated by the surveillance camera in step 701, and this information is associated with the surveillance image to generate detailed information of the target.

Images