A device and method for fusing radar imagery with video images

Abstract

The application discloses a device and method for fusing radar imaging and video images, and relates to the field of multi-sensing data fusion. The correction device in the device comprises at least three rotating structures; all the rotating structures can enclose a closed area; the rotating structure comprises: two blades of spherical structures; a video shooting device shoots the rotating structure rotating in a target monitoring area to obtain a video image; a detection radar emits a radar signal to the rotating structure rotating in the target monitoring area and receives a reflected radar echo signal; a processor filters the video image and the radar echo signal by using an adaptive subtraction algorithm, extracts the center coordinates of each rotating structure in the filtered target image and target signal by using a Hough transformation algorithm, obtains video feature points and radar feature points, and matches the video feature points and the radar feature points to realize the registration of radar imaging and video images. The application can efficiently and accurately realize the registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-sensor data fusion, in particular to a device and method for fusing radar imaging and video image. BACKGROUND

[0002] At present, for outdoor monitoring, especially in the fields of border, coastal defense, etc., both optical camera and microwave radar must be used. Optical camera and millimeter wave radar have their own advantages and disadvantages in monitoring. Optical imaging is intuitive and easy to identify, but it is greatly affected by outdoor environment, and the imaging quality is unstable. Outdoor environment is changeable, such as light, temperature, humidity, rain, fog, etc. Microwave radar imaging makes up for the deficiency of optical imaging, and can work at any time and in any climate, and is not affected by changes in outdoor environment, and the imaging quality is stable. But radar imaging is an electromagnetic wave image representation, which is not intuitive and not easy to identify targets.

[0003] When optical camera and microwave radar are used at the same time, the monitoring picture of optical camera and the imaging picture of microwave radar do not completely coincide, and there is a certain deviation, which causes the judgment accuracy of the monitoring system to decrease, and even the subsequent monitoring algorithm to be wrong. Therefore, appropriate devices and methods must be used to effectively and accurately fuse video image and millimeter wave radar imaging together.

[0004] At present, the methods for fusing video image and millimeter wave radar imaging mainly include the following:

[0005] (1) Using pure image registration algorithm. The principle of this method is: first, install the camera and the millimeter wave radar, and their directions are basically consistent. Second, take an image and perform millimeter wave radar imaging on a certain place. Then, extract feature points in the image and radar imaging respectively. Finally, use image registration algorithm for calibration. This method extracts n feature points from the camera and m feature points from the millimeter wave radar imaging, usually n and m are not equal, and there is a problem of feature point selection in image registration, in many cases, the feature points are still selected manually, and the registration algorithm is relatively complex.

[0006] (2) Using a registration algorithm with the help of a correction device. For example, patent CN113239948B uses a white stick to achieve this. The correction device of this patent includes a white stick and a black screen. The characteristics of this correction device are that the stick is static, and the feature reflected by the stick is a straight line. Therefore, for the camera, the video captured is static; for the millimeter wave radar, the reflected radar echo signal is also static, and the millimeter wave radar is also static. For the image captured by the camera, the extracted feature is a straight line, and the algorithm used is a straight line extraction algorithm; for millimeter wave radar imaging, the extracted feature is a straight line, and the algorithm used is a straight line extraction algorithm. At the same time, the preprocessing algorithm for the radar echo signal is the filtering and enhancement of static echo signals. In addition to the white stick, this patent also needs a black curtain in the correction device, and the device is relatively complex. In terms of millimeter wave radar imaging and feature line extraction, the algorithm for filtering static echo signals is more complex and less accurate. SUMMARY

[0007] Based on this, the embodiments of the present application provide a device and method for fusing radar imaging and video images to reduce the complexity of calculation and improve the accuracy of calculation, thereby efficiently and accurately realizing registration.

[0008] To achieve the above object, the embodiments of the present application provide the following solutions:

[0009] A device for fusing radar imaging and video images, comprising: a correction device, a detection device and a processor; the correction device is arranged in a target monitoring area; the detection device is connected with the processor;

[0010] The correction device comprises at least three rotating structures; all the rotating structures can enclose a closed area; the rotating structure comprises two blades; the blade is a spherical structure;

[0011] The detection device comprises a video shooting device and a detection radar;

[0012] The video shooting device is used for:

[0013] Photographing the rotating structure rotating in the target monitoring area to obtain a video image;

[0014] The detection radar is used for:

[0015] Transmitting a radar signal to the rotating structure rotating in the target monitoring area and receiving a radar echo signal reflected by the rotating structure;

[0016] The processor is used for:

[0017] The video image is filtered by using an adaptive subtraction algorithm to obtain a target image, and a Hough transform algorithm is used to extract the center coordinates of each rotating structure in the target image to obtain video feature points;

[0018] The radar echo signal is filtered by using an adaptive subtraction algorithm to obtain a target signal, and a radar echo image is generated according to the target signal, and a Hough transform algorithm is used to extract the center coordinates of each rotating structure in the radar echo image to obtain radar feature points;

[0019] The video feature points and the radar feature points are matched, and the positions of the video shooting device and the detection radar are adjusted according to the matching result to realize registration of the video image and the radar echo signal.

[0020] Optionally, the color of the blade is white.

[0021] Optionally, the rotating structure further comprises a rotating motor.

[0022] The rotating motor is used to drive the two blades to rotate.

[0023] Optionally, the correction device comprises four rotating structures, and the four rotating structures can enclose a closed square area.

[0024] Optionally, the processor, in terms of generating a radar echo image according to the target signal, is specifically configured to:

[0025] The radar echo image is generated according to the target signal by using a time-frequency analysis method.

[0026] Optionally, the processor, in terms of matching the video feature points and the radar feature points, adjusting the positions of the video shooting device and the detection radar according to the matching result, and realizing registration of the video image and the radar echo signal, is specifically configured to:

[0027] A manual correction result is obtained, the manual correction result is obtained by adjusting the positions of the video shooting device and the detection radar by using a manual method and taking coincidence of the video feature points and the radar feature points as a target, and coincidence degrees of the video feature points and the radar feature points in the manual correction result reach a set coarse registration coincidence degree;

[0028] The video feature points and the radar feature points in the manual correction result are re-registered by using an image registration algorithm, so that the video feature points and the radar feature points are completely coincident, and a mapping relationship between the video feature points and the radar feature points is determined;

[0029] The mapping relationship is used to adjust the positions of the video shooting device and the detection radar again to realize registration of the video image and the radar echo signal.

[0030] Optionally, the device for fusing radar imaging and video image further comprises a display; the display is used to display the closed area surrounded by the video feature points and the closed area surrounded by the radar feature points.

[0031] The application further provides a method for fusing radar imaging and video image, comprising:

[0032] acquiring a video image and a radar echo signal; wherein the video image is obtained by shooting a rotating structure rotating in a target monitoring area; the radar echo signal is a radar echo signal reflected by a rotating structure after radar signals are emitted to the rotating structure rotating in the target monitoring area; the rotating structure is at least three; all the rotating structures can surround a closed area; the rotating structure comprises two blades; the blade is a spherical structure;

[0033] an adaptive subtraction algorithm is used to filter the video image to obtain a target image, and a Hough transform algorithm is used to extract the center coordinates of each rotating structure in the target image to obtain video feature points;

[0034] an adaptive subtraction algorithm is used to filter the radar echo signal to obtain a target signal, and a radar echo image is generated according to the target signal, and a Hough transform algorithm is used to extract the center coordinates of each rotating structure in the radar echo image to obtain radar feature points;

[0035] the video feature points and the radar feature points are matched, and the positions of the video shooting device and the detection radar are adjusted according to the matching result to realize registration of the video image and the radar echo signal.

[0036] Optionally, the video feature points and the radar feature points are matched, and the positions of the video shooting device and the detection radar are adjusted according to the matching result to realize registration of the video image and the radar echo signal, and the method specifically comprises:

[0037] a manual correction result is acquired; the manual correction result is obtained by adjusting the positions of the video shooting device and the detection radar in a manual manner with the video feature points and the radar feature points coinciding as a target; the coincidence degree of the video feature points and the radar feature points in the manual correction result reaches a set coarse registration coincidence degree;

[0038] The image registration algorithm is used for re-registering the video feature points and the radar feature points in the manual correction result, so that the video feature points and the radar feature points are completely overlapped, and a mapping relationship between the video feature points and the radar feature points is determined.

[0039] The mapping relationship is used for re-adjusting positions of the video shooting device and the detection radar, so that registration of the video image and the radar echo signal is realized.

[0040] Optionally, a radar echo image is generated according to the target signal, and specifically includes:

[0041] According to the target signal, a radar echo image is generated by using a time-frequency analysis method.

[0042] According to the specific embodiments provided in the present application, the following technical effects are disclosed:

[0043] The leaf of the correction device in the device is rotatable, which can provide dynamic video images and dynamic radar echo signals, so that an adaptive background subtraction algorithm can be used for filtering, the algorithm is efficient and accurate; the leaf of the correction device is spherical, which can reflect millimeter wave radar signals at different angles, and simplifies the installation requirements; the Hough transformation algorithm is used to extract feature points, and the accuracy is high. Therefore, the embodiment of the present application can reduce the complexity of calculation in the fusion of radar imaging and video images, improve the accuracy of calculation, and efficiently and accurately realize registration. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The structural diagram of the device for fusing radar imaging and video images provided by the embodiment of the present application is shown in the figure;

[0046] Figure 2 The schematic diagram of the gradient method for detecting the center of the circle provided by the embodiment of the present application is shown in the figure;

[0047] Figure 3 The structural diagram of the rotating structure provided by the embodiment of the present application is shown in the figure;

[0048] Figure 4 The position schematic diagram of the four rotating structures provided by the embodiment of the present application is shown in the figure;

[0049] Figure 5 The schematic diagram of the first set of feature points provided for the embodiment of the present application;

[0050] Figure 6 The schematic diagram of the second set of feature points provided for the embodiment of the present application;

[0051] Figure 7 The schematic diagram of the correction result of the manual correction provided for the embodiment of the present application;

[0052] Figure 8 The schematic diagram of the result of the fine registration provided for the embodiment of the present application.

[0053] Symbol explanation:

[0054] Video shooting device - 1, detection radar - 2, detection device - 3, processor - 4, detection support - 5, blade - 6, correction support - 7. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0056] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0057] Embodiment one

[0058] Referring to Figure 1 The radar imaging and video image fusion device of the present embodiment comprises a correction device, a detection device 3 and a processor 4; the correction device is arranged in a target monitoring area; the detection device 3 is connected with the processor 4.

[0059] The correction device comprises at least three rotating structures; all the rotating structures can enclose a closed area. Referring to Figure 2 The rotating structure comprises two blades 6; the blade 6 is a spherical structure.

[0060] The detection device 3 comprises a video shooting device 1 and a detection radar 2. The direction of the detection radar 2 is shown as an arrow a, and the direction of the video shooting device 1 is shown as an arrow b.

[0061] The video shooting device 1 is used for shooting the rotating structure rotating in the target monitoring area to obtain a video image.

[0062] The detection radar 2 is configured to: emit radar signals to a rotating structure rotating in the target monitoring area, and receive radar echo signals reflected by the rotating structure.

[0063] The processor 4 is configured to: filter the video image by using an adaptive subtraction algorithm to obtain a target image, and extract the center coordinates of each rotating structure in the target image by using a Hough transform algorithm to obtain video feature points; filter the radar echo signal by using the adaptive subtraction algorithm to obtain a target signal, and generate a radar echo image according to the target signal, extract the center coordinates of each rotating structure in the radar echo image by using the Hough transform algorithm to obtain radar feature points; and match the video feature points and the radar feature points, and adjust the positions of the video shooting device 1 and the detection radar 2 according to a matching result, so as to realize registration of the video image and the radar echo signal.

[0064] The filtering of the video image by using the adaptive subtraction algorithm specifically includes:

[0065] (1) Extract a video frame image.

[0066] The video image is composed of a series of continuously shot images. The video frame image is an image or some images extracted from the video image.

[0067] In this step, the video frame image at time t-1 is extracted as a previous frame image, and the video frame image at time t is extracted as a current frame image according to the video time sequence.

[0068] (2) Perform frame image difference operation, that is, subtract the video frame images at adjacent two time points.

[0069] Subtract the current frame image from the previous frame image, that is, perform image difference operation, to obtain a video frame image difference result. Since the camera and the millimeter wave sensor do not move at time t and time t-1, the same targets are shot or imaged at the same place, and the time difference between time t and time t-1 is very small, therefore, the current frame image and the previous frame image are mostly the same, and only the images of the part of the rotating detection device are different. The result of the image difference operation cancels most of the image area, and only the images of the part of the rotating detection device are reserved.

[0070] (3) Video frame image difference result enhancement processing.

[0071] The result after the video frame image difference operation is the dynamic change part of the image. Since the time difference between time t and time t-1 is very small, the value of the dynamic change part is also very small. Through image enhancement processing, the small change is enlarged, so that the difference value of the image is larger and more obvious. The image after the enhancement processing is used as the target image at time t.

[0072] In the embodiment, the gradient method is used to detect the center of the circle. In the embodiment, the Hough transform algorithm is used to extract the center coordinates of each rotating structure in the target image, specifically including:

[0073] (1) Perform Canny edge detection on the target image to obtain a binary image of edge detection.

[0074] (2) Perform a Sobel operator on the target image to calculate the neighborhood gradient value of all pixels.

[0075] (3) Initialize the circle center space N(a, b), and set all N(a, b) = 0.

[0076] (4) Traverse all non-zero pixel points in the binary image of Canny edge detection, draw a line along the gradient direction (the perpendicular direction of the tangent), and add 1 to N(a, b) of all points (a, b) in the accumulator that the line segment passes through.

[0077] (5) Sort N(a, b), and the maximum value in the sequence corresponds to the circle center, thereby obtaining the center coordinates.

[0078] For example, for the target image shown in part (a) of Figure 2 , the target image includes a circle, a triangle, and a rectangle. After Hough transform, the larger the value in the circle center space, the whiter it is, and the smaller the value, the darker it is. The whitest point in part (b) of Figure 2 corresponds to the circle center. The center coordinates of each image are the center coordinates of the image.

[0079] In one example, the color of the blade 6 can be white to increase the light reflection performance of the blade 6, facilitating night installation operation.

[0080] In one example, the detection radar 2 can be a millimeter wave detection radar; the video shooting device 1 can be a video camera; and the processor 4 can be a computer.

[0081] In one example, the rotating structure further includes a rotating motor, and the rotating motor is used to drive the two blades 6 to rotate.

[0082] In one example, the rotating structure further includes a correction support 7, and the two blades 6 and the rotating motor are mounted on the rotating support, and the blade 6 is rotatably connected to the correction support 7 through a connecting piece, which can be a connecting rod. For example, the upper and lower ends of the blade 6 are rotatably connected to the top end of the correction support 7 through a connecting rod, as shown in Figure 3 , and the arrow in Figure 3 indicates the rotation direction.

[0083] In one example, still referring to Figure 1 , the radar imaging and video image fusion device further comprises a display; the display is used to display the closed area surrounded by the video feature points and the closed area surrounded by the radar feature points.

[0084] In one example, still referring to Figure 1 , the detection device 3 further comprises a detection bracket 5; the detection bracket 5 is used to place the video shooting device 1 and the detection radar 2. For example, the video shooting device 1 and the detection radar 2 are installed by adjustable bolts, and after installation, the direction of the detection radar 2 and the video shooting device 1 can be fine-tuned to ensure that the detection areas of the two are highly overlapped.

[0085] In one example, the processor 4, in terms of generating a radar echo image according to the target signal, is specifically configured to: generate a radar echo image according to the target signal by using a time-frequency analysis method.

[0086] In one example, the processor 4, in terms of matching the video feature points and the radar feature points and adjusting the positions of the video shooting device 1 and the detection radar 2 according to the matching result to realize the registration of the video image and the radar echo signal, is specifically configured to:

[0087] (1) Obtain a manual correction result; the manual correction result is obtained by adjusting the positions of the video shooting device 1 and the detection radar 2 in a manual manner with the goal of overlapping the video feature points and the radar feature points; the overlapping degree of the video feature points and the radar feature points in the manual correction result reaches a set coarse registration overlapping degree. Specifically:

[0088] Fix the video shooting device 1, fine-tune the direction of the detection radar 2, and make the overlapping degree of the video feature points and the radar feature points reach the set coarse registration overlapping degree. At this time, the video feature points and the radar feature points have not completely overlapped.

[0089] (2) Use an image registration algorithm to re-register the video feature points and the radar feature points in the manual correction result, so that the video feature points and the radar feature points completely overlap, thereby determining the mapping relationship between the video feature points and the radar feature points. The mapping relationship is used to adjust the positions of the video shooting device 1 and the detection radar 2 again to realize the registration of the video image and the radar echo signal. Specifically:

[0090] With the video feature points in the manual correction result as a benchmark, image registration algorithms such as translation, rotation, scaling, etc. are used to make the radar feature points in the manual correction result completely registered with the video feature points in the manual correction result, and the mapping relationship from the radar feature points to the video feature points is recorded and saved. When the system is working normally, the mapping relationship is used to realize the registration of the video image and the radar echo signal.

[0091] In one example, the correction device comprises four rotating structures, and the four rotating structures can enclose a closed square area, as shown in Figure 4 .

[0092] In the following, the working process of the above-mentioned device for fusing radar imaging and video images will be further described in detail by taking the four rotating structures as an example.

[0093] Step 1: Select a monitoring area and install a detection device 3.

[0094] The main function of the detection device 3 is to return video image signals and millimeter wave radar imaging signals in the monitoring area, which is composed of a millimeter wave detection radar, a camera, and a support. The millimeter wave radar and the camera are installed on the same support. The millimeter wave radar and the camera are installed by adjustable bolts, and after installation, the direction of the millimeter wave radar and the camera can be fine-tuned to ensure that the detection areas of the two are highly overlapped. The camera and the millimeter wave detection radar are connected to a computer.

[0095] The computer uses dual network port communication; the millimeter wave radar and the computer use network port communication; and the camera and the computer use network port communication. Other accessories of the computer include a host, a display, a keyboard, a mouse, etc.

[0096] Step 2:

[0097] Install a correction device in the monitoring area, and the main function of the correction device is to provide four feature points for the entire system. In this embodiment, the correction device comprises four rotating structures, still referring to Figure 4 , a rotating structure is similar to a "windmill".

[0098] The correction device is composed of 2 blades, 1 motor, and 1 support. The shape of the blade is a white spherical surface, and the motor uses a reduction motor to drive the blade to rotate slowly.

[0099] When calibrating the feature points of the video image, the white blade can fully reflect the light, and the spherical shape can reflect light in different directions.

[0100] When calibrating the feature points of the radar imaging, the spherical shape can reflect millimeter wave signals in different directions, and the rotating blade reflects dynamic changes of the millimeter wave radar signals, so that the device is suitable for not only dynamic capture of the millimeter wave radar but also static millimeter wave radar.

[0101] The main function of the computer is to process the detected video image signal and millimeter wave radar imaging signal. When the system is calibrated, 4 feature points of the calibration device video image are extracted to form a first group of feature points. 4 feature points of the millimeter wave radar imaging of the calibration device are extracted to form a second group of feature points. The first group of feature points is configured with the second group of feature points as the reference to establish the direct mapping relationship between the video image and the millimeter wave radar imaging, as shown in steps 3-8.

[0102] Step 3:

[0103] Start the motor of the calibration device to rotate the blades; take pictures of the monitoring area through the camera, and filter the video image by using the adaptive subtraction algorithm; then extract the coordinates of the 4 "windmills" (Xp1, Yp1), (Xp2, Yp2), (Xp3, Yp3), (Xp4, Yp4) as the first group of feature points by using the Hough transform algorithm, as shown in Figure 5 .

[0104] Step 4:

[0105] Start the motor of the calibration device to rotate the blades; take pictures of the monitoring area through the camera, and filter the video image by using the adaptive subtraction algorithm; then extract the coordinates of the 4 "windmills" (Xp1, Yp1), (Xp2, Yp2), (Xp3, Yp3), (Xp4, Yp4) as the first group of feature points by using the Hough transform algorithm, as shown in Figure 6 .

[0106] Step 5:

[0107] Manual calibration. Fix the camera, and fine-tune the direction of the millimeter wave radar to make (Xp1, Yp1), (Xp2, Yp2), (Xp3, Yp3), (Xp4, Yp4) and (Xw1, Yw1), (Xw2, Yw2), (Xw3, Yw3), (Xw4, Yw4) coincide as much as possible, as shown in Figure 7 .

[0108] Step 6:

[0109] Software fine-tuning, using image registration algorithm. Take the first group of feature points as the reference, and use translation, rotation, scaling and other image registration algorithms to make the second group of feature points completely registered with the first group of feature points, as shown in Figure 8 .

[0110] Step 7:

[0111] Record and save the mapping relationship from the second set of features to the first set of features.

[0112] Step 8:

[0113] When the system is working normally, according to the mapping relationship of step 7, the registration of millimeter wave radar imaging and video image is realized.

[0114] The radar imaging and video image fusion device of the above embodiment has the following advantages:

[0115] (1) The detection device has a manual adjustment function, which can reduce the registration error.

[0116] (2) The leaves of the correction device are rotating, which can provide dynamic video signals and dynamic millimeter wave radar echo signals, so that the adaptive background subtraction algorithm can be used, and the filtering effect is good. Stop the rotation of the leaves to provide static signals, which is equivalent to the current method.

[0117] (3) The leaves of the correction device are spherical, which can reflect millimeter wave radar signals of different angles, simplifying the installation requirements.

[0118] (4) The leaves of the correction device are white, which increases the reflectivity of the leaves and facilitates night installation and operation.

[0119] (5) In terms of signal filtering, the adaptive background subtraction algorithm is used, which has high efficiency and good accuracy. Especially for the processing of millimeter wave radar echo signals, the adaptive background subtraction algorithm has great advantages. The radar echo signal of patent CN113239948B is static and cannot use this algorithm. In the feature point extraction aspect, the Hough transform algorithm is used to extract the rotation center point, which has high accuracy. Four feature points are used to ensure the accuracy of image registration while simplifying the registration algorithm, making the software run more smoothly.

[0120] Embodiment two

[0121] In order to realize the device corresponding to the above embodiment one, with corresponding functions and technical effects, a method for fusing radar imaging and video image is provided below.

[0122] The method comprises:

[0123] (1) Obtain a video image and a radar echo signal; wherein the video image is obtained by photographing a rotating structure rotating in a target monitoring area; the radar echo signal is a radar signal emitted to a rotating structure rotating in a target monitoring area, and the radar echo signal reflected by the rotating structure; the rotating structure is at least three; all rotating structures can enclose a closed area; the rotating structure comprises two leaves; the leaves are spherical structures.

[0124] (2) filtering the video image by using an adaptive subtraction algorithm to obtain a target image, and extracting the center coordinates of each rotating structure in the target image by using a Hough transform algorithm to obtain video feature points.

[0125] (3) filtering the radar echo signal by using an adaptive subtraction algorithm to obtain a target signal, generating a radar echo image according to the target signal, and extracting the center coordinates of each rotating structure in the radar echo image by using a Hough transform algorithm to obtain radar feature points.

[0126] The specific process of generating the radar echo image is: generating the radar echo image by using a time-frequency analysis method according to the target signal.

[0127] (4) matching the video feature points and the radar feature points, and adjusting the positions of the video shooting device and the detection radar according to the matching result to realize the registration of the video image and the radar echo signal. Specifically:

[0128] obtaining a manual correction result; the manual correction result is obtained by adjusting the positions of the video shooting device and the detection radar in a manual manner with the goal of overlapping the video feature points and the radar feature points; and the overlapping degree of the video feature points and the radar feature points in the manual correction result reaches a set coarse registration overlapping degree.

[0129] re-registering the video feature points and the radar feature points in the manual correction result by using an image registration algorithm, so that the video feature points and the radar feature points completely overlap, thereby determining the mapping relationship between the video feature points and the radar feature points.

[0130] adjusting the positions of the video shooting device and the detection radar according to the mapping relationship to realize the registration of the video image and the radar echo signal.

[0131] The radar imaging and video image fusion method of the embodiment corrects the device by using rotating blades to generate dynamic images (video) and millimeter wave radar imaging; for dynamic images, an adaptive background subtraction algorithm is used to filter out background signals and extract dynamic signals; for dynamic signals, a Hough transform algorithm is used to extract the centers of each rotating blade as image feature points; according to the feature points, an image registration algorithm is used to realize the mutual registration of the video image and the millimeter wave radar imaging, establish the corresponding relationship between the two kinds of images, and realize the mutual registration of the video image and the millimeter wave radar imaging.

[0132] The various embodiments described in this specification are presented for the purpose of illustrating the principles of the application and its best mode of operation. Each of the embodiments described in this specification has been provided for the purpose of illustration and is not intended to limit the application. The same or similar reference numerals in different drawings represent the same or similar elements.

[0133] The principles and operation of the present application have been explained so far with the help of specific examples. The examples have been presented for the purpose of illustration and are not intended to limit the method of the present application. The skilled person in the art will be able to make modifications in the specific embodiments and the scope of application based on the principles of the present application. The description of the specification should not be construed as limiting the present application.

Claims

1. An apparatus for fusing radar imagery with video images, comprising: The application relates to a correction device, a detection device and a processor. The correction device is arranged in a target monitoring area. The detection device is connected with the processor. The correction device comprises at least three rotating structures, all the rotating structures can enclose a closed area, the rotating structure comprises two blades, and the blades are spherical structures. The detection device comprises a video shooting device and a detection radar. The video shooting device is used for: Shooting the rotating structures in the target monitoring area to obtain video images. The detection radar is used for: Transmitting radar signals to the rotating structures in the target monitoring area and receiving radar echo signals reflected by the rotating structures. The processor is used for: Filtering the video images by using an adaptive subtraction algorithm to obtain target images, extracting the center coordinates of each rotating structure in the target images by using a Hough transformation algorithm to obtain video feature points, filtering the radar echo signals by using the adaptive subtraction algorithm to obtain target signals, generating a radar echo image according to the target signals, extracting the center coordinates of each rotating structure in the radar echo image by using the Hough transformation algorithm to obtain radar feature points, and matching the video feature points and the radar feature points to adjust the positions of the video shooting device and the detection radar according to a matching result to realize registration of the video images and the radar echo signals. The color of the blades is white. The rotating structure further comprises a rotating motor. The rotating motor is used for driving the two blades to rotate.

2. The apparatus of claim 1, wherein, The correction device comprises four rotating structures, and the four rotating structures can enclose a closed square area.

3. The apparatus of claim 1, wherein, In terms of generating a radar echo image according to the target signals, the processor is specifically used for: Generating a radar echo image according to the target signals by using a time-frequency analysis method.

4. The apparatus of claim 1, wherein, In terms of matching the video feature points and the radar feature points, adjusting the positions of the video shooting device and the detection radar according to a matching result to realize registration of the video images and the radar echo signals, the processor is specifically used for:

5. The apparatus of claim 1, wherein, Obtaining a manual correction result, the manual correction result is obtained by adjusting the positions of the video shooting device and the detection radar by using a manual method and taking coincidence of the video feature points and the radar feature points as a target, the coincidence degree of the video feature points and the radar feature points in the manual correction result reaches a set coarse registration coincidence degree, re-registering the video feature points and the radar feature points in the manual correction result by using an image registration algorithm to make the video feature points and the radar feature points completely coincide, and determining a mapping relationship of the video feature points and the radar feature points. The mapping relationship is used for adjusting the positions of the video shooting device and the detection radar again to realize registration of the video images and the radar echo signals.

6. The apparatus of claim 1, wherein, Further comprising: A display. The display is used for displaying a closed area enclosed by the video feature points and a closed area enclosed by the radar feature points. Further comprising:

7. The apparatus of claim 1, wherein, ​ ​ ​ 8. A method of fusing radar imagery with video images, the method comprising: ​ Acquire video images and radar echo signals; wherein, the video images are obtained by video shooting device for rotating structure in the target monitoring area to shoot; the radar echo signal is through the detection radar to the target monitoring area in the rotating structure emitting radar signal, the rotating structure reflected radar echo signal; the rotating structure is at least three; all of the rotating structure can surround a closed area; the rotating structure, including: two blades; the blade is a spherical structure; Adopt adaptive subtraction algorithm to filter the video image, obtain target image, and adopt Hough transform algorithm to extract the center coordinates of each rotating structure in the target image, obtain video feature points; Adopt adaptive subtraction algorithm to filter the radar echo signal, obtain target signal, and generate radar echo image according to the target signal, adopt Hough transform algorithm to extract the center coordinates of each rotating structure in the radar echo image, obtain radar feature points; Matching the video feature points and the radar feature points, adjusting the position of the video shooting device and the detection radar according to the matching result, to realize the registration of the video image and the radar echo signal.

9. The method of claim 8, wherein, Matching the video feature points and the radar feature points, adjusting the position of the video shooting device and the detection radar according to the matching result, to realize the registration of the video image and the radar echo signal, specifically including: Acquire manual correction result; the manual correction result is obtained by adjusting the position of the video shooting device and the detection radar with the video feature points and the radar feature points coinciding as the target; the coincidence degree of the video feature points and the radar feature points in the manual correction result reaches the set coarse registration coincidence degree; Using image registration algorithm to re register the video feature points and the radar feature points in the manual correction result, so that the video feature points and the radar feature points completely coincide, so as to determine the mapping relationship between the video feature points and the radar feature points; According to the mapping relationship, adjust the position of the video shooting device and the detection radar again, to realize the registration of the video image and the radar echo signal.

10. The method of claim 8, wherein, According to the target signal, generate radar echo image, specifically including: According to the target signal, generate radar echo image by time-frequency analysis method.

Images