Millimeter wave radar and video fusion method and device
By acquiring and processing millimeter-wave radar and video data, extracting and fusing feature maps, the problem of poor fusion effect caused by data information loss in the prior art is solved, and a more efficient millimeter-wave radar and video fusion effect is achieved.
Patent Information
- Application Number
- CN202311754182.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2025-06-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the existing millimeter-wave radar and video fusion methods, both feature-level and result-level fusion lead to loss of data information, resulting in poor fusion effect.
By acquiring the image data in multiple echo signals and video data of the millimeter wave radar, the distance azimuth map corresponding to each echo signal is determined, and feature extraction is performed based on the distance azimuth map and image data, the millimeter wave feature map and image feature map are obtained, and the millimeter wave feature map and image feature map are fused.
By converting the original echo signal of the millimeter-wave radar into a distance azimuth map, the loss of data information is reduced and the effect of subsequent fusion is improved.
Smart Images
Figure CN120214780A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of target detection, and particularly relates to a method and device for millimeter-wave radar and video fusion. Background Art
[0002] Based on the fusion of data obtained by millimeter-wave radar and camera, real-time and reliable target detection can be achieved.
[0003] In related technologies, most of the fusion methods of millimeter-wave radar and video focus on feature-level fusion and result-level fusion. However, both of these two fusion methods inevitably cause loss of data information, resulting in poor fusion effect of millimeter-wave radar and video. Summary of the Invention
[0004] The embodiments of this application provide a method and device for millimeter-wave radar and video fusion, which can improve the fusion effect of millimeter-wave radar and video.
[0005] The first aspect of the embodiments of this application provides a method for millimeter-wave radar and video fusion, including: obtaining a plurality of echo signals of the millimeter-wave radar, and obtaining image data corresponding to the echo signals in the video data; determining a range-azimuth map corresponding to each echo signal; respectively performing feature extraction based on each range-azimuth map and the image data to obtain a millimeter-wave feature map and an image feature map; and fusing the millimeter-wave feature map and the image feature map.
[0006] Optionally, in a possible implementation manner of the first aspect, the above determining a range-azimuth map corresponding to each echo signal includes:
[0007] Performing fast Fourier transform processing in the range direction on each echo signal to obtain range-direction spectrum information;
[0008] Performing fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain a range-azimuth map.
[0009] Optionally, in another possible implementation manner of the first aspect, before the above performing fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain a range-azimuth map, the method further includes:
[0010] Filtering the range-direction spectrum information by using a low-pass filter.
[0011] Optionally, in another possible implementation manner of the first aspect, the above respectively performing feature extraction based on each range-azimuth map and the image data to obtain a millimeter-wave feature map and an image feature map includes:
[0012] Combining the features of each range-azimuth map into one frame to obtain millimeter-wave range-azimuth map features;
[0013] Extract the features of the millimeter-wave range-azimuth map to obtain a millimeter-wave feature map;
[0014] Extract the features of the image data to obtain an image feature map.
[0015] Optionally, in another possible implementation manner of the first aspect, the above-mentioned merging the features of each range-azimuth map into one frame to obtain the millimeter-wave range-azimuth map features includes:
[0016] Input each range-azimuth map into a preset convolutional neural network. The preset convolutional neural network extracts the temporal features of each range-azimuth map and merges the features, and outputs the millimeter-wave range-azimuth map features.
[0017] Optionally, in another possible implementation manner of the first aspect, the above-mentioned extracting the features of the millimeter-wave range-azimuth map features to obtain a millimeter-wave feature map includes:
[0018] Obtain the annotation data corresponding to each range-azimuth map;
[0019] Input the millimeter-wave range-azimuth map features and the annotation data corresponding to each range-azimuth map into a preset feature extraction model, and the preset feature extraction model outputs a millimeter-wave feature map.
[0020] Optionally, in another possible implementation manner of the first aspect, the above-mentioned extracting the features of the image data to obtain an image feature map includes:
[0021] Obtain the annotation data corresponding to the image data;
[0022] Input the image data and the annotation data corresponding to the image data into a preset feature extraction model, and the preset feature extraction model outputs an image feature map.
[0023] Optionally, in another possible implementation manner of the first aspect, the above-mentioned fusing the millimeter-wave feature map and the image feature map includes:
[0024] Convert the millimeter-wave feature map and the image feature map to the bird's-eye view perspective respectively to obtain a millimeter-wave bird's-eye feature map and an image bird's-eye feature map;
[0025] Fuse the millimeter-wave bird's-eye feature map and the image bird's-eye feature map.
[0026] Optionally, in another possible implementation manner of the first aspect, the above-mentioned converting the millimeter-wave feature map and the image feature map to the bird's-eye view perspective respectively to obtain a millimeter-wave bird's-eye feature map and an image bird's-eye feature map includes:
[0027] By using the pre - calculation and intermittent reduction methods, the millimeter - wave feature map and the image feature map are respectively converted to the bird's - eye view perspective to obtain the millimeter - wave bird's - eye feature map and the image bird's - eye feature map.
[0028] Optionally, in another possible implementation manner of the first aspect, the above - mentioned video data is acquired by a camera. Before acquiring the multiple echo signals of the millimeter - wave radar and acquiring the image data corresponding to the echo signals in the video data, the method further includes:
[0029] Using a timing server to synchronize the timing of the millimeter - wave radar and the camera.
[0030] A second aspect of the embodiments of the present application provides a device for millimeter - wave radar and video fusion, including:
[0031] A data acquisition module, configured to acquire multiple echo signals of the millimeter - wave radar and acquire image data corresponding to the echo signals in the video data;
[0032] A signal processing module, configured to determine the range - azimuth map corresponding to each echo signal;
[0033] A feature extraction module, configured to perform feature extraction based on each range - azimuth map and the image data respectively to obtain a millimeter - wave feature map and an image feature map;
[0034] A fusion processing module, configured to fuse the millimeter - wave feature map and the image feature map.
[0035] A third aspect of the embodiments of the present application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for millimeter - wave radar and video fusion in the first aspect is implemented.
[0036] A fourth aspect of the embodiments of the present application provides a computer - readable storage medium. The computer - readable storage medium stores a computer program. When the computer program is executed by a processor, the method for millimeter - wave radar and video fusion in the first aspect is implemented.
[0037] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device is enabled to execute the method for millimeter - wave radar and video fusion in the first aspect.
[0038] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: The embodiments of the present application disclose a method and device for millimeter-wave radar and video fusion. Among them, the method first obtains a plurality of echo signals of the millimeter-wave radar and obtains image data corresponding to the echo signals in the video data. Then, a distance-azimuth map corresponding to each echo signal is determined. Next, feature extraction is respectively performed based on each distance-azimuth map and the image data to obtain a millimeter-wave feature map and an image feature map. Finally, the millimeter-wave feature map and the image feature map are fused. Thus, by converting the original echo signals of the millimeter-wave radar into a distance-azimuth map, the loss of data information is minimized as much as possible, thereby improving the subsequent fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the prior art description. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0040] Figure 1 is a schematic flowchart of a method for millimeter-wave radar and video fusion provided by an embodiment of the present application;
[0041] Figure 2 is a schematic structural diagram of a device for millimeter-wave radar and video fusion provided by an embodiment of the present application;
[0042] Figure 3 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0043] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.
[0044] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.
[0045] It should also be understood that the term "and / or" as used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.
[0046] As used in the specification and appended claims of this application, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".
[0047] In addition, in the description of the specification and appended claims of this application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and should not be construed as indicating or implying relative importance.
[0048] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.
[0049] It should be understood that the magnitude of the sequence numbers of the steps in this embodiment does not mean the sequence of execution, and the execution sequence of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.
[0050] In the related art, most millimeter-wave radar and video fusion methods focus on feature-level fusion and result-level fusion. However, both of these fusion methods inevitably cause loss of data information, resulting in poor effects of millimeter-wave radar and video fusion.
[0051] In view of this, embodiments of the present application provide a method and apparatus for millimeter-wave radar and video fusion. First, a plurality of echo signals of the millimeter-wave radar are obtained, and image data corresponding to the echo signals in the video data is obtained. Then, a distance-azimuth map corresponding to each echo signal is determined. Next, feature extraction is respectively performed based on each distance-azimuth map and the image data to obtain a millimeter-wave feature map and an image feature map. Finally, the millimeter-wave feature map and the image feature map are fused. Thus, by converting the original echo signals of the millimeter-wave radar into distance-azimuth maps, the loss of data information is minimized as much as possible, thereby improving the subsequent fusion effect.
[0052] To illustrate the technical solution of the present application, specific embodiments are used below for illustration.
[0053] Refer to Figure 1 , which shows a schematic flowchart of a method for millimeter-wave radar and video fusion provided by an embodiment of the present application. As Figure 1 shown, the method for millimeter-wave radar and video fusion may include the following steps:
[0054] Step 101, obtain a plurality of echo signals of the millimeter-wave radar, and obtain image data corresponding to the echo signals in the video data.
[0055] It should be noted that the millimeter-wave radar can send a special frequency-modulated signal to the detection area and then receive the signal reflected by the target. This frequency-modulated signal is a signal that changes linearly in frequency, that is, a signal whose frequency changes with time. In radar applications, this signal is usually called a chirp signal. In a millimeter-wave radar, the chirp signal is used to scan a certain frequency range. When the chirp signal is transmitted and hits the target, the target will reflect a series of signals with different time delays. These signals are processed by the receiving end to form chirp echo signals.
[0056] Among them, the plurality of echo signals may be echo signals collected by the millimeter-wave radar within one frame or multiple frames.
[0057] In an embodiment of the present application, the echo signals of the millimeter-wave radar carry timestamp tags. While collecting the echo signals of the millimeter-wave radar, the video data is saved, and then the image data synchronized with the echo signals can be found by frame extraction in the video data according to the timestamp tags of the millimeter-wave radar echo signals. It should be understood that the video data can be obtained through an image sensor such as a camera, and the detection areas of the millimeter-wave radar and the camera should be kept consistent.
[0058] In a possible implementation, to ensure that the data collected by the millimeter-wave radar and the camera have the same time reference for subsequent precise and effective fusion processing, before step 101, a timing server can be used to synchronize the timing of the millimeter-wave radar and the camera. Among them, the timing server can provide high-precision time information and ensure that each device operates according to the same time axis. The millimeter-wave radar and the camera communicate with the timing server and obtain accurate timestamps, and then the millimeter-wave radar and the camera adjust their local clocks according to this timestamp to ensure that the millimeter-wave radar and the camera work on the same time axis.
[0059] Step 102, determine the range-azimuth map corresponding to each echo signal.
[0060] It should be noted that the range-azimuth map can provide the range and azimuth information of the target.
[0061] In a possible implementation, after obtaining a chirp echo signal of the millimeter-wave radar, fast Fourier transform (Fast-FFT) processing can be performed in the range direction of the echo signal, and then a fast Fourier transform processing in the azimuth direction is performed to obtain a range-azimuth map with a timestamp label. That is to say, the above step 102 can include: performing fast Fourier transform processing in the range direction on each echo signal to obtain range-direction spectrum information; performing fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain a range-azimuth map.
[0062] Among them, the fast Fourier transform is an efficient signal processing algorithm used to convert a time-domain signal into a frequency-domain signal. In the millimeter-wave radar echo signal, the range direction represents the range information of different targets received by the radar. By performing Fast-FFT processing on the echo signal, it can be converted from the time domain to the frequency domain to obtain the spectrum information in the range direction, which can more conveniently analyze and detect targets at different ranges. Next, performing Fast-FFT processing in the azimuth direction on the range-direction spectrum information can obtain a range-azimuth map with a timestamp label.
[0063] Specifically, the fast Fourier transform processing can be performed through the following formula:
[0064]
[0065] Among them, X(k) is the kth frequency component in the frequency domain, representing the contribution of the original time-domain signal x(n) at frequency k; N represents the total number of samples of the time-domain signal x(n); x(n) represents the nth sample of the time-domain signal; e -jπnt / N is the complex exponential function, representing the contribution of each sample to the specific frequency component k; j is the imaginary unit.
[0066] Furthermore, after Fast-FFT processing, there may be some clutter and noise interference in the obtained range spectrum. Therefore, after obtaining the range spectrum information through the first Fast-FFT processing, a low-pass filter (LPF) can be used to filter the range spectrum information to remove part of the clutter interference. Among them, the low-pass filter can be used to remove the part of the spectrum above a certain frequency, thereby removing part of the clutter interference. The low-pass filter allows the low-frequency components to pass through while blocking the high-frequency components, which can effectively improve the signal-to-noise ratio of the signal.
[0067] Step 103: Based on each range-azimuth map and image data respectively, perform feature extraction to obtain a millimeter-wave feature map and an image feature map.
[0068] After obtaining the range-azimuth map, the steps of feature extraction and subsequent fusion can be carried out.
[0069] In a possible implementation manner, in order to make full use of the temporal information between different data frames of the range-azimuth map, the features of each range-azimuth map can be merged into one frame and then subsequent feature extraction can be performed. That is to say, the above step 103 can include: merging the features of each range-azimuth map into one frame to obtain the millimeter-wave range-azimuth map feature; performing feature extraction on the millimeter-wave range-azimuth map feature to obtain a millimeter-wave feature map; and performing feature extraction on the image data to obtain an image feature map.
[0070] As an example, merging the features of each range-azimuth map into one frame can be processed by a convolutional neural network (CNN). That is to say, each range-azimuth map can be input into a preset convolutional neural network, and the preset convolutional neural network extracts the temporal features of each range-azimuth map and merges the features, and outputs the millimeter-wave range-azimuth map feature.
[0071] Specifically, an M-Net network module that is essentially a preset convolutional neural network can be constructed. The M-Net network module mainly includes a temporal deformable convolutional module and a temporal max pooling module. The role of the temporal deformable convolution is mainly to extract the temporal features in each echo signal. The temporal deformable convolution can achieve a good extraction effect on the moving target features in each echo signal. After extracting the features of the range histogram corresponding to each echo signal, all the features are merged through the temporal max pooling layer.
[0072] It should also be noted that the input dimension of the M-Net network module is (2, n, H, W), where 2 represents the 2 channels of each millimeter-wave echo signal. Since the millimeter-wave echo is generally a complex signal, it includes the real part and the imaginary part; H represents the height of the range-azimuth map; W is the width of the range-azimuth map; n represents the range-azimuth maps corresponding to multiple echo signals in 1 frame. If n = 1, it means randomly selecting the data of one echo signal to input into the network. Otherwise, the data of some echo signals in this frame are input into this M-Net network module together, achieving the effect of feature merging and enhancement in time series. The output dimension of the M-Net network module is (C, H, W), where C represents the number of time-domain convolution filters, and H and W are the height and width of the image respectively. After processing each millimeter-wave radar frame by the M-Net network module, the features in the range-azimuth maps extracted from all the input frames will be connected over time.
[0073] In the embodiment of this application, after obtaining the millimeter-wave range-azimuth map features, the millimeter-wave range-azimuth map features (C, H, W) and the image data (3, H, W) can be respectively input into a preset feature extraction model for feature extraction to obtain a millimeter-wave feature map and an image feature map.
[0074] Among them, for the millimeter-wave range-azimuth map features, first, the annotation data corresponding to each range-azimuth map can be obtained, and then the millimeter-wave range-azimuth map features and the annotation data corresponding to each range-azimuth map are input into the preset feature extraction model, and the preset feature extraction model outputs the millimeter-wave feature map.
[0075] As an example, the annotation data corresponding to each range-azimuth map can be obtained by annotating the target position, target category, and target radar reflection intensity in the range-azimuth map.
[0076] Among them, the preset feature extraction model can be Swin-Transformer. Swin-Transformer is a deep learning model based on the Transformer architecture, which is specifically used for image processing tasks. Swin-Transformer introduces a self-attention mechanism in the image, which can effectively capture the correlation and semantic information at different positions in the image. At the same time, Swin-Transformer also adopts a hierarchical structure, enabling it to process large-size image data.
[0077] For the feature extraction of the image feature map, similarly, the annotation data corresponding to the image data can be obtained first, and then the image data and the annotation data corresponding to the image data are input into the preset feature extraction model, and the preset feature extraction model outputs the image feature map.
[0078] Step 104, fuse the millimeter-wave feature map and the image feature map.
[0079] In a possible implementation, to more conveniently process the information of the target in the three-dimensional space, the millimeter-wave feature map and the image feature map can be respectively converted to the bird's-eye view (BEV) perspective to obtain the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map, and then the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map are fused.
[0080] It should be noted that the bird's-eye view perspective observes the scene from a top-down angle and uses the ground plane as the reference plane. The bird's-eye view helps to better retain its semantic feature information and fully reduce the loss of data information. In addition, the bird's-eye view is presented in a planar manner, enabling the information obtained by different sensors to be represented in the same coordinate system, facilitating the comparison and fusion of information. The bird's-eye view also provides a global perspective of the environment, helping to better understand the positions and interrelationships of objects in the scene. The bird's-eye view can also avoid the problem of target occlusion.
[0081] In the embodiments of the present application, to accelerate BEV pooling and reduce the time consumption of BEV space conversion, the millimeter-wave feature map and the image feature map can be respectively converted to the bird's-eye view perspective by using the pre-computation and intermittent reduction method to obtain the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map.
[0082] It should be noted that pre-computation can avoid complex calculations in real-time applications by pre-computing the transformation matrix or parameters required for conversion to the bird's-eye view. Intermittent reduction means performing bird's-eye view conversion intermittently, without the need to perform conversion at each moment, but selecting an appropriate timing for conversion according to the application scenario and requirements.
[0083] The method for millimeter-wave radar and video fusion disclosed in the above embodiments of the present application first obtains multiple echo signals of the millimeter-wave radar and obtains the image data corresponding to the echo signals in the video data. Then, the distance-azimuth map corresponding to each echo signal is determined. Next, feature extraction is respectively performed based on each distance-azimuth map and the image data to obtain the millimeter-wave feature map and the image feature map. Finally, the millimeter-wave feature map and the image feature map are fused. Thus, by converting the original echo signals of the millimeter-wave radar into distance-azimuth maps, the loss of data information is minimized as much as possible, thereby improving the subsequent fusion effect.
[0084] See Figure 2 , which shows a schematic structural diagram of a device for millimeter-wave radar and video fusion provided by the embodiments of the present application. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.
[0085] The device for millimeter-wave radar and video fusion may specifically include the following modules:
[0086] The data acquisition module 201 is configured to acquire multiple echo signals of the millimeter-wave radar and acquire image data corresponding to the echo signals in the video data.
[0087] The signal processing module 202 is configured to determine the range-azimuth map corresponding to each echo signal.
[0088] The feature extraction module 203 is configured to perform feature extraction based on each range-azimuth map and the image data respectively to obtain a millimeter-wave feature map and an image feature map.
[0089] The fusion processing module 204 is configured to fuse the millimeter-wave feature map and the image feature map.
[0090] The device for millimeter-wave radar and video fusion disclosed in the above embodiments of the present application first acquires multiple echo signals of the millimeter-wave radar and acquires image data corresponding to the echo signals in the video data. Then it determines the range-azimuth map corresponding to each echo signal. Next, it performs feature extraction based on each range-azimuth map and the image data respectively to obtain a millimeter-wave feature map and an image feature map. Finally, it fuses the millimeter-wave feature map and the image feature map. Thus, by converting the original echo signals of the millimeter-wave radar into range-azimuth maps, the loss of data information is minimized as much as possible, thereby improving the subsequent fusion effect.
[0091] Further, in a possible implementation manner of the embodiment of the present application, the above signal processing module 202 may specifically include the following sub-modules:
[0092] The first processing sub-module is configured to perform fast Fourier transform processing in the range direction on each echo signal to obtain range-direction spectrum information.
[0093] The second processing sub-module is configured to perform fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain a range-azimuth map.
[0094] Further, in another possible implementation manner of the embodiment of the present application, the above signal processing module 202 may specifically further include the following sub-modules:
[0095] The third processing sub-module is configured to filter the range-direction spectrum information by using a low-pass filter.
[0096] Further, in yet another possible implementation manner of the embodiment of the present application, the above feature extraction module 203 may specifically include the following sub-modules:
[0097] The fourth processing sub-module is configured to merge the features of each range-azimuth map into one frame to obtain millimeter-wave range-azimuth map features.
[0098] The fifth processing sub-module is used to extract features from the millimeter-wave range-azimuth map features to obtain a millimeter-wave feature map.
[0099] The sixth processing sub-module is used to extract features from the image data to obtain an image feature map.
[0100] Furthermore, in another possible implementation manner of the embodiment of the present application, the above-mentioned fourth processing sub-module may specifically include the following units:
[0101] The first processing unit is used to input each range-azimuth map into a preset convolutional neural network. The preset convolutional neural network extracts the temporal features of each range-azimuth map and merges the features, and outputs millimeter-wave range-azimuth map features.
[0102] Furthermore, in another possible implementation manner of the embodiment of the present application, the above-mentioned fifth processing sub-module may specifically include the following units:
[0103] The first acquisition unit is used to acquire the annotation data corresponding to each range-azimuth map.
[0104] The second processing unit is used to input the millimeter-wave range-azimuth map features and the annotation data corresponding to each range-azimuth map into a preset feature extraction model. The preset feature extraction model outputs a millimeter-wave feature map.
[0105] Furthermore, in another possible implementation manner of the embodiment of the present application, the above-mentioned sixth processing sub-module may specifically include the following units:
[0106] The second acquisition unit is used to acquire the annotation data corresponding to the image data.
[0107] The third processing unit is used to input the image data and the annotation data corresponding to the image data into a preset feature extraction model. The preset feature extraction model outputs an image feature map.
[0108] Furthermore, in another possible implementation manner of the embodiment of the present application, the above-mentioned fusion processing module 204 may specifically include the following sub-modules:
[0109] The seventh processing sub-module is used to convert the millimeter-wave feature map and the image feature map to the bird's-eye view perspective respectively to obtain a millimeter-wave bird's-eye view feature map and an image bird's-eye view feature map.
[0110] The eighth processing sub-module is used to fuse the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map.
[0111] Furthermore, in another possible implementation manner of the embodiment of the present application, the above-mentioned seventh processing sub-module may specifically include the following units:
[0112] A fourth processing unit, configured to use pre - calculation and intermittent reduction methods to respectively convert the millimeter - wave feature map and the image feature map to the bird's - eye view perspective, obtaining a millimeter - wave bird's - eye feature map and an image bird's - eye feature map.
[0113] Further, in another possible implementation manner of the embodiments of the present application, the above - mentioned video data is obtained by a camera, and the millimeter - wave radar and video fusion device may specifically further include the following modules:
[0114] A timing module, configured to synchronously time the millimeter - wave radar and the camera by using a timing server.
[0115] The millimeter - wave radar and video fusion device provided by the embodiments of the present application can be applied to the method embodiments described above. For details, refer to the description of the above - mentioned method embodiments and will not be repeated here.
[0116] Figure 3 It is a schematic structural diagram of a terminal device provided by the embodiments of the present application. As Figure 3 shown, the terminal device 300 of this embodiment includes: at least one processor 310 ( Figure 3 only one is shown in the figure), a processor, a memory 320, and a computer program 321 stored in the memory 320 and executable on the at least one processor 310. When the processor 310 executes the computer program 321, it implements the steps in the method embodiments of the above - mentioned millimeter - wave radar and video fusion.
[0117] The terminal device 300 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor 310 and a memory 320. Those skilled in the art can understand that Figure 3 merely examples of the terminal device 300, and do not constitute a limitation on the terminal device 300. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.
[0118] The so-called processor 310 may be a Central Processing Unit (CPU), and this processor 310 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.
[0119] In some embodiments, the memory 320 may be an internal storage unit of the terminal device 300, such as the hard disk or memory of the terminal device 300. In other embodiments, the memory 320 may also be an external storage device of the terminal device 300, such as a plug-in hard disk equipped on the terminal device 300, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 320 may also include both the internal storage unit of the terminal device 300 and the external storage device. The memory 320 is used to store an operating system, application programs, a Boot Loader, data, and other programs, such as the program code of the computer program, etc. The memory 320 may also be used to temporarily store data that has been output or is to be output.
[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.
[0121] In the above embodiments, the descriptions of the various embodiments each have their own emphasis. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.
[0122] Those of ordinary skill in the art will realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0123] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.
[0124] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0125] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.
[0126] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.
[0127] To implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-described method embodiments when executed.
[0128] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.
Claims
1. A method for millimeter-wave radar and video fusion, characterized in that Including: Obtaining a plurality of echo signals of a millimeter-wave radar, and obtaining image data corresponding to the echo signals in video data; Determining a range-azimuth map corresponding to each of the echo signals; Performing feature extraction respectively based on each of the range-azimuth maps and the image data to obtain a millimeter-wave feature map and an image feature map; Fusing the millimeter-wave feature map and the image feature map.
2. The method for millimeter-wave radar and video fusion according to claim 1, wherein, The determining a range-azimuth map corresponding to each of the echo signals includes: Performing fast Fourier transform processing in the range direction on each of the echo signals to obtain range-direction spectrum information; Performing fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain the range-azimuth map.
3. The method for millimeter-wave radar and video fusion according to claim 2, wherein, Before the performing fast Fourier transform processing in the azimuth direction on the range-direction spectrum information to obtain the range-azimuth map, the method further includes: Filtering the range-direction spectrum information by using a low-pass filter.
4. The method for millimeter-wave radar and video fusion according to claim 1, wherein The performing feature extraction respectively based on each of the range-azimuth maps and the image data to obtain a millimeter-wave feature map and an image feature map includes: Merging the features of each of the range-azimuth maps into one frame to obtain millimeter-wave range-azimuth map features; Performing feature extraction on the millimeter-wave range-azimuth map features to obtain the millimeter-wave feature map; Performing feature extraction on the image data to obtain the image feature map.
5. The method for millimeter-wave radar and video fusion according to claim 4, wherein The merging the features of each of the range-azimuth maps into one frame to obtain millimeter-wave range-azimuth map features includes: Inputting each of the range-azimuth maps into a preset convolutional neural network, and the preset convolutional neural network extracts the temporal features of each of the range-azimuth maps and performs feature merging to output the millimeter-wave range-azimuth map features.
6. The method for millimeter-wave radar and video fusion according to claim 4, characterized in that, The performing feature extraction on the millimeter-wave range-azimuth map features to obtain the millimeter-wave feature map includes: Obtaining annotation data corresponding to each of the range-azimuth maps; Inputting the millimeter-wave range-azimuth map features and the annotation data corresponding to each of the range-azimuth maps into a preset feature extraction model, and the preset feature extraction model outputs the millimeter-wave feature map.
7. The method for millimeter-wave radar and video fusion according to claim 4, characterized in that, The performing feature extraction on the image data to obtain the image feature map includes: Obtaining annotation data corresponding to the image data; Inputting the image data and the annotation data corresponding to the image data into a preset feature extraction model, and the preset feature extraction model outputs the image feature map.
8. The method for millimeter-wave radar and video fusion according to claim 1, characterized in that The fusing the millimeter-wave feature map and the image feature map includes: Respectively converting the millimeter-wave feature map and the image feature map to a bird's-eye view perspective to obtain a millimeter-wave bird's-eye view feature map and an image bird's-eye view feature map; Fusing the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map.
9. The method for millimeter-wave radar and video fusion according to claim 8, characterized in that, The respectively converting the millimeter-wave feature map and the image feature map to a bird's-eye view perspective to obtain a millimeter-wave bird's-eye view feature map and an image bird's-eye view feature map includes: Using a pre-computation and intermittent reduction method to respectively convert the millimeter-wave feature map and the image feature map to a bird's-eye view perspective to obtain the millimeter-wave bird's-eye view feature map and the image bird's-eye view feature map.
10. The method for millimeter-wave radar and video fusion according to any one of claims 1-9, characterized in that, The video data is acquired by a camera. Before acquiring multiple echo signals of the millimeter-wave radar and acquiring image data corresponding to the echo signals in the video data, the method further includes: Synchronously timing the millimeter-wave radar and the camera by using a timing server.
11. An apparatus for millimeter-wave radar and video fusion, characterized in that, It includes: A data acquisition module, configured to acquire multiple echo signals of the millimeter-wave radar and acquire image data corresponding to the echo signals in the video data; A signal processing module, configured to determine a range-azimuth map corresponding to each of the echo signals; A feature extraction module, configured to perform feature extraction based on each of the range-azimuth maps and the image data respectively to obtain a millimeter-wave feature map and an image feature map; A fusion processing module, configured to fuse the millimeter-wave feature map and the image feature map.
12. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the method described in any one of claims 1 to 10 is implemented.