Millimeter wave radar data and image fusion method and device under high-speed scenario
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- VANJEE TECHNOLOGY CO LTD
- Filing Date
- 2024-12-02
- Publication Date
- 2026-06-09
Smart Images
Figure CN122172180A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of target detection technology, and in particular relates to a method and apparatus for fusing millimeter-wave radar data and images in high-speed scenarios. Background Technology
[0002] In high-speed scenarios, utilizing multi-sensor fusion technology to achieve performance complementarity between different sensors has become a common solution for improving the performance of road vehicle target detection and tracking.
[0003] In related technologies, the common deployment scheme for roadside sensing devices is to use a combination of radar and bullet cameras to carry out subsequent multi-sensor fusion target detection processes.
[0004] However, the sensing range of a gun-type camera is much smaller than that of a radar. There is a visual perception blind spot at medium and long distances, which leads to a large gap between the visual perception range and the radar perception range, reducing the target fusion performance and thus affecting the target detection accuracy. Summary of the Invention
[0005] This application provides a method and apparatus for fusing millimeter-wave radar data and images in high-speed scenarios, which can improve target detection accuracy.
[0006] In a first aspect, embodiments of this application provide a method for fusing millimeter-wave radar data and images in a high-speed scenario, applied to a first base station on the side of a highway. The first base station includes at least a first millimeter-wave radar, a second millimeter-wave radar, and a spherical camera. The first and second millimeter-wave radars are arranged back-to-back, and the spherical camera has multiple focal lengths. The method includes: acquiring millimeter-wave radar data within a first detection area using the first millimeter-wave radar, and acquiring millimeter-wave radar data within a second detection area using the second millimeter-wave radar. The first detection area is a road area along a first driving direction of the highway, starting from the first base station, and the second detection area is a road area along a second driving direction of the highway, starting from the first base station. The first driving direction is opposite to the second driving direction. By adjusting the focal length of the spherical camera, images corresponding to each focal length are acquired. The coverage area of each focal length of the spherical camera includes both the first and second detection areas. The millimeter-wave radar data of the first and second detection areas and the images corresponding to each focal length are then fused.
[0007] Optionally, in one possible implementation of the first aspect, a focal length overlap region is provided between adjacent focal lengths of the aforementioned spherical camera.
[0008] Optionally, in another possible implementation of the first aspect, the above-mentioned fusion of millimeter-wave radar data of the first detection area and the second detection area with images corresponding to each focal length includes: determining radar-sensing targets based on millimeter-wave radar data of the first detection area and the second detection area; determining camera-sensing targets based on images corresponding to each focal length; and performing spatiotemporal synchronization of radar-sensing targets and camera-sensing targets.
[0009] Optionally, in another possible implementation of the first aspect, the above-mentioned spatiotemporal synchronization of radar-sensing targets and camera-sensing targets includes: spatially synchronizing radar-sensing targets and camera-sensing targets using a calibration algorithm within the coverage area of each focal length of the spherical camera; and temporally synchronizing radar-sensing targets and camera-sensing targets using a timestamp synchronization algorithm in response to the operation of switching the focal length of the spherical camera.
[0010] Optionally, in another possible implementation of the first aspect, the method further includes: acquiring millimeter-wave radar blind zone data within a preset range around the first base station using a first millimeter-wave radar and a second millimeter-wave radar; acquiring blind zone images within a preset range around the first base station using a spherical camera; and fusing the millimeter-wave radar data of the first detection area and the second detection area with images corresponding to each focal length, including: fusing the millimeter-wave radar data of the first detection area and the second detection area, images corresponding to each focal length, millimeter-wave radar blind zone data, and blind zone images.
[0011] Optionally, in another possible implementation of the first aspect, a sensing overlap area is provided between the first base station and the second base station, and the second base station is the next adjacent base station of the first base station.
[0012] Optionally, in another possible implementation of the first aspect, the method further includes: when the target moves into the sensing overlap area, sending the target detection result after target fusion to the second base station through an identity matching algorithm, so that the second base station can continue to perform fusion detection on the target based on the target detection result.
[0013] Optionally, in another possible implementation of the first aspect, the method further includes: determining the coverage range of each focal length of the spherical camera based on the sensing range of the first millimeter-wave radar and the second millimeter-wave radar.
[0014] Optionally, in another possible implementation of the first aspect, the method further includes: acquiring the speed of each target in the first detection area and the second detection area within a preset time period; updating the focal length switching frequency of the spherical camera according to the speed of each target; and acquiring an image corresponding to each focal length by adjusting the focal length of the spherical camera, including: adjusting the focal length of the spherical camera based on the focal length switching frequency to acquire an image corresponding to each focal length.
[0015] Optionally, in another possible implementation of the first aspect, updating the focal length switching frequency of the spherical camera according to the speed of each target includes: determining the lowest speed among the speeds of each target; and updating the focal length switching frequency of the spherical camera according to the lowest speed.
[0016] Secondly, embodiments of this application provide a device for fusing millimeter-wave radar data and images in high-speed scenarios, applied to a first base station on the side of a highway. The first base station includes at least a first millimeter-wave radar, a second millimeter-wave radar, and a spherical camera. The first and second millimeter-wave radars are arranged back-to-back, and the spherical camera has multiple focal lengths. The device includes:
[0017] The acquisition module is used to acquire millimeter-wave radar data in a first detection area via a first millimeter-wave radar, and to acquire millimeter-wave radar data in a second detection area via a second millimeter-wave radar. The first detection area is a road area along a first driving direction of the highway starting from a first base station, and the second detection area is a road area along a second driving direction of the highway starting from a first base station. The first driving direction is opposite to the second driving direction.
[0018] The processing module is used to acquire the image corresponding to each focal length by adjusting the focal length of the spherical camera, wherein the range covered by each focal length of the spherical camera includes the first detection area and the second detection area;
[0019] The processing module is also used to fuse the millimeter-wave radar data of the first and second detection areas with the images corresponding to each focal length.
[0020] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device is able to implement any of the methods described in the first aspect above.
[0021] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, enable the electronic device to perform any of the methods described in the first aspect.
[0022] Fifthly, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device is able to implement any of the methods described in the first aspect.
[0023] The beneficial effects of this application embodiment compared with the prior art are as follows: This application discloses a method and apparatus for fusing millimeter-wave radar data and images in high-speed scenarios. In this method, two millimeter-wave radars arranged back-to-back and a variable-focus spherical camera are combined to achieve target detection in highway scenarios. First, the two millimeter-wave radars acquire radar data from both directions of the highway. By adjusting the focal length of the spherical camera, visual perception data at each focal length can be obtained. Finally, the perception data from the millimeter-wave radar and the spherical camera are fused to obtain the final target detection result. Thus, the variable-focus spherical camera ensures that there are no blind spots at medium to long distances, thereby controlling the visual perception range to be consistent with the radar perception range and improving target detection accuracy. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a schematic diagram of the deployment of a roadside sensing device according to an embodiment of this application;
[0026] Figure 2 This is a flowchart illustrating a method for fusing millimeter-wave radar data and images in a high-speed scenario, as provided in an embodiment of this application.
[0027] Figure 3 This is a schematic diagram of the structure of a device for fusing millimeter-wave radar data and images in a high-speed scenario, provided in an embodiment of this application.
[0028] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0030] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0031] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0032] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0033] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0034] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0035] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.
[0036] In related technologies, a common deployment scheme for roadside sensing devices typically employs a combination of radar and bullet cameras for subsequent multi-sensor fusion target detection. However, the sensing range of a bullet camera is much smaller than that of radar, resulting in visual blind spots at medium to long distances. This leads to a significant difference between the visual and radar sensing ranges, reducing target fusion performance and consequently affecting target detection accuracy.
[0037] Secondly, bullet cameras are unidirectional and have a limited sensing range. To improve target detection accuracy, the only option is to increase the number of base stations and shorten the distance between them, which undoubtedly increases costs.
[0038] In view of this, this application provides a method and apparatus for fusing millimeter-wave radar data and images in high-speed scenarios. This method employs a combination of two millimeter-wave radars arranged back-to-back and a variable-focus spherical camera to achieve target detection in highway scenarios. First, the two millimeter-wave radars acquire radar data from both directions of the highway. By adjusting the focal length of the spherical camera, visual perception data at each focal length can be obtained. Finally, the perception data from the millimeter-wave radar and the spherical camera are fused to obtain the final target detection result. Thus, the variable-focus spherical camera ensures that there are no blind spots at medium to long distances, thereby controlling the visual perception range to be consistent with the radar perception range and improving target detection accuracy.
[0039] To illustrate the technical solution of this application, specific embodiments are described below.
[0040] Figure 1 This is a schematic diagram illustrating the deployment of a roadside sensing device according to an embodiment of this application. Figure 1 As shown, a first base station is installed on the pole. The first base station includes at least a first millimeter-wave radar, a second millimeter-wave radar, and a spherical camera. The first millimeter-wave radar and the second millimeter-wave radar are arranged back to back.
[0041] It should be noted that millimeter-wave radar is radar that operates in the millimeter-wave band for detection. Millimeter waves typically refer to the 30–300 GHz frequency range (wavelength 1–10 mm). Millimeter-wave radar is commonly used to detect traffic targets (people, non-motorized vehicles, motorized vehicles, etc.) and obtain information such as target location, speed, heading angle, and type.
[0042] A PTZ camera is a surveillance camera that integrates a camera system, a variable focal length lens, and an electronic pan-tilt unit. PTZ cameras can rotate 360 degrees via their electronic pan-tilt system, achieving full coverage of the monitored area. The built-in variable focal length lens allows for clear monitoring of targets at both near and far distances.
[0043] The first millimeter-wave radar can be responsible for sensing a range of 10 to 300m (an example of the first detection area), and the corresponding second millimeter-wave radar can be responsible for sensing a range of -10 to -300 (an example of the second detection area).
[0044] In this embodiment, the coverage area of each focal length of the spherical camera can be matched with the common sensing range of the first millimeter-wave radar and the second millimeter-wave radar, thereby ensuring that the visual sensing range is consistent with the radar sensing range. Figure 1 As shown, from the first driving direction (i.e. Figure 1 Looking at the view from the right, the spherical camera selects four focal lengths with a coverage range of approximately 80m: 10–80m, 70–160m, 150–230m, and 220–300m (an example of multiple focal lengths set by the spherical camera), responsible for visual perception from near to far within 10–300m, consistent with the first detection area corresponding to the aforementioned first millimeter-wave radar. Furthermore, since the spherical camera can achieve a 360-degree wide-range perception, the aforementioned four focal lengths also support visual perception from -10 to 300m.
[0045] It should be understood that the number of focal lengths of the spherical camera can be set according to actual needs, such as two, three, four, etc., and this application embodiment does not limit this.
[0046] The coverage range of each focal length can be determined by considering the image quality acquired by the dome camera itself. For example, if a dome camera produces better image quality within an 80m range, then the focal length can be set based on 80m.
[0047] In one embodiment, a focal length overlap region can be set between adjacent focal lengths of the spherical camera. For example, a 10m area (70-80m) between 10-80m and 70-160m is the focal length overlap region. By setting the focal length overlap region, the problem of missing visual perception of the intersection range of adjacent points can be solved.
[0048] In one embodiment, the height of the millimeter-wave radar and the spherical camera above the ground can be 4 to 6 meters, and the coverage range of the millimeter-wave radar and the spherical camera can be expanded by increasing the height.
[0049] In one embodiment, setting two millimeter-wave radars back-to-back may result in a sensing blind zone within a range of -10 to -10 meters around the pole (an example of a preset range around the first base station), causing this range to become a sensing blind zone. Therefore, an additional sensing field of view with a range of 20 meters can be added under the pole, and a spherical camera can also be added accordingly to avoid the situation of missing sensing.
[0050] In this embodiment of the application, the sensing range of the two base stations can be set to an overlapping sensing area of about 20m. In this overlapping sensing area, the fused target detection data can be transmitted to achieve target detection in the entire range.
[0051] Reference Figure 2 The diagram illustrates a flowchart of a method for fusing millimeter-wave radar data and images in high-speed scenarios, as provided in an embodiment of this application. Figure 2 As shown, the method may include the following steps:
[0052] Step 201: Acquire millimeter-wave radar data within the first detection area using the first millimeter-wave radar, and acquire millimeter-wave radar data within the second detection area using the second millimeter-wave radar.
[0053] The first detection area is the road area along the first driving direction of the highway, starting from the first base station. See the specific details for reference. Figure 1 In the area between 0 and 300m, the first direction of travel can be... Figure 1 The view shows the direction to the right. The second detection area is the road area along the second driving direction of the highway, starting from the first base station. The first driving direction is opposite to the second driving direction. See [reference needed] for details. Figure 1 In the area between -300 and 0m, the second direction of travel can be... Figure 1 The leftward direction in the view.
[0054] In one embodiment, millimeter-wave radar data may include information such as the location, speed, heading angle, and type of all targets within the detection area.
[0055] Step 202: By adjusting the focal length of the spherical camera, obtain the image corresponding to each focal length.
[0056] In this embodiment, the spherical camera performs live scanning at different focal lengths according to a set frequency, thereby acquiring images at each focal length.
[0057] It should be noted that the coverage area of each focal length of the spherical camera includes both the first and second detection areas. That is, the coverage area of each focal length of the spherical camera can match the common sensing range of the first and second millimeter-wave radars, thus ensuring that the visual sensing range is consistent with the radar sensing range.
[0058] Step 203: Fuse the millimeter-wave radar data of the first and second detection areas with the images corresponding to each focal length.
[0059] It is understandable that sensor data fusion is based on time and space synchronization, meaning that the radar data and image data used for target fusion detection should be synchronized in both time and space. In this embodiment, synchronization is a relative concept; that is, as long as the temporal or spatial error between two sets of data is within an acceptable range, the two sets of data can be considered synchronized. In other words, the radar-sensing target can first be determined based on the millimeter-wave radar data from the first and second detection areas; then, the camera-sensing target can be determined based on the images corresponding to each focal length; finally, the radar-sensing target and the camera-sensing target can be synchronized in time and space.
[0060] In one embodiment, a calibration algorithm can be used to spatially synchronize radar-sensing targets and camera-sensing targets within the coverage area of each focal length of the spherical camera. After setting the focal lengths of the spherical camera, spatial synchronization between camera-sensing targets and radar-sensing targets can be achieved by calibrating the target position within each focal length range. Specifically, at the set focal length, the target's position will be displayed in the image captured by the spherical camera. Simultaneously, the millimeter-wave radar will detect the target's presence and provide information such as the target's distance and velocity. Through the calibration algorithm, the field of view of the spherical camera is matched with the detection area of the millimeter-wave radar to ensure that both can represent the target's position in the same coordinate system.
[0061] In one embodiment, in response to switching the focal length of the spherical camera, a timestamp synchronization algorithm can be used to synchronize the time of the radar-sensing target and the camera-sensing target. Time synchronization can be triggered by a signal sent after the zoom operation is completed, and then performed using a timestamp synchronization algorithm between the camera and the radar. Specifically, after the zoom PTZ camera completes its focal length adjustment, both the spherical camera and the millimeter-wave radar record timestamps when they capture data. By using the timestamp synchronization algorithm, the time data from the spherical camera and the millimeter-wave radar can be aligned.
[0062] As one possible implementation, the focal length of the spherical camera can be adaptively adjusted based on the speed of the targets within the detection range. That is, the speed of each target in the first and second detection areas can be acquired within a preset time period; based on the focal length switching frequency, the focal length of the spherical camera can be adjusted to acquire the image corresponding to each focal length.
[0063] In one embodiment, the focal length switching frequency can be set according to the lowest speed among the speeds of each target within a preset time period. For example, if the speed of each target is in the range of 80km / h to 120km / h within the preset time period, then the focal length switching frequency can be set according to 80km / h to ensure that images of all targets can be scanned.
[0064] In one embodiment, the back-to-back arrangement of two millimeter-wave radars may result in a sensing blind zone around the pole, making this area a sensing blind zone. Therefore, the millimeter-wave radar blind zone data within a preset range around the first base station can be obtained using the first and second millimeter-wave radars; the blind zone image within the preset range around the first base station can be obtained using a spherical camera; finally, the millimeter-wave radar data of the first and second detection areas, the images corresponding to each focal length, the millimeter-wave radar blind zone data, and the blind zone image are fused to avoid the situation of missing perception.
[0065] The method for fusing millimeter-wave radar data and images in high-speed scenarios disclosed in the above embodiments of this application employs a combination of two millimeter-wave radars arranged back-to-back and a variable-focus spherical camera to achieve target detection in highway scenarios. First, the two millimeter-wave radars acquire radar data from both directions of the highway. By adjusting the focal length of the spherical camera, visual perception data at each focal length can be obtained. Finally, the perception data from the millimeter-wave radars and the spherical camera are fused to obtain the final target detection result. Thus, the variable-focus spherical camera ensures that there are no blind spots at mid-to-long distances, thereby controlling the visual perception range to be consistent with the radar perception range and improving target detection accuracy. Furthermore, the arrangement of two millimeter-wave radars arranged back-to-back and the spherical camera to acquire radar data and images from both directions of the highway can reduce the number of roadside poles required, saving costs.
[0066] In this embodiment of the application, the sensing range of the two base stations can be set to an overlapping sensing area of about 20m. In this overlapping sensing area, the fused target detection data can be transmitted to achieve target detection in the entire range.
[0067] In one possible implementation of this application, a sensing overlap region is established between the first base station and the second base station, where the second base station is the next adjacent base station of the first base station, i.e., a downstream base station. By using the overlapping detection region between adjacent upstream and downstream base stations as a matching region, when the tracked target enters the matching region, an identity matching algorithm is executed to extend the single-point range perception to the entire range. Specifically, when the target moves into the sensing overlap region, the target detection result after fusion is sent to the second base station through the identity matching algorithm, so that the second base station can continue to perform fusion detection on the target based on the target detection result.
[0068] See Figure 3 The diagram shows a structural schematic of a device for fusing millimeter-wave radar data and images in a high-speed scenario, provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.
[0069] A device for fusing millimeter-wave radar data and images in high-speed scenarios may specifically include the following modules:
[0070] The acquisition module 301 is used to acquire millimeter-wave radar data in a first detection area via a first millimeter-wave radar, and to acquire millimeter-wave radar data in a second detection area via a second millimeter-wave radar. The first detection area is a road area along the first driving direction of the highway starting from the first base station, and the second detection area is a road area along the second driving direction of the highway starting from the first base station. The first driving direction is opposite to the second driving direction.
[0071] The processing module 302 is used to acquire the image corresponding to each focal length by adjusting the focal length of the spherical camera, wherein the range covered by each focal length of the spherical camera includes the first detection area and the second detection area.
[0072] The processing module 302 is also used to fuse the millimeter-wave radar data of the first detection area and the second detection area with the image corresponding to each focal length.
[0073] The millimeter-wave radar data and image fusion device for high-speed scenarios disclosed in the above embodiments of this application employs a combination of two millimeter-wave radars arranged back-to-back and a variable-focus spherical camera to achieve target detection in highway scenarios. First, the two millimeter-wave radars acquire radar data from both directions of the highway. By adjusting the focal length of the spherical camera, visual perception data at each focal length can be obtained. Finally, the perception data from the millimeter-wave radars and the spherical camera are fused to obtain the final target detection result. Thus, the variable-focus spherical camera ensures that there are no blind spots at medium to long distances, thereby controlling the visual perception range to be consistent with the radar perception range and improving target detection accuracy. Furthermore, the arrangement of two millimeter-wave radars arranged back-to-back and the spherical camera to acquire radar data and images from both directions of the highway can reduce the number of roadside poles required, saving costs.
[0074] Furthermore, in one possible implementation of this application embodiment, a focal length overlap region is provided between adjacent focal lengths of the above-mentioned spherical camera.
[0075] Furthermore, in one possible implementation of this application embodiment, the processing module 302 may specifically include the following units:
[0076] The first processing unit is used to determine the radar-sensing target based on the millimeter-wave radar data from the first detection area and the second detection area.
[0077] The second processing unit is used to determine the camera-sensing target based on the images corresponding to each focal length.
[0078] The third processing unit is used for spatiotemporal synchronization of radar-sensing targets and camera-sensing targets.
[0079] Furthermore, in another possible implementation of the embodiments of this application, the third processing unit described above may be used to: spatially synchronize radar-sensing targets and camera-sensing targets within the coverage area of each focal length of the spherical camera using a calibration algorithm; and temporally synchronize radar-sensing targets and camera-sensing targets using a timestamp synchronization algorithm in response to the operation of switching the focal length of the spherical camera.
[0080] Furthermore, in another possible implementation of this application embodiment, the processing module 302 is further configured to acquire millimeter-wave radar blind zone data within a preset range around the first base station using the first millimeter-wave radar and the second millimeter-wave radar; acquire blind zone images within a preset range around the first base station using a spherical camera; and fuse the millimeter-wave radar data of the first detection area and the second detection area, the images corresponding to each focal length, the millimeter-wave radar blind zone data, and the blind zone images.
[0081] Furthermore, in another possible implementation of this application embodiment, a sensing overlap area is provided between the first base station and the second base station, and the second base station is the next adjacent base station of the first base station.
[0082] Furthermore, in another possible implementation of this application embodiment, the processing module 302 is further configured to send the target detection result after target fusion to the second base station through an identity matching algorithm when the target moves into the perception overlap area, so that the second base station can continue to perform target fusion detection based on the target detection result.
[0083] Furthermore, in another possible implementation of this application embodiment, the processing module 302 is further configured to determine the coverage range of each focal length of the spherical camera based on the sensing range of the first millimeter-wave radar and the second millimeter-wave radar.
[0084] Furthermore, in another possible implementation of this application embodiment, the processing module 302 is further configured to acquire the speed of each target in the first detection area and the second detection area within a preset time period; update the focal length switching frequency of the spherical camera according to the speed of each target; and adjust the focal length of the spherical camera based on the focal length switching frequency to acquire the image corresponding to each focal length.
[0085] Furthermore, in another possible implementation of this application embodiment, the processing module 302 is further configured to update the focal length switching frequency of the spherical camera according to the speed of each target, including: determining the lowest speed among the speeds of each target; and updating the focal length switching frequency of the spherical camera according to the lowest speed.
[0086] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 of this embodiment includes: at least one processor 410 ( Figure 4 (Only one is shown in the diagram) a processor, a memory 420, and a computer program 421 stored in the memory 420 and executable on the at least one processor 410, wherein the processor 410 executes the computer program 421 to implement the steps in the above-described embodiment of the method for fusing millimeter-wave radar data and images in high-speed scenarios.
[0087] The electronic device 400 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 400 and does not constitute a limitation on electronic device 400. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.
[0088] The processor 410 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0089] In some embodiments, the memory 420 may be an internal storage unit of the electronic device 400, such as a hard disk or memory of the electronic device 400. In other embodiments, the memory 420 may be an external storage device of the electronic device 400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 400. Furthermore, the memory 420 may include both internal and external storage units of the electronic device 400. The memory 420 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 420 can also be used to temporarily store data that has been output or will be output.
[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0093] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0096] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0097] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the various method embodiments described above.
[0098] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for fusing millimeter-wave radar data and images in high-speed scenarios, characterized in that, A first base station applied to the roadside of a highway, the first base station including at least a first millimeter-wave radar, a second millimeter-wave radar, and a spherical camera, the first millimeter-wave radar and the second millimeter-wave radar being arranged back-to-back, and the spherical camera having multiple focal lengths; the method includes: The first millimeter-wave radar acquires millimeter-wave radar data within a first detection area, and the second millimeter-wave radar acquires millimeter-wave radar data within a second detection area. The first detection area is a road area along a first driving direction of the highway, starting from the first base station, and the second detection area is a road area along a second driving direction of the highway, starting from the first base station. The first driving direction is opposite to the second driving direction. By adjusting the focal length of the spherical camera, an image corresponding to each focal length is obtained, wherein the range covered by each focal length of the spherical camera includes the first detection area and the second detection area; The millimeter-wave radar data of the first detection area and the second detection area are fused with the images corresponding to each focal length.
2. The method according to claim 1, characterized in that, The spherical camera has an overlapping focal length region between adjacent focal lengths.
3. The method according to claim 1, characterized in that, The process of fusing millimeter-wave radar data from the first and second detection areas with images corresponding to each focal length includes: The radar-sensing target is determined based on the millimeter-wave radar data from the first detection area and the second detection area; The target to be perceived by the camera is determined based on the image corresponding to each of the aforementioned focal lengths; The radar-sensing target and the camera-sensing target are synchronized in time and space.
4. The method according to claim 3, characterized in that, The spatiotemporal synchronization of the radar-sensing target and the camera-sensing target includes: Within the coverage area of each focal length of the spherical camera, a calibration algorithm is used to spatially synchronize the radar-sensing target and the camera-sensing target; In response to the operation of switching the focal length of the spherical camera, a timestamp synchronization algorithm is used to synchronize the time of the radar-sensing target and the camera-sensing target.
5. The method according to claim 1, characterized in that, The method further includes: The first millimeter-wave radar and the second millimeter-wave radar are used to obtain millimeter-wave radar blind zone data within a preset range around the first base station; The spherical camera acquires images of the blind zone within a preset range around the first base station; The process of fusing millimeter-wave radar data from the first and second detection areas with images corresponding to each focal length includes: The millimeter-wave radar data of the first detection area and the second detection area, the image corresponding to each focal length, the millimeter-wave radar blind zone data, and the blind zone image are fused together.
6. The method according to claim 1, characterized in that, A sensing overlap area is set between the first base station and the second base station, and the second base station is the next adjacent base station of the first base station.
7. The method according to claim 6, characterized in that, The method further includes: When a target moves into the overlapping area of perception, the target detection result after fusion is sent to the second base station through an identity matching algorithm, so that the second base station can continue to perform fusion detection on the target based on the target detection result.
8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the sensing range of the first millimeter-wave radar and the second millimeter-wave radar, the coverage range of each focal length of the spherical camera is determined.
9. The method according to any one of claims 1-7, characterized in that, The method further includes: The velocity of each target in the first detection area and the second detection area is acquired within a preset time period; The focal length switching frequency of the spherical camera is updated according to the speed of each target. The step of acquiring an image corresponding to each focal length by adjusting the focal length of the spherical camera includes: Based on the focal length switching frequency, the focal length of the spherical camera is adjusted to obtain an image corresponding to each focal length.
10. The method according to any one of claims 9, characterized in that, The step of updating the focal length switching frequency of the spherical camera based on the speed of each target includes: Determine the lowest speed among the speeds of the various targets; The focal length switching frequency of the spherical camera is updated based on the minimum speed.
11. A device for fusing millimeter-wave radar data and images in high-speed scenarios, characterized in that, A first base station applied to the roadside of a highway, the first base station including at least a first millimeter-wave radar, a second millimeter-wave radar, and a spherical camera, the first millimeter-wave radar and the second millimeter-wave radar being arranged back-to-back, and the spherical camera having multiple focal lengths; the device includes: The acquisition module is used to acquire millimeter-wave radar data in a first detection area through the first millimeter-wave radar, and to acquire millimeter-wave radar data in a second detection area through the second millimeter-wave radar. The first detection area is a road area along a first driving direction of the highway starting from the first base station, and the second detection area is a road area along a second driving direction of the highway starting from the first base station. The first driving direction is opposite to the second driving direction. The processing module is used to acquire an image corresponding to each focal length by adjusting the focal length of the spherical camera, wherein the range covered by each focal length of the spherical camera includes the first detection area and the second detection area; The processing module is further configured to fuse the millimeter-wave radar data of the first detection area and the second detection area with the image corresponding to each focal length.
12. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.
13. A computer program product, characterized in that, The computer program product includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 10.