Vehicle data fusion method and system, electronic equipment and storage medium
By designing dynamically activated fusion modules and cache queue units in the autonomous driving system, the problem of data fusion processing delay in the prior art is solved, and more flexible and efficient data fusion processing is achieved, reducing functional delay.
Patent Information
- Application Number
- CN202311650462.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-04
- Publication Date
- 2025-06-06
AI Technical Summary
When existing autonomous driving systems use long fixed cycles to converge camera data and radar data, the cycle will increase, especially in complex scenarios, resulting in delays in perceived information processing.
A vehicle data fusion method is designed, and image information is obtained through the perception module and stored in the cache queue unit. If the fusion module is in a dormant state, it is activated for data fusion processing, and after the processing is completed, whether the cache queue unit has updated information, and if none is entered, it will enter a dormant state.
By dynamically activating the fusion module and generating trigger fusion processing based on the perceived information, resources can be fully utilized, functional delay can be reduced, and only radar detection data can be fused for processing when the perceived information is not obtained.
Smart Images
Figure CN120105320A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a vehicle data fusion method, system, electronic device and storage medium. Background Art
[0002] In recent years, autonomous driving and assisted driving technologies have shown a booming momentum. Vehicle autonomous driving systems and assisted driving systems usually use a variety of sensors to obtain information about the surrounding environment. These sensors include cameras, lidar, millimeter-wave radar, ultrasonic sensors, and GPS. These sensors can obtain data such as images, distance, speed, and location around the vehicle. It is very important to fuse the data of these sensors because each sensor has its own advantages and limitations. Fusion of data can improve the system's perception of the surrounding environment and reduce the possibility of misjudgment and missed judgment.
[0003] When fusing camera data and radar data, the image data obtained by the camera is first identified to obtain perception information, and then the perception information and radar detection data are fused. At present, the fusion process is usually performed in a relatively long fixed cycle. However, the perception information cycle is uncertain. In simple scenes, the cycle is short, and in complex scenes, the cycle will increase. If the fusion process is performed in a relatively long fixed cycle, there will sometimes be a long delay. Summary of the invention
[0004] The present invention is completed in order to solve the above-mentioned problems, and its purpose is to provide a vehicle data fusion method, system, electronic device and storage medium, which can fully utilize resources and reduce functional delays.
[0005] According to one aspect of the present invention, a vehicle data fusion method is provided, the vehicle comprising a vehicle-mounted camera, a vehicle-mounted radar, a perception module, a cache queue unit and a fusion module, the vehicle-mounted camera acquiring image information around the vehicle, the vehicle-mounted radar acquiring radar detection information, comprising the following steps: the perception module performs recognition processing based on the image information acquired by the vehicle-mounted camera, acquires perception information, and stores the perception information in a cache queue unit; after the perception information is stored in the cache queue unit, if the fusion module is in a dormant state, the fusion module is activated; after the fusion module is activated, the fusion module acquires the perception information and the radar detection information, performs fusion processing on the perception information and the radar detection information, and outputs a fusion result; and after the fusion processing is completed, the fusion module queries the cache queue unit whether there is updated perception information, and if not, enters a dormant state.
[0006] Preferably, the vehicle data fusion method further comprises: if the fusion module maintains a dormant state for a preset time, the fusion module activates itself, obtains the radar detection information, performs fusion processing on the radar detection information, and outputs a fusion result.
[0007] Preferably, the vehicle-mounted radar includes a forward radar and multiple corner radars.
[0008] Preferably, the perception information includes: lane line information, traffic sign target information, vehicle target information, pedestrian target information and other obstacle target information.
[0009] Preferably, the cache queue unit is set with a cache quantity of perception information.
[0010] Preferably, the preset duration is set to 60ms.
[0011] According to another aspect of the present invention, a vehicle data fusion system is provided, which includes a vehicle-mounted camera, a vehicle-mounted radar, a cache queue unit, a perception module and a fusion module. The vehicle-mounted camera obtains image information around the vehicle, and the vehicle-mounted radar obtains radar detection information. The perception module performs recognition processing based on the image information obtained by the vehicle-mounted camera, obtains perception information, and stores the perception information in the cache queue unit. After the perception information is stored in the cache queue unit, if the fusion module is in a dormant state, the fusion module is activated. After the fusion module is activated, the fusion module obtains the perception information and the radar detection information, and performs fusion processing on the perception information and the radar detection information, and outputs a fusion result. After the fusion processing is completed, the fusion module queries the cache queue unit whether there is updated perception information. If not, the fusion module enters a dormant state.
[0012] According to another aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executes the method described in one of the above aspects.
[0013] According to another aspect of the present invention, a computer storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded by a processor and executes the method according to the above aspect of claim 1. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a structural block diagram of a vehicle data fusion system provided by an embodiment of the present invention.
[0015] Figure 2It is a flow chart of a vehicle data fusion method provided in an embodiment of the present invention.
[0016] Figure 3 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0017] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0018] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.
[0019] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.
[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.
[0021] Figure 1 2 is a structural block diagram of a vehicle data fusion system 200 provided in an embodiment of the present invention.
[0022] The vehicle data fusion system 200 involved in this embodiment includes a vehicle-mounted camera 201, a vehicle-mounted radar 202, a perception module 203, a cache queue unit 204 and a fusion module 205. The vehicle-mounted camera 201 obtains image information around the vehicle 1, and the vehicle-mounted radar 202 obtains radar detection information.
[0023] The vehicle-mounted camera 201 may be a forward-facing high-definition single-lens camera, which is installed directly behind the front windshield of the vehicle and is used to obtain image information around the vehicle 1. The vehicle-mounted radar 202 may include a forward-facing long-range millimeter-wave radar and multiple corner radars. The vehicle-mounted radar 202 obtains radar detection information, which includes information such as the distance, speed, and direction of multiple targets (other vehicles, pedestrians, obstacles, etc.) around the vehicle 1. In addition, in some embodiments, the vehicle-mounted radar may also include a laser radar.
[0024] The perception module 203 performs recognition processing based on the image information collected by the vehicle-mounted camera 201, and is used to identify lane lines, traffic signs, targets (vehicles, pedestrians and other obstacles), etc., so as to obtain perception information, which includes lane line information, traffic sign target information, vehicle target information, pedestrian target information and other obstacle target information. Then, the perception module 203 can store the recognized perception information into the cache queue unit 204. In addition, the number of cached perception information in the cache queue unit 204 can be pre-set so that the fusion processing can obtain appropriate perception information. After the perception information is stored in the cache queue unit 204, if the fusion module 205 is in a dormant state, the fusion module 205 is activated.
[0025] After the fusion module 205 is activated, it starts to perform fusion processing. First, it obtains the perception information from the cache queue unit 204 and the radar detection information from the vehicle-mounted radar 202, and performs fusion processing on the perception information and the radar detection information to output the fusion result. After the fusion processing is completed, the fusion module 205 queries the cache queue unit 204 whether there is updated perception information. If not, it enters a dormant state. In other words, after the fusion processing is completed, the fusion module 205 will query whether there is new perception information stored in the cache queue unit 204 during the fusion processing. If not, the fusion module 205 will temporarily enter a dormant state.
[0026] Figure 2 It is a flowchart of the vehicle data fusion method provided by an embodiment of the present invention. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment).
[0027] The vehicle to which the vehicle data fusion method provided in this embodiment is applicable includes a vehicle camera 201, a vehicle radar 202, a perception module 203, a cache queue unit 204 and a fusion module 205. The vehicle camera 201 obtains image information around the vehicle 1, and the vehicle radar 202 obtains radar detection information. Figure 2 As shown, the vehicle data fusion method provided in this embodiment includes the following steps.
[0028] S101: The perception module 203 performs recognition processing based on the image information obtained by the vehicle-mounted camera 201, obtains perception information, and stores the perception information in the cache queue unit 204.
[0029] The vehicle camera 201 acquires image information around the vehicle 1, and the perception module 203 performs recognition processing based on the image information acquired by the vehicle camera 201, thereby obtaining perception information including lane line information, traffic sign target information, vehicle target information, pedestrian target information, and other obstacle target information, etc. The perception module 203 stores the perception information in the cache queue unit 204.
[0030] As described above, the amount of cached perception information in the cache queue unit 204 may be preset so as to obtain appropriate perception information through fusion processing.
[0031] S102: After the perception information is stored in the cache queue unit 204, if the fusion module 205 is in a dormant state, the fusion module 205 is activated.
[0032] After the perception information is stored in the cache queue unit 204 , it is necessary to confirm the current state of the fusion module 205 , whether it is in a dormant state. If the fusion module 205 is in a dormant state, the fusion module 205 is activated.
[0033] S103: After the fusion module 205 is activated, the fusion module 205 obtains the perception information and the radar detection information, performs fusion processing on the perception information and the radar detection information, and outputs a fusion result.
[0034] After the fusion module 205 is activated, the fusion process begins. First, the perception information is obtained from the cache queue unit 204, and the radar detection information is obtained from the vehicle-mounted radar 202. The perception information and the radar detection information are fused and the fusion result is output.
[0035] S104: After the fusion process is completed, the fusion module 205 queries whether the cache queue unit has updated perception information. If not, it enters a dormant state.
[0036] After each fusion process is completed, the fusion module 205 will query whether the cache queue unit 204 has updated perception information, that is, whether new perception information is stored in the cache queue unit 204 during the fusion process. If not, it will enter a dormant state and wait for the next activation.
[0037] In addition, in order to prevent the fusion module from being blocked and waiting due to failure to obtain perception information, the vehicle data fusion method of this embodiment may further include: if the fusion module 205 maintains a dormant state for a preset period of time, the fusion module 205 activates itself, obtains radar detection information, performs fusion processing on the radar detection information, and outputs a fusion result.
[0038] That is, the timing can be started from the time when the fusion module 205 just enters the sleep state. If the fusion module 205 maintains the sleep state for a preset time, it means that no perception information is stored in the cache queue unit 204 within the preset time. At this time, in order to prevent the fusion module from blocking and waiting due to failure to obtain perception information, only radar detection data can be obtained for fusion. Therefore, if the fusion module 205 maintains the sleep state for a preset time, the fusion module 205 activates itself and only obtains radar detection information, performs fusion processing on the radar detection information, and outputs the fusion result. In this embodiment, the preset time is set to 60ms, but it can also be set to other values.
[0039] According to the present invention, the generation of perception information can be used to trigger fusion processing, so that resources can be fully utilized and functional delay can be reduced. In addition, when perception information cannot be obtained, only radar detection data can be fused to output the fusion result.
[0040] Figure 3 is a structural block diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, the present invention also provides an electronic device 300, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by the processor and executes the method described in the above embodiment.
[0041] The present invention also provides a computer storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the method described in the above embodiment.
[0042] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.
[0043] Those skilled in the art should be able to appreciate that the modules, units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0044] Although the present invention has been described with reference to the current specific embodiments, those skilled in the art should recognize that the scope of the invention involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.
Claims
1. A vehicle data fusion method, the vehicle comprising a vehicle-mounted camera, a vehicle-mounted radar, a perception module, a cache queue unit and a fusion module, the vehicle-mounted camera acquires image information around the vehicle, the vehicle-mounted radar acquires radar detection information, It is characterized in that The steps include: The perception module performs recognition processing based on the image information acquired by the vehicle-mounted camera, acquires perception information, and stores the perception information in a cache queue unit; After the perception information is stored in the cache queue unit, if the fusion module is in a dormant state, activating the fusion module; After the fusion module is activated, the fusion module obtains the perception information and the radar detection information, performs fusion processing on the perception information and the radar detection information, and outputs a fusion result; as well as After the fusion process is completed, the fusion module queries the cache queue unit whether there is updated perception information, and if not, enters a dormant state.
2. The vehicle data fusion method according to claim 1, It is characterized in that Further including: If the fusion module maintains the dormant state for a preset time period, the fusion module activates itself, obtains the radar detection information, performs fusion processing on the radar detection information, and outputs a fusion result.
3. The vehicle data fusion method according to claim 1 or 2, It is characterized in that The vehicle-mounted radar includes a forward radar and multiple corner radars.
4. The vehicle data fusion method according to claim 1 or 2, It is characterized in that The perception information includes: lane line information, traffic sign target information, vehicle target information, pedestrian target information and other obstacle target information.
5. The vehicle data fusion method according to claim 1 or 2, It is characterized in that The cache queue unit sets the number of caches with sensing information.
6. The vehicle data fusion method according to claim 2, It is characterized in that The preset duration is set to 60ms.
7. A vehicle data fusion system, comprising a vehicle camera, a vehicle radar, a cache queue unit, a perception module and a fusion module, wherein the vehicle camera acquires image information around the vehicle, and the vehicle radar acquires radar detection information. It is characterized in that The perception module performs recognition processing based on the image information obtained by the vehicle-mounted camera, obtains perception information, and stores the perception information in a cache queue unit. After the perception information is stored in the cache queue unit, if the fusion module is in a dormant state, the fusion module is activated. After the fusion module is activated, the fusion module obtains the perception information and the radar detection information, and performs fusion processing on the perception information and the radar detection information, and outputs a fusion result. After the fusion processing is completed, the fusion module queries the cache queue unit whether there is updated perception information. If not, the fusion module enters a dormant state.
8. An electronic device, It is characterized in that The electronic device comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by the processor and executes the method according to any one of claims 1 to 6.
9. A computer storage medium, It is characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by a processor and executes the method according to any one of claims 1 to 6.