Multi-sensor fusion method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-sensor fusion method and device, electronic equipment and a storage medium, the method is applied to a fusion module, and the fusion method comprises the following steps: in response to measurement information in each sensor, matching an obstacle tracking trajectory to obtain a matching result; and obtaining a fusion result of the plurality of sensors for each obstacle tracking trajectory in the matching result according to modes of type fusion, shape fusion and motion fusion, the type fusion adopting an asynchronous mode, and the shape fusion and the motion fusion adopting a synchronous mode. By means of the method, the advantages of synchronous fusion and asynchronous fusion are fused, continuous information can be reserved, and discrete information can be obtained.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multi-sensor fusion method, device, electronic device, and storage medium. Background Art

[0002] The fusion module in autonomous driving refers to technology that integrates and processes data from multiple sensors (such as lidar, millimeter-wave radar, and cameras) to enhance the perception, decision-making, and execution capabilities of autonomous driving systems. The fusion module plays a crucial role in autonomous driving systems. By fusing multi-sensor information, it improves the autonomous vehicle's ability to understand and respond to its surroundings.

[0003] In related technologies, both synchronous fusion and asynchronous fusion have their own problems, such as inference time and fusion accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a multi-sensor fusion method, device, electronic device, and storage medium to combine the advantages of synchronous and asynchronous fusion, improve fusion accuracy, and reduce inference time.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a multi-sensor fusion method, wherein the fusion method is applied to a fusion module, and the fusion method includes:

[0007] In response to the measurement information from each sensor, matching the obstacle tracking trajectory to obtain a matching result; and

[0008] In the matching results, each obstacle tracking trajectory is sequentially fused according to the type fusion, shape fusion and motion fusion methods to obtain the fusion results of multiple sensors.

[0009] The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

[0010] In some embodiments, matching the obstacle tracking trajectory in response to the measurement information from each sensor to obtain a matching result includes:

[0011] The measurement information received from each sensor is stored in the corresponding cache queue;

[0012] determining a primary sensor among the plurality of sensors;

[0013] In response to the measurement information of the main sensor, obtaining a frame with a timestamp closest to the main sensor from the buffer queue of each sensor;

[0014] According to the frame with the latest timestamp of the main sensor, the obstacle tracking trajectory is polled and matched with the measurement information in each sensor to obtain the matching result.

[0015] The matching result includes one of the following: matched information of the obstacle tracking trajectory and the measurement information, unmatched obstacle tracking trajectory information, and unmatched measurement information.

[0016] In some embodiments, matching the obstacle tracking trajectory to obtain a matching result in response to the measurement information from each sensor further includes:

[0017] After polling the obstacle tracking trajectory and matching it with the measurement information in each sensor, generating a new obstacle tracking trajectory for the measurement information that is not matched;

[0018] The obtained matching result includes newly generated obstacle tracking trajectory information.

[0019] In some embodiments, the type fusion specifically includes:

[0020] Traversing each obstacle tracking trajectory and the measurement information in the matched sensor to obtain a cumulative probability of the obstacle type;

[0021] According to the accumulated probability, a new set of obstacle type probabilities is obtained after fusing each matched sensor;

[0022] After polling all matched sensor measurement information, a set of the latest obstacle type probabilities is obtained, and the obstacle type with the highest probability is selected to determine the type fusion result.

[0023] In some embodiments, the shape fusion specifically includes:

[0024] Traverse each obstacle tracking trajectory to perform shape fusion including at least obstacle Yaw information and obstacle Size information;

[0025] Obtain all matching information of each obstacle tracking trajectory to obtain a matching value, and obtain a fusion result of the measurement value based on the matching value;

[0026] The fusion results of the measurement values ​​are input into the filtering algorithms corresponding to the Yaw information and Size information respectively, and the filtered Yaw information and Size information are obtained for updating.

[0027] In some embodiments, the motion fusion specifically includes:

[0028] Traversing each obstacle tracking trajectory to perform motion fusion including at least obstacle position, obstacle speed, and obstacle outline information;

[0029] Obtain all matching information of each obstacle tracking trajectory and obtain the fusion result of the measurement value;

[0030] The fusion result including the measured position and speed is input into the corresponding filtering algorithm to obtain the filtered position and speed information for updating.

[0031] In some embodiments, updating the obstacle profile information further includes:

[0032] The obstacle contour information is directly updated with the fusion result according to the scene selection;

[0033] Alternatively, the obstacle contour information is generated according to the updated position information, Yaw information, and Size information.

[0034] In a second aspect, an embodiment of the present application further provides a multi-sensor fusion device, which is applied to a fusion module and includes:

[0035] a matching module, configured to match the obstacle tracking trajectory in response to the measurement information from each sensor to obtain a matching result; and

[0036] A fusion processing module is used to obtain the fusion results of multiple sensors by performing type fusion, shape fusion and motion fusion on each obstacle tracking trajectory in the matching results.

[0037] The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, enable the processor to perform the above method.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0040] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: in response to the measurement information in each sensor, the obstacle tracking trajectory Track is matched to obtain a matching result. In the matching result, each obstacle tracking trajectory is sequentially fused according to the type fusion, shape fusion and motion fusion methods, and finally the fusion results of multiple sensors are obtained. And in type fusion, an asynchronous fusion method is adopted, and in shape fusion or motion fusion, a synchronous fusion method is adopted. Through the above method, the respective advantages of synchronous and asynchronous fusion are combined. While obtaining all the matching information of the current frame, it is possible to retain the advantages of asynchronous fusion for discrete information (such as Type type information), and to retain the advantages of synchronous fusion for continuous information (such as Yaw information, Size information, position information, speed information, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 This is a schematic diagram of the Track matching process in the multi-sensor fusion method in an embodiment of the present application;

[0043] Figure 2 Schematic diagram of the multi-sensor fusion method in an embodiment of the present application;

[0044] Figure 3 Schematic diagram of the principle of the multi-sensor fusion method in the embodiment of the present application;

[0045] Figure 4 This is a schematic structural diagram of a multi-sensor fusion device in an embodiment of the present application;

[0046] Figure 5 This is a schematic structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0049] like Figure 1As shown, sensor 1, sensor 2, and sensor 3 are used to represent various types of sensors on the vehicle, and there is no specific limit on the number of sensors. The obstacle tracking tracks Track1, Track2, and Track3 are used to represent the trajectory information obtained from the obstacle tracking results. In actual scenarios, by collecting all sensor measurements that need to be fused in the current fusion processing cycle, and then matching each sensor measurement value with Track, all matching measurement information of Track is obtained; then, different perception information of the measurement values, such as Type, Yaw, Size, position, speed, Polygon (contour information), etc., are fused to obtain the result of the measurement value fusion.

[0050] The present application embodiment provides a multi-sensor fusion method, such as Figure 2 As shown, a schematic flow chart of a multi-sensor fusion method according to an embodiment of the present application is provided, wherein the method comprises at least the following steps S210 to S220:

[0051] Step S210 : In response to the measurement information from each sensor, the obstacle tracking trajectory is matched to obtain a matching result.

[0052] In the field of autonomous driving, especially large autonomous vehicles, a diverse suite of sensors is installed, including long-range lidar, short-range blind spot radar, millimeter-wave radar, and cameras. Different sensor types have varying perception capabilities, specifically for different sensory information such as position, velocity, yaw angle, polygon (contour information), and type. Consequently, perception capabilities vary across different areas, creating a complex situation. A suitable fusion framework is required to effectively integrate the measurement information from each sensor. Fusion generally involves first collecting measurement information from each sensor and then performing the fusion.

[0053] The fusion module further matches all sensor measurements required for fusion in the current fusion processing cycle with the obstacle tracking track to obtain a matching result. If a measurement does not match a track, a new obstacle tracking track is created. For obstacle tracking tracks that do not match a measurement, the state of the track remains unchanged.

[0054] In step S220, each obstacle tracking trajectory in the matching result is sequentially fused according to the type fusion, shape fusion, and motion fusion methods to obtain fusion results of multiple sensors, wherein the type fusion is fused asynchronously, and the shape fusion and the motion fusion are fused synchronously.

[0055] The fusion module combines the different sensor information from the measurements to generate a fused result. It then performs filtering updates or other processing on the different information to generate the fused obstacle tracking trajectory. Finally, it performs track management on the obstacle tracking trajectory. To achieve the best fusion results from multiple sensors, it is best to perform type fusion first, then shape fusion, and finally motion fusion. If shape fusion is used first, the types may not match, resulting in large errors.

[0056] As you can understand, fusion frameworks are categorized into synchronous and asynchronous fusion. The synchronous fusion framework matches all sensor measurements with the obstacle tracking track. After obtaining all sensor measurements that match the obstacle tracking track, the system fuses the measurements and performs a final filter update. Synchronous fusion involves matching all sensors, fusing the measurements, and performing tracking filtering.

[0057] The advantages of using a synchronous fusion architecture are simplicity and reduced time consumption. Based on prior information, some measurement values ​​can be discarded, reducing the impact on the fusion results. For example, for continuous information like position, since the position of an obstacle detected by millimeter wave sensors is generally not the obstacle's center, when both millimeter wave sensors and lidar sensors detect the same obstacle, the millimeter wave position information can be discarded in favor of the lidar position information. However, the disadvantage of using a synchronous fusion architecture is that for discrete values ​​such as target types, information derived through probabilistic calculations is difficult to obtain through direct selection or weighting.

[0058] It can be understood that the asynchronous fusion framework matches each sensor measurement value separately, and then filters and updates it. By adjusting the filter parameters, the weight of each sensor measurement value is adjusted to achieve fusion.

[0059] The advantage of using an asynchronous fusion architecture is that it enables automatic fusion of perception information by filtering, updating, and adjusting parameters for each sensor's measurement information. For example, discrete information like type is difficult to fuse directly. Visual and lidar perception types each have their own strengths and weaknesses, and their perception areas do not completely overlap. Using asynchronous fusion, which calculates cumulative probabilities based on type confidence, can better achieve type fusion. However, asynchronous fusion only obtains matching information between the obstacle tracking trajectory and the current and previous measurements. The matching relationship with the remaining sensor measurements is unknown. Therefore, information from each sensor must be fused, which undoubtedly includes some information with clear perception differences, affecting fusion accuracy.

[0060] Through the above method, not only the respective advantages of synchronous and asynchronous fusion are combined, but also both discrete information and continuous information can be retained while obtaining all matching information of the current frame.

[0061] In one embodiment of the present application, matching the obstacle tracking trajectory in response to measurement information from each sensor to obtain a matching result includes: storing the received measurement information from each sensor in a corresponding cache queue; determining a master sensor among multiple sensors; obtaining, from the cache queue of each sensor, a frame closest to the timestamp of the master sensor in response to the measurement information of the master sensor; and polling the obstacle tracking trajectory and matching it with the measurement information from each sensor based on the frame closest to the timestamp of the master sensor to obtain a matching result, where the matching result includes one of the following: matched information between the obstacle tracking trajectory and measurement information, unmatched obstacle tracking trajectory information, and unmatched measurement information.

[0062] like Figure 3 As shown, the fusion module stores multiple frames of data from each sensor and extracts the frame closest to the timestamp of the master sensor. The fusion module receives measurement information from each sensor and stores it in its own buffer queue. Because the arrival times of each sensor in the buffer queue vary, a single sensor is selected as the master sensor and fusion is performed at the master sensor's frequency. The fusion frequency is the lidar sensor, and the frame with the closest timestamp is selected.

[0063] like Figure 3 As shown in the figure, when the fusion module receives sensor measurement information, it retrieves the frame closest to the master sensor's timestamp from each sensor's buffer queue. The fusion module then sequentially polls all obstacle tracking tracks and matches them with each sensor's measurement information. This module obtains information about matched obstacle tracking tracks and measurement values, as well as information about unmatched obstacle tracking tracks and unmatched measurement values ​​(for example, newly detected obstacles).

[0064] In one embodiment of the present application, matching the obstacle tracking trajectory in response to the measurement information in each sensor to obtain a matching result further includes: after polling the obstacle tracking trajectory and matching it with the measurement information in each sensor, generating a new obstacle tracking trajectory for the measurement information that did not match; the obtained matching result includes the newly generated obstacle tracking trajectory information.

[0065] After each round of polling, a new obstacle tracking track is generated for the unmatched measurement information that meets certain conditions. The newly generated obstacle tracking track can participate in the matching of the next round of polling. Ultimately, all matching information and all obstacle tracking track information are obtained, including the newly generated obstacle tracking track.

[0066] In one embodiment of the present application, the type fusion specifically includes: traversing each obstacle tracking trajectory and the measurement information in the matched sensors to obtain a cumulative probability of the obstacle type; fusing each matched sensor based on the cumulative probability to obtain a new set of obstacle type probabilities; after polling the measurement information of all matched sensors, obtaining a set of the latest obstacle type probabilities, and selecting the obstacle type with the highest probability.

[0067] like Figure 3 As shown, each obstacle tracking track is traversed for type fusion. Specifically, the measurement information from all matching sensors is traversed for each obstacle tracking track, and the cumulative probability of the type is calculated using probability calculation. After fusing each matching sensor, a new set of type probabilities is obtained. After polling all matching sensor measurements, the latest set of type probabilities is obtained. The type with the highest probability is selected, and type fusion is performed asynchronously. For example, if the obstacle tracking track matches four sensors, each sensor can be filtered to obtain the obstacle type, and fusion is completed during the filtering process.

[0068] It is understood that probability calculation may adopt methods including but not limited to DS inference method, Bayesian probability statistics, etc.

[0069] In one embodiment of the present application, the shape fusion specifically includes: traversing each obstacle tracking track to perform shape fusion including at least the obstacle Yaw information and the obstacle Size information; obtaining all matching information of each obstacle tracking track to obtain a matching value, and obtaining a fusion result of the measurement value based on the matching value; inputting the fusion result of the measurement value into the filtering algorithm corresponding to the Yaw information and the Size information respectively, obtaining the filtered Yaw information and Size information for updating.

[0070] like Figure 3As shown, shape fusion is performed synchronously, including but not limited to Yaw and Size information. Shape fusion is performed by traversing each obstacle tracking track. This involves obtaining all matching information for each track, weighting each matching value or directly selecting it, and obtaining a fused measurement result. This fused measurement result is then fed into the corresponding Yaw and Size filtering algorithms, resulting in an update after filtering. For example, given matching information from all sensors, if the laser sensor has a higher confidence level, the laser will be used directly, fusing the information first and then performing a filtering operation.

[0071] In one embodiment of the present application, the motion fusion specifically includes: traversing each obstacle tracking trajectory to perform motion fusion including at least obstacle position, obstacle speed, and obstacle contour information; obtaining all matching information of each obstacle tracking trajectory to obtain a fusion result of the measurement value; inputting the fusion result including the measurement value position and speed into the corresponding filtering algorithm to obtain the filtered position and speed information for updating.

[0072] like Figure 3 As shown, motion fusion is performed synchronously, including but not limited to position, velocity, and polygon (contour) information. Each track is traversed for motion fusion. This involves obtaining all matching information for each track, weighting each matching value or directly selecting it, and obtaining a fused measurement result. The fused position and velocity results are then fed into a filtering algorithm, resulting in the filtered position and velocity information for update.

[0073] In one embodiment of the present application, updating the obstacle contour information further includes: directly updating the obstacle contour information with a fusion result according to scene selection; or, generating the obstacle contour information according to the updated position information, Yaw information, and Size information.

[0074] Note that polygons (contour information) are special and do not require filtering, so the sensor fusion results are used directly for updates. Alternatively, Yaw and Size information can be used. Depending on the scenario, you can choose to directly update with the fusion results or generate updated position, Yaw, and Size information.

[0075] like Figure 3 As shown in the figure, after completing the above operations, all basic information of the obstacle tracking track (Type, shape, motion, ID, period) has been updated, and then various track management algorithms can be performed, including but not limited to dynamic and static recognition, Denoise, deduplication, residence, etc.

[0076] The embodiment of the present application also provides a multi-sensor fusion device 400, such as Figure 4 , a schematic structural diagram of a multi-sensor fusion device according to an embodiment of the present application is provided. The multi-sensor fusion device 200 includes at least: a matching module 410 and a fusion processing module 420, wherein:

[0077] In one embodiment of the present application, the matching module 410 is specifically configured to: respond to measurement information from each sensor, match the obstacle tracking trajectory to obtain a matching result.

[0078] In the field of autonomous driving, especially large autonomous vehicles, a diverse suite of sensors is installed, including long-range lidar, short-range blind spot radar, millimeter-wave radar, and cameras. Different sensor types have varying perception capabilities, specifically for different sensory information such as position, velocity, yaw angle, polygon (contour information), and type. Consequently, perception capabilities vary across different areas, creating a complex situation. A suitable fusion framework is required to effectively integrate the measurement information from each sensor. Fusion generally involves first collecting measurement information from each sensor and then performing the fusion.

[0079] The fusion module further matches all sensor measurements required for fusion in the current fusion processing cycle with the obstacle tracking track to obtain a matching result. If a measurement does not match a track, a new obstacle tracking track is created. For obstacle tracking tracks that do not match a measurement, the state of the track remains unchanged.

[0080] In one embodiment of the present application, the fusion processing module 420 is specifically used to: obtain the fusion results of multiple sensors in the matching results for each obstacle tracking trajectory in turn according to the type fusion, shape fusion and motion fusion methods, wherein the type fusion is fused asynchronously, and the shape fusion and the motion fusion are fused synchronously.

[0081] The fusion module perceives different information of the measurement values, obtains the result of the measurement value fusion, and then performs filtering updates or other processing on the different information to obtain the fused obstacle tracking trajectory track; finally, the obstacle tracking trajectory track is managed accordingly.

[0082] As you can understand, fusion frameworks are categorized into synchronous and asynchronous fusion. The synchronous fusion framework matches all sensor measurements with the obstacle tracking track. After obtaining all sensor measurements that match the obstacle tracking track, the system fuses the measurements and performs a final filter update. Synchronous fusion involves matching all sensors, fusing the measurements, and performing tracking filtering.

[0083] The advantages of using a synchronous fusion architecture are simplicity and reduced time consumption. Based on prior information, some measurement values ​​can be discarded, reducing the impact on the fusion results. For example, for continuous information like position, since the position of an obstacle detected by millimeter wave sensors is generally not the obstacle's center, when both millimeter wave sensors and lidar sensors perceive the same obstacle, the millimeter wave position information can be discarded in favor of the lidar position information. However, the disadvantage of using a synchronous fusion architecture is that for discrete values ​​such as target type, information derived through probabilistic calculations is difficult to obtain through direct selection or weighting.

[0084] It can be understood that the asynchronous fusion framework matches each sensor measurement value separately, and then filters and updates it. By adjusting the filter parameters, the weight of each sensor measurement value is adjusted to achieve fusion.

[0085] The advantage of using an asynchronous fusion architecture is that it enables automatic fusion of perceptual information by filtering, updating, and adjusting parameters for each sensor's measurement information. For example, discrete information such as type is difficult to fuse directly. Visual and lidar perception types each have their own strengths and weaknesses, and their perception areas do not completely overlap. Using asynchronous fusion, which calculates cumulative probabilities based on type confidence, can better achieve type fusion. However, asynchronous fusion only obtains matching information between the track and the current measurement value and previous measurements. The matching relationship with the remaining sensor measurements is unknown. Therefore, information from each sensor must be fused, which inevitably results in some information with clearly poor perception being fused, affecting fusion accuracy.

[0086] In one embodiment of the present application, the matching module 410 is further configured to:

[0087] The measurement information received from each sensor is stored in the corresponding cache queue;

[0088] determining a primary sensor among the plurality of sensors;

[0089] In response to the measurement information of the main sensor, obtaining a frame with a timestamp closest to the main sensor from the buffer queue of each sensor;

[0090] According to the frame with the latest timestamp of the main sensor, the obstacle tracking trajectory is polled and matched with the measurement information in each sensor to obtain the matching result.

[0091] The matching result includes one of the following: matched information of the obstacle tracking trajectory and the measurement information, unmatched obstacle tracking trajectory information, and unmatched measurement information.

[0092] In one embodiment of the present application, the matching module 410 is further configured to:

[0093] After polling the obstacle tracking trajectory and matching it with the measurement information in each sensor, generating a new obstacle tracking trajectory for the measurement information that is not matched;

[0094] The obtained matching result includes a newly generated obstacle tracking trajectory.

[0095] In one embodiment of the present application, the type fusion specifically includes:

[0096] Traversing each obstacle tracking trajectory and the measurement information in the matched sensor to obtain a cumulative probability of the obstacle type;

[0097] According to the accumulated probability, a new set of obstacle type probabilities is obtained after fusing each matched sensor;

[0098] After polling all matched sensor measurement information, a set of the latest obstacle type probabilities is obtained, and the obstacle type with the highest probability is selected.

[0099] In one embodiment of the present application, the shape fusion specifically includes:

[0100] Traverse each obstacle tracking trajectory to perform shape fusion including at least obstacle Yaw information and obstacle Size information;

[0101] Obtain all matching information of each obstacle tracking trajectory to obtain a matching value, and obtain a fusion result of the measurement value based on the matching value;

[0102] The fusion results of the measurement values ​​are input into the filtering algorithms corresponding to the Yaw information and Size information respectively, and the filtered Yaw information and Size information are obtained for updating.

[0103] In one embodiment of the present application, the motion fusion specifically includes:

[0104] Traversing each obstacle tracking trajectory to perform motion fusion including at least obstacle position, obstacle speed, and obstacle outline information;

[0105] Obtain all matching information of each obstacle tracking trajectory and obtain the fusion result of the measurement value;

[0106] The fusion result including the measured position and speed is input into the corresponding filtering algorithm to obtain the filtered position and speed information for updating.

[0107] In one embodiment of the present application, updating the obstacle profile information further includes:

[0108] The obstacle contour information is directly updated with the fusion result according to the scene selection;

[0109] Alternatively, the obstacle contour information is generated according to the updated position information, Yaw information, and Size information.

[0110] It can be understood that the above-mentioned multi-sensor fusion device can implement the various steps of the multi-sensor fusion method provided in the aforementioned embodiment. The relevant explanations about the multi-sensor fusion method are applicable to the multi-sensor fusion device and will not be repeated here.

[0111] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0112] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0113] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0114] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a multi-sensor fusion device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0115] In response to the measurement information from each sensor, matching the obstacle tracking trajectory to obtain a matching result; and

[0116] In the matching results, each obstacle tracking trajectory is sequentially fused according to the type fusion, shape fusion and motion fusion methods to obtain the fusion results of multiple sensors.

[0117] The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

[0118] The above application Figure 2 The methods performed by the multi-sensor fusion device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0119] The electronic device may also perform Figure 2Method for executing the multi-sensor fusion device in the Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0120] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 2 The method performed by the multi-sensor fusion device in the embodiment shown is specifically used to perform:

[0121] In response to the measurement information from each sensor, matching the obstacle tracking trajectory to obtain a matching result; and

[0122] In the matching results, each obstacle tracking trajectory is sequentially fused according to the type fusion, shape fusion and motion fusion methods to obtain the fusion results of multiple sensors.

[0123] The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

[0124] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0129] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0130] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A multi-sensor fusion method, wherein: Applied to the fusion module, the fusion method includes: In response to the measurement information from each sensor, matching the obstacle tracking trajectory to obtain a matching result; and In the matching results, each obstacle tracking trajectory is sequentially fused according to the type fusion, shape fusion and motion fusion methods to obtain the fusion results of multiple sensors. The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

2. The method according to claim 1, wherein: The step of matching the obstacle tracking trajectory to obtain a matching result in response to the measurement information from each sensor includes: The measurement information received from each sensor is stored in the corresponding cache queue; determining a primary sensor among the plurality of sensors; In response to the measurement information of the main sensor, obtaining a frame with a timestamp closest to the main sensor from the buffer queue of each sensor; According to the frame with the latest timestamp of the main sensor, the obstacle tracking trajectory is polled and matched with the measurement information in each sensor to obtain the matching result. The matching result includes one of the following: matched information of the obstacle tracking trajectory and the measurement information, unmatched obstacle tracking trajectory information, and unmatched measurement information.

3. The method according to claim 2, wherein: The step of matching the obstacle tracking trajectory to obtain a matching result in response to the measurement information from each sensor further includes: After polling the obstacle tracking trajectory and matching it with the measurement information in each sensor, generating a new obstacle tracking trajectory for the measurement information that is not matched; The obtained matching result includes newly generated obstacle tracking trajectory information.

4. The method according to claim 1, wherein: The type fusion specifically includes: Traversing each obstacle tracking trajectory and the measurement information in the matched sensor to obtain a cumulative probability of the obstacle type; According to the accumulated probability, a new set of obstacle type probabilities is obtained after fusing each matched sensor; After polling all matched sensor measurement information, a set of the latest obstacle type probabilities is obtained, and the obstacle type with the highest probability is selected to determine the type fusion result.

5. The method of claim 1, wherein: The shape fusion specifically includes: Traverse each obstacle tracking trajectory to perform shape fusion including at least obstacle Yaw information and obstacle Size information; Obtain all matching information of each obstacle tracking trajectory to obtain a matching value, and obtain a fusion result of the measurement value based on the matching value; The fusion results of the measurement values ​​are input into the filtering algorithms corresponding to the Yaw information and Size information respectively, and the filtered Yaw information and Size information are obtained for updating.

6. The method of claim 1, wherein: The motion fusion specifically includes: Traversing each obstacle tracking trajectory to perform motion fusion including at least obstacle position, obstacle speed, and obstacle outline information; Obtain all matching information of each obstacle tracking trajectory and obtain the fusion result of the measurement value; The fusion result including the measured position and speed is input into the corresponding filtering algorithm to obtain the filtered position and speed information for updating.

7. The method of claim 6, wherein: Updating the obstacle profile information further includes: The obstacle contour information is directly updated with the fusion result according to the scene selection; Alternatively, the obstacle contour information is generated according to the updated position information, Yaw information, and Size information.

8. A multi-sensor fusion device, wherein: Applied to the fusion module, the fusion device includes: a matching module, configured to match the obstacle tracking trajectory in response to the measurement information from each sensor to obtain a matching result; and A fusion processing module is used to obtain the fusion results of multiple sensors by performing type fusion, shape fusion and motion fusion on each obstacle tracking trajectory in the matching results. The type fusion is performed in an asynchronous manner, and the shape fusion and the motion fusion are performed in a synchronous manner.

9. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute the method according to any one of claims 1 to 7.

Images