Fusion positioning method, device, storage medium and program product

Abstract

Embodiments of the present application provide a fusion positioning method, device, storage medium and program product. The method comprises: obtaining at least one visual frame and at least one radar frame corresponding to a current fusion period based on a preset fusion period, the visual frame comprising first perception data of at least one target obtained by a camera, and the radar frame comprising second perception data of the at least one target obtained by a radar sensor; selecting a to-be-fused frame from the at least one visual frame and the at least one radar frame, the to-be-fused frame comprising a to-be-fused visual frame and a corresponding to-be-fused radar frame; fusing and compensating the to-be-fused frame to obtain a positioning result corresponding to the to-be-fused frame; and determining track information according to the fusion result corresponding to the to-be-fused frame, to generate automatic driving decision information according to the track information. The embodiments of the present application can reduce the robustness of the system and the hardware cost.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a fusion positioning method, device, storage medium, and program product. Background Technology

[0002] Autonomous driving systems enhance vehicle safety and comfort. These systems typically consist of a perception module and a decision-making module. The perception module provides the foundational data for the decision-making module; therefore, the performance of the perception module largely determines the overall performance of the system.

[0003] In related technologies, perception modules are often equipped as fusion systems of multiple radars or even multiple cameras to improve performance.

[0004] However, in the process of realizing this application, the inventors discovered that the prior art has at least the following problems: the above-mentioned methods not only have high hardware costs, but also the complex design of the fusion algorithm affects the robustness of the system. Summary of the Invention

[0005] This application provides a fusion positioning method, device, storage medium, and program product to improve the robustness of autonomous driving systems and reduce costs.

[0006] In a first aspect, embodiments of this application provide a fusion positioning method, including:

[0007] Based on a preset fusion period, at least one visual frame and at least one radar frame corresponding to the current fusion period are acquired; the visual frame includes first perception data of at least one target obtained through a camera; the radar frame includes second perception data of at least one target obtained through a radar sensor.

[0008] A frame to be fused is selected from at least one of the visual frames and at least one of the radar frames; the frame to be fused includes the visual frame to be fused and the corresponding radar frame to be fused.

[0009] The frames to be fused are fused and compensated to obtain the localization results corresponding to the frames to be fused.

[0010] Based on the positioning results corresponding to the frames to be fused, the trajectory information is determined, and autonomous driving decision information is generated based on the trajectory information.

[0011] In one possible design, fusing and compensating the frames to be fused to obtain the localization result corresponding to the frames to be fused includes:

[0012] At least one fusionable target is determined from at least one target in the visual frame to be fused and at least one target in the radar frame to be fused; the fusionable target includes a first target in the visual frame to be fused and a second target in the radar frame to be fused, and the first target and the second target satisfy a preset fusion condition;

[0013] For each fusionable target, the fusion result of the fusionable target is determined based on a filtering algorithm;

[0014] Based on the fusion result of the at least one fusionable target, the positioning result corresponding to the frame to be fused is determined.

[0015] In one possible design, determining the fusion result of the fusionable targets based on the Kalman filter algorithm includes:

[0016] Based on the perception data of the first target, determine the estimated value of the fusionable target at the input time of the second target;

[0017] Based on the estimated value and the perceived data of the second target, determine the difference between the two;

[0018] The fusion result of the fusionable target is determined based on the estimated value, the difference, and the Kalman gain.

[0019] In one possible design, the step of fusing and compensating the frames to be fused to obtain the positioning result corresponding to the frames to be fused further includes:

[0020] From at least one target in the visual frame to be fused and at least one target in the radar frame to be fused, at least one single-sensor target is determined; the single-sensor target is either a single visual target or a single radar target.

[0021] For each of the at least one single-sensor target, an estimation result of the single-sensor target is generated based on a nonlinear motion model, according to the sensing data and compensation time of the single-sensor target.

[0022] The step of determining the positioning result corresponding to the frame to be fused based on the fusion result of the at least one fusionable target includes:

[0023] Based on the fusion result of the at least one fusionable target and the estimation result of the at least one single-sensor target, the positioning result corresponding to the frame to be fused is determined.

[0024] In one possible design, determining the track information based on the positioning result corresponding to the frame to be fused includes:

[0025] Assign identifiers to each target in the fusion result corresponding to the frame to be fused in the current fusion cycle;

[0026] Obtain the identifiers of each target in the fusion result corresponding to the frame to be fused in the previous fusion cycle;

[0027] Based on the identifier, the fusion results of the two fusion cycles are associated, the track information corresponding to the previous fusion cycle is updated, and the track information corresponding to the current fusion cycle is obtained.

[0028] In one possible design, the assignment of identifiers to each target in the fusion result corresponding to the frame to be fused in the current fusion cycle includes:

[0029] Assign new identifiers to newly appearing targets in the fusion results of the frames to be fused in the current fusion cycle;

[0030] For targets in the fusion result of the frame to be fused in the current fusion cycle that are the same as targets in the fusion result of the frame to be fused in the previous fusion cycle, the corresponding identifiers from the previous fusion cycle will continue to be used.

[0031] In one possible design, the association of fusion results from two fusion cycles based on the identifier includes:

[0032] Identify targets with the same identifier among the targets in the fusion results corresponding to the frames to be fused in the current fusion cycle and the previous fusion cycle;

[0033] The fusion results of targets with the same identifier in the previous fusion cycle and the fusion results in the current fusion cycle are associated.

[0034] In one possible design, obtaining at least one visual frame and at least one radar frame corresponding to the current fusion cycle includes:

[0035] Obtain at least one original visual frame and at least one original radar frame corresponding to the current fusion cycle;

[0036] The at least one original visual frame and the at least one original radar frame are preprocessed to obtain at least one visual frame and at least one radar frame corresponding to the current fusion cycle.

[0037] In one possible design, the preprocessing of the at least one original visual frame and the at least one original radar frame includes:

[0038] Erroneous data in multiple original visual frames and multiple original radar frames are deleted.

[0039] Secondly, embodiments of this application provide a fusion positioning device, including:

[0040] The acquisition module is used to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period; the visual frame includes perception data of at least one target obtained through a camera; the radar frame includes perception data of at least one target obtained through a radar sensor.

[0041] A selection module is used to select a frame to be fused from at least one visual frame and at least one radar frame; the frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused.

[0042] The fusion module is used to fuse and compensate the frame to be fused to obtain the positioning result corresponding to the frame to be fused.

[0043] The determination module is used to determine the trajectory information based on the positioning result corresponding to the frame to be fused, so as to generate autonomous driving decision information based on the trajectory information.

[0044] Thirdly, embodiments of this application provide a fusion positioning device, including: at least one processor and a memory;

[0045] The memory stores computer-executed instructions;

[0046] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect and various possible designs of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and various possible designs of the first aspect.

[0049] Sixthly, embodiments of this application provide an autonomous driving system, including a vehicle, and a single camera and a single radar sensor mounted on the vehicle;

[0050] The single camera is used to acquire visual frames and send the visual frames to the vehicle, the visual frames including first perception data of at least one target;

[0051] The single radar sensor is used to acquire radar frames and send the radar frames to the vehicle, the radar frames including second perception data of at least one target;

[0052] The vehicle is configured to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period; the visual frame includes first perception data of at least one target obtained through a camera; the radar frame includes second perception data of at least one target obtained through a radar sensor; select a frame to be fused from at least one visual frame and at least one radar frame; the frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused; fuse and compensate the frame to be fused to obtain a positioning result corresponding to the frame to be fused; determine trajectory information based on the positioning result corresponding to the frame to be fused, and generate autonomous driving decision information based on the trajectory information.

[0053] The fusion positioning method, device, storage medium, and program product provided in this embodiment acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period. The visual frame includes first perception data of at least one target obtained through a camera, and the radar frame includes second perception data of at least one target obtained through a radar sensor. A frame to be fused is selected from the at least one visual frame and at least one radar frame. The frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused. The frame to be fused is fused and compensated to obtain the positioning result corresponding to the frame to be fused. Based on the fusion result corresponding to the frame to be fused, trajectory information is determined to generate autonomous driving decision information based on the trajectory information. The fusion positioning method provided in this embodiment adopts a single radar and single camera configuration and a loosely coupled framework. It uses a Kalman filter algorithm to fuse and compensate the perception data output by the two sensors, which can reduce the robustness of the system and reduce hardware costs, providing a foundation for the popularization of autonomous driving functions in various vehicle models. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram illustrating the speech recognition principle of the speech recognition system provided in the embodiments of this application;

[0056] Figure 2 Flowchart of the fusion positioning method provided in the embodiments of this application Figure 1 ;

[0057] Figure 3a A schematic diagram of the fusion principle provided for the embodiments of this application. Figure 1 ;

[0058] Figure 3b A schematic diagram of the fusion principle provided for the embodiments of this application. Figure 2 ;

[0059] Figure 4 Schematic diagram three illustrating the fusion principle provided in this application embodiment;

[0060] Figure 5 This is a schematic diagram of the structure of the fusion positioning device provided in the embodiments of this application;

[0061] Figure 6 This is a schematic diagram of the hardware structure of the fusion positioning device provided in the embodiments of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The vehicle employs an autonomous driving system that includes multiple autonomous driving functions (such as full-speed adaptive cruise control and active braking in Level 2 autonomous driving), which can improve driving safety and comfort. An autonomous driving system typically includes a perception module and a decision-making module. The perception module provides the basic data for the decision-making module; therefore, the performance of the perception module largely determines the performance of the entire system.

[0064] In related technologies, the perception module is usually equipped with a fusion system of multiple radars or even multiple cameras to improve performance. However, the above methods not only have high hardware costs, but also the complex design of fusion algorithms can affect the robustness of the system.

[0065] To address the aforementioned technical problems, the inventors of this application have discovered that reducing the number of cameras and radars can lower perception costs. However, considering that single cameras are susceptible to the effects of rain, snow, and lighting conditions, and have low accuracy in position detection, especially in lateral distance detection, while millimeter-wave radar is unaffected by rain or snow, its detection characteristics can lead to false alarms. Therefore, a fusion perception system combining single cameras and single radars can be used to ensure performance while reducing hardware costs. Based on this, embodiments of this application provide a fusion positioning method.

[0066] Compared with the forward autonomous driving solution of multiple cameras and multiple radars in the related art, the single camera and single radar solution provided by this application can actually effectively solve the adaptive cruise and automatic braking scenarios in autonomous driving at a low cost. Although the forward autonomous driving solution of multiple cameras and multiple radars may achieve autonomous driving functions at the L2+ level, its cost performance is relatively low.

[0067] In addition, there are two ways, loose coupling or tight coupling, that can be adopted in the fusion algorithm. Among them, tight coupling originates from the SLAM algorithm framework, which means directly fusing at the raw data layer, combining the advantages of multiple sensors to make it an output of a high-precision sensor. For example, what the camera receives is an image, and what the millimeter-wave radar receives is a point cloud. Tight coupling will combine the detection information of the point cloud and the image to uniformly output information such as position and speed; loose coupling means correcting and integrating the output information after passing through the internal algorithms of their respective sensors. For example, the camera outputs information such as position and speed through a deep learning algorithm, and the millimeter-wave radar outputs information such as position and speed through a point cloud algorithm. The loose coupling algorithm corrects the information of both to output information. The fusion positioning method provided by the embodiments of this application can adopt a loose coupling framework, so that when implementing this solution, the sensors and the autonomous driving solution can be developed distributively, enabling each R & D party to have its own quality control link, and ensuring the robustness and stability of the entire solution at each link. And 1V1R can combine the different advantages of the camera and the radar to output more accurate and rich environmental perception information than a single sensor.

[0068] The fusion positioning method provided by this embodiment has a high degree of safety redundancy when a sensor fails compared with the perception of a single camera or a single radar. Compared with the autonomous driving solution of multiple Vs and multiple Rs, when dealing with L2 application scenarios, the algorithm complexity is also greatly reduced, improving the robustness.

[0069] Figure 1 It is a schematic diagram of the application scenario of the fusion positioning method provided by the embodiments of this application. As Figure 1As shown, the vehicle 100 is equipped with a camera 101, a radar 102, and a controller 103. The camera 101 acquires image data, processes the image data to obtain corresponding visual frames, and sends the visual frames to the controller 103. The radar 102 acquires point cloud data, processes the point cloud data to obtain corresponding radar frames, and sends the radar frames to the controller 103. The visual frames include first perception data of at least one target; the radar frames include second perception data of at least one target. The first or second perception data includes target (vehicle, pedestrian, lane line, etc.) speed information, acceleration information, position information, attributes, and other data. The controller 103 is used to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period; select a frame to be fused from the at least one visual frame and at least one radar frame; the frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused; fuse and compensate the frame to be fused to obtain the positioning result corresponding to the frame to be fused; determine trajectory information based on the positioning result corresponding to the frame to be fused, and generate autonomous driving decision information based on the trajectory information. Optionally, the camera 101 can be installed on the windshield of the vehicle 100, and the radar can be installed on the front bumper of the vehicle 100.

[0070] In the specific implementation process, during the movement of the vehicle 100, the camera 101 and radar 102 collect data according to their respective sampling periods. The camera 101 acquires image data, processes the image data to obtain the corresponding visual frame, and sends the visual frame to the controller 103. The radar 102 acquires point cloud data, processes the point cloud data to obtain the corresponding radar frame, and sends the radar frame to the controller 103. The controller 103 fuses the visual frame and radar frame output by the camera 101 and radar 102 according to a preset fusion period. Specifically, based on the preset fusion period, the controller 103 acquires at least one visual frame and at least one radar frame corresponding to the current fusion period; selects a frame to be fused from the at least one visual frame and at least one radar frame; the frame to be fused includes the visual frame to be fused and the corresponding radar frame to be fused; fuses and compensates the frame to be fused to obtain the positioning result corresponding to the frame to be fused; and determines the trajectory information according to the positioning result corresponding to the frame to be fused, so as to generate autonomous driving decision information based on the trajectory information. The fusion positioning method provided in this embodiment adopts a single radar and single camera configuration and a loosely coupled framework. It uses a Kalman filter algorithm to fuse and compensate the perception data output by the two sensors. This not only reduces the robustness of the system but also reduces hardware costs, laying the foundation for the popularization of autonomous driving functions in various vehicle models.

[0071] It should be noted that, Figure 1The schematic diagram shown is merely an example. The fusion positioning method and scenario described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0072] The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0073] Figure 2 Flowchart of the fusion positioning method provided in the embodiments of this application Figure 1 .like Figure 2 As shown, the method includes:

[0074] 201. Based on a preset fusion period, acquire at least one visual frame and at least one radar frame corresponding to the current fusion period; the visual frame includes first perception data of at least one target obtained through a camera; the radar frame includes second perception data of at least one target obtained through a radar sensor.

[0075] The execution subject in this embodiment can be a device with fusion positioning function, such as... Figure 1 The vehicle 100 and the controller 103 of the vehicle 100 are shown.

[0076] Specifically, the camera captures image data in a first sampling period. The image data includes multiple image frames, each containing at least one target. After processing the image data, the camera obtains visual frames corresponding to each image frame. Each visual frame includes first perception data of at least one target in the corresponding image frame. The radar (e.g., millimeter-wave radar) transmits and receives millimeter waves in a second sampling period. By processing the changes in millimeter-wave frequency between periods, target motion information is obtained. All target information obtained in one processing cycle constitutes the corresponding radar frame. The first and second sampling periods can be different or the same; this embodiment does not limit this. The fusion period can be greater than or equal to the first and second sampling periods. Therefore, within each fusion period, at least one visual frame and at least one radar frame can be obtained.

[0077] In some embodiments, to improve data processing efficiency, data preprocessing can be performed to reduce interference from redundant or erroneous data. Specifically, obtaining at least one visual frame and at least one radar frame corresponding to the current fusion period may include: obtaining at least one original visual frame and at least one original radar frame corresponding to the current fusion period; preprocessing the at least one original visual frame and the at least one original radar frame to obtain at least one visual frame and at least one radar frame corresponding to the current fusion period.

[0078] In some embodiments, preprocessing may include a pre-performed coordinate transformation to convert data from different coordinate systems to the same coordinate system so that subsequent calculations are based on the transformed target coordinate system. Specifically, the preprocessing of the at least one original visual frame and the at least one original radar frame may include converting all of the original visual frames and the original radar frames to the target coordinate system.

[0079] Specifically, the raw visual frames and raw radar frames can be preprocessed. Preprocessing can include calibration, which can use the center of the vehicle's front bumper as the coordinate origin and combine this with the sensor's installation position to make the output result the coordinates of the front bumper center as the coordinate origin. Preprocessing can also include error information processing. For example, millimeter-wave radar has many false alarms, which can be filtered out based on attributes and other information.

[0080] 202. Select a frame to be fused from at least one of the visual frames and at least one of the radar frames; the frame to be fused includes the visual frame to be fused and the corresponding radar frame to be fused.

[0081] In this embodiment, frames to be fused can be selected from at least one visual frame and at least one radar frame based on the principle of temporal proximity for fusion. Specifically, there can be various fusion methods.

[0082] In one possible implementation, such as Figure 3a As shown, the nearest neighbor principle can be used to fuse visual frames and radar frames whose time intervals meet preset conditions. Specifically, the step of selecting a frame to be fused from at least one visual frame and at least one radar frame may include: if there is a target visual frame and a target radar frame among at least one visual frame and at least one radar frame whose input time interval is less than or equal to a preset threshold, then the target visual frame is determined as the visual frame to be fused in the frames to be fused, and the target radar frame is determined as the radar frame to be fused in the frames to be fused.

[0083] For example, during the forward movement, a fusion positioning system is used to obtain the position of the target vehicle ahead relative to the vehicle. From a temporal perspective, the received radar frames and visual frames are asynchronous. Before performing inter-frame target matching (matching between radar frames and visual frames, i.e., obtaining the fused frame), the definitions of fusion time and space are as follows, presented in timestamp format: Figure 3a As shown, the data from the most recent frame can be used for fusion. That is, the visual frame is fused with the radar frame at the most recent moment (which can be the radar frame with the smallest time interval between the visual frame and the preset time interval limit). The next frame is based on the visual frame, and the data of the next frame after fusion is the fusion of the visual update and the radar information at the most recent moment. For the previous fusion frame Nfusion and the next fusion frame N+1fusion, they are both obtained by matching the visual frame output by the camera with the radar frame output by the nearest radar.

[0084] In another possible implementation, such as Figure 3b As shown, fusion and pairing can be performed based on the timing of data updates; that is, fusion occurs as soon as updated data is obtained, regardless of whether other conditions are met. Specifically, updates are performed whenever data arrives; for fusion frames Nfusion, N+1fusion, N+2fusion, and N+3fusion, fusion occurs immediately upon data update.

[0085] In some embodiments, to improve the efficiency and accuracy of fusion processing, erroneous targets can be removed from the radar frames to be fused before fusion processing. Specifically, radar (e.g., millimeter-wave radar) detection results are relatively accurate but have many false alarms and are unaffected by rain or snow. Millimeter-wave radar has many false alarms, requiring filtering out false alarms based on attributes and other information.

[0086] 203. Perform fusion and compensation on the frames to be fused to obtain the positioning results corresponding to the frames to be fused.

[0087] For example, suppose that the initial observation time of the fusion observation time corresponding to each fusion cycle is based on the camera's input time, such as... Figure 4 As shown, the visual sensor, such as a camera, inputs the visual frame to be fused to the fusion module of the controller at time t0, and the radar inputs the radar frame to be fused to the fusion module of the controller at time δ of time t0. t1 Afterwards, the data is processed by the fusion algorithm of the fusion module and then fed into the autonomous driving functions (Adaptive Cruise Control (ACC) and Automatic Emergency Braking (AEB)). This data then undergoes further processing by the δ... t2The time interval is N, so the moment when the autonomous driving function module receives and can use the fused information is N. In other words, there is an error between the data application time and the actual data input time. To eliminate this error, δ can be compensated based on the timestamp or test estimation. t1 δ t2 The resulting error.

[0088] In some embodiments, fusing and compensating the frames to be fused to obtain the localization result corresponding to the frames to be fused may include: determining at least one fusible target from at least one target in the visual frame to be fused and at least one target in the radar frame to be fused; the fusible target includes a first target in the visual frame to be fused and a second target in the radar frame to be fused, and the first target and the second target satisfy a preset fusion condition; for each fusible target, determining the fusion result of the fusible target based on a filtering algorithm; and determining the localization result corresponding to the frames to be fused based on the fusion result of the at least one fusible target.

[0089] The filtering algorithm can be the Kalman filter algorithm, and the preset fusion condition means that the distance between the first target and the second target is less than or equal to a preset threshold.

[0090] Specifically, since the first target in the visual frame to be fused and the second target in the radar frame to be fused meet the preset fusion conditions, they can be considered as the same target. Therefore, the first target and the second target can be fused. By performing fusion, the visual results and radar results of the same target can be combined, making the localization result of the same target more accurate.

[0091] In some embodiments, determining the fusion result of the fusionable targets based on the Kalman filter algorithm includes:

[0092] Based on the perception data of the first target, determine the estimated value of the fusionable target at the input time of the second target;

[0093] Based on the estimated value and the perceived data of the second target, determine the difference between the two;

[0094] The fusion result of the fusionable target is determined based on the estimated value, the difference, and the Kalman gain.

[0095] For example, the fusion result at time N. The solution method is as follows:

[0096]

[0097] Here, X1, X2, and X3 represent the states of each target after correction by the fusion algorithm. Since there are actually several targets to be fused (e.g., vehicles traveling ahead), we will temporarily assume there are three (the visual frame to be fused includes camera target 1, camera target 2, and camera target 3; the radar frame to be fused includes radar target 1, radar target 2, and radar target 3). Then, X1 can represent the states of camera target 1 and radar target 1 after fusion correction. By distributing these states, we can solve for... It has become X, which is used to solve for several objectives.

[0098] Next, let the state of the camera (camera) at time t0 be x0, and t0+δ t1 The time estimate can be obtained using the Kalman filter algorithm:

[0099] x vision-est =Fx0+u (2)

[0100] Where F is the state transition matrix and u is the external influence.

[0101] The state transition covariance matrix P is then solved as follows:

[0102] P′=FPF T +Q (3)

[0103] In the first cyclic calculation, P takes a preset initial value, and Q is the process noise, temporarily set as the identity matrix, because the uncertainty is uncertain. At t0+δ t1 At a certain time, the radar data is updated. The input radar frame at this moment is actually the observation in the Kalman filter model. Therefore, there must be a difference between the observation and the prediction. So, the difference can be calculated as follows:

[0104] y = z radar -Hx vision-est (4)

[0105] Next, we solve for the Kalman gain:

[0106] S=HP′H T +R (5)

[0107] K = P'H T S -1 (6)

[0108] Where H is the measurement matrix and R is the measurement noise matrix. R should be provided by the sensor manufacturer to indicate the product uncertainty, as measuring it yourself would be too costly.

[0109] Now, the corrected state can be obtained as follows:

[0110]

[0111] Construct the closed-loop solution state covariance matrix:

[0112] P=(I-KH)P′ (8)

[0113] In subsequent iterative calculations, the value of P is the value of P′ in the previous cycle.

[0114] In some embodiments, fusing and compensating the frames to be fused to obtain the localization result corresponding to the frames to be fused may further include: determining at least one single-sensor target from at least one target in the visual frame to be fused and at least one target in the radar frame to be fused; the single-sensor target is a single visual target or a single radar target; for each of the at least one single-sensor targets, generating an estimation result of the single-sensor target based on a nonlinear motion model, according to the perception data and compensation time of the single-sensor target; determining the localization result corresponding to the frames to be fused based on the fusion result of the at least one fusionable target may include: determining the localization result corresponding to the frames to be fused based on the fusion result of the at least one fusionable target and the estimation result of the at least one single-sensor target.

[0115] Specifically, cameras and radar sensors have different sensing capabilities and ranges. Some targets are better perceived by cameras, and in some situations, can only be detected by cameras, such as pedestrians. Other targets are better perceived by radar sensors, and in some situations, can only be detected by radar sensors, such as large trucks. A single-vision target is one that exists only in the first sensing data obtained by the camera, and not in the second sensing data obtained by the radar sensor. A single radar target exists only in the second sensing data obtained by the radar sensor, and not in the first sensing data obtained by the camera.

[0116] For example, for single-sensor targets such as single-vision targets or single-radar targets, an open-loop solution can be used, employing a nonlinear motion model for state estimation, that is, based on velocity, acceleration, and time interval (δ). t1 and δ t2 Assuming the target is undergoing uniformly accelerated motion, we estimate its magnitude to obtain the estimation result for a single-sensor target.

[0117] 204. Based on the positioning results corresponding to the frames to be fused, determine the trajectory information, and generate autonomous driving decision information based on the trajectory information.

[0118] Specifically, track management records the fusion status at each moment and manages the entry and exit queues based on the label ID. That is, if the ID of the previous moment is the same as the ID of the next moment, it can be considered as the same target. The same target will be continuously recorded in a queue to form a continuous track, which is track association. If a match cannot be found, a new track will be generated, which is track splitting.

[0119] For example, during vehicle operation, fusion localization is continuously performed, and a fusion result is obtained in each fusion cycle. Each fusion result includes perception data (speed, acceleration, position, etc.) of different targets (e.g., vehicles in front). In order to associate the fusion results of multiple cycles, an identifier (e.g., a label) can be assigned to each target in the fusion result. In the identification of different fusion frames, the same target (e.g., a white convertible driving in front) is assigned the same label. If label 5 is included in 5 consecutive fusion cycles, the localization data of the target corresponding to label 5 can be associated to form a track association. If label 5 exists in one of two adjacent fusion cycles and label 5 does not exist in the other, then the track is determined to be split.

[0120] In some embodiments, determining the trajectory information based on the positioning result corresponding to the frame to be fused may include: assigning an identifier to each target in the fusion result corresponding to the frame to be fused in the current fusion period; obtaining the identifier of each target in the fusion result corresponding to the frame to be fused in the previous fusion period; associating the fusion results of the two fusion periods based on the identifier, updating the trajectory information corresponding to the previous fusion period, and obtaining the trajectory information corresponding to the current fusion period.

[0121] In some embodiments, assigning identifiers to each target in the fusion result corresponding to the frame to be fused in the current fusion cycle may include: assigning new identifiers to newly appearing targets in the fusion result corresponding to the frame to be fused in the current fusion cycle; and continuing to use the identifiers corresponding to the previous fusion cycle for targets in the fusion result corresponding to the frame to be fused in the current fusion cycle that are the same as the targets in the fusion result of the frame to be fused in the previous fusion cycle.

[0122] In some embodiments, associating the fusion results of two fusion cycles based on the identifier may include: determining targets with the same identifier among the targets in the fusion results of the frames to be fused in the current fusion cycle and the previous fusion cycle; and associating the fusion results of the targets with the same identifier in the previous fusion cycle with the fusion results in the current fusion cycle.

[0123] Specifically, for frames to be fused based on the fusion period, i.e., discrete data, it is necessary to transform them into continuous spatiotemporal states, which involves data association. Several possible states are defined for each frame, such as fusionable target, new target, target that was fused in the previous moment but is now a single visual target, target that was fused in the previous moment but is now a single radar target, single radar target becomes fused target, single visual target becomes fused target, lost target, etc. At each moment, the target ID and the current fusion state are recorded. At the next moment, the ID of the previous moment is tracked to record continuous matching states, and track management is performed based on the tracking results. Track management records the fusion state at each moment and manages entry and exit queues based on the ID number. That is, if the ID of the previous moment is the same as the ID of the next moment, it is considered to be the same target. The same target will be continuously recorded in a queue, forming a continuous trajectory, which is track association. If no match is found, a new trajectory will be generated, which is track splitting. During this track management process, for a single sensor target, an identifier, such as a label ID, can be assigned to the included target. This allows for association between a single sensor target and a fused frame, as well as association between single sensor targets, based on the ID.

[0124] The fusion positioning method provided in this embodiment adopts a single radar and single camera configuration and a loosely coupled framework. It uses a Kalman filter algorithm to fuse and compensate the perception data output by the two sensors. This not only reduces the robustness of the system but also reduces hardware costs, laying the foundation for the popularization of autonomous driving functions in various vehicle models.

[0125] Figure 5 This is a schematic diagram of the structure of the fusion positioning device provided in an embodiment of this application. Figure 5 As shown, the fusion positioning device 50 includes: an acquisition module 501, a selection module 502, a fusion module 503, and a determination module 504.

[0126] The acquisition module 501 is used to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period; the visual frame includes perception data of at least one target obtained through a camera; the radar frame includes perception data of at least one target obtained through a radar sensor.

[0127] The selection module 502 is used to select a frame to be fused from at least one visual frame and at least one radar frame; the frame to be fused includes the visual frame to be fused and the corresponding radar frame to be fused.

[0128] The fusion module 503 is used to fuse and compensate the frame to be fused to obtain the positioning result corresponding to the frame to be fused.

[0129] The determining module 504 is used to determine the trajectory information based on the positioning result corresponding to the frame to be fused, so as to generate autonomous driving decision information based on the trajectory information.

[0130] The fusion positioning device provided in this application adopts a single radar and single camera configuration and a loosely coupled framework. It uses a Kalman filter algorithm to fuse and compensate the perception data output by two sensors, which can reduce the robustness of the system and reduce hardware costs, thus providing a foundation for the popularization of autonomous driving functions in various vehicle models.

[0131] The fusion positioning device provided in this application embodiment can be used to execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0132] This application embodiment also provides an autonomous driving system, including a vehicle, and a single camera and a single radar sensor mounted on the vehicle; the single camera is used to acquire visual frames and send the visual frames to the vehicle, the visual frames including first perception data of at least one target; the single radar sensor is used to acquire radar frames and send the radar frames to the vehicle, the radar frames including second perception data of at least one target; the vehicle is used to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period; the visual frames include first perception data of at least one target obtained through the camera; the radar frames include second perception data of at least one target obtained through the radar sensor; select a frame to be fused from at least one visual frame and at least one radar frame; the frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused; fuse and compensate the frame to be fused to obtain a positioning result corresponding to the frame to be fused; determine trajectory information based on the positioning result corresponding to the frame to be fused, and generate autonomous driving decision information based on the trajectory information.

[0133] The autonomous driving system provided in this application adopts a single radar and single camera configuration and a loosely coupled framework. It uses a Kalman filter algorithm to fuse and compensate the perception data output by the two sensors, which can reduce the robustness of the system and reduce hardware costs, thus providing a foundation for the popularization of autonomous driving functions in various vehicle models.

[0134] The autonomous driving system provided in this application embodiment can be used to execute the above-described method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0135] Figure 6 This is a structural block diagram of the fusion positioning device provided in the embodiments of this application. The device may be a computer, a messaging device, a tablet device, a medical device, etc.

[0136] Device 60 may include one or more of the following components: processing component 601, memory 602, power supply component 603, multimedia component 604, audio component 605, input / output (I / O) interface 606, sensor component 607, and communication component 608.

[0137] Processing component 601 typically controls the overall operation of device 60, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 601 may include one or more processors 609 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 601 may include one or more modules to facilitate interaction between processing component 601 and other components. For example, processing component 601 may include a multimedia module to facilitate interaction between multimedia component 604 and processing component 601.

[0138] Memory 602 is configured to store various types of data to support the operation of device 60. Examples of this data include instructions for any application or method operating on device 60, contact data, phonebook data, messages, pictures, videos, etc. Memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0139] Power supply component 603 provides power to various components of device 60. Power supply component 603 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 60.

[0140] Multimedia component 604 includes a screen that provides an output interface between the device 60 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 604 includes a front-facing camera and / or a rear-facing camera. When the device 60 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0141] Audio component 605 is configured to output and / or input audio signals. For example, audio component 605 includes a microphone (MIC) configured to receive external audio signals when device 60 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 602 or transmitted via communication component 608. In some embodiments, audio component 605 also includes a speaker for outputting audio signals.

[0142] I / O interface 606 provides an interface between processing component 601 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0143] Sensor assembly 607 includes one or more sensors for providing state assessments of various aspects of device 60. For example, sensor assembly 607 can detect the on / off state of device 60, the relative positioning of components such as the display and keypad of device 60, changes in the position of device 60 or a component of device 60, the presence or absence of user contact with device 60, the orientation or acceleration / deceleration of device 60, and temperature changes of device 60. Sensor assembly 607 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 607 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 607 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0144] Communication component 608 is configured to facilitate wired or wireless communication between device 60 and other devices. Device 60 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 608 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 608 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0145] In an exemplary embodiment, device 60 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0146] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 609 of device 60 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0147] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0148] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0149] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0150] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the fusion positioning method executed by the fusion positioning device as described above.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A fusion positioning method, characterized in that, include: Based on a preset fusion period, acquire at least one visual frame and at least one radar frame corresponding to the current fusion period; The visual frame includes first perception data of at least one target obtained through a single camera; The radar frame includes second sensing data of at least one target obtained through a single radar sensor; Select a frame to be fused from at least one of the visual frames and at least one of the radar frames; The frames to be fused include visual frames to be fused and corresponding radar frames to be fused; The frames to be fused are fused and compensated to obtain the localization results corresponding to the frames to be fused. Based on the positioning results corresponding to the frames to be fused, the trajectory information is determined, and autonomous driving decision information is generated based on the trajectory information. The step of determining the track information based on the positioning result corresponding to the frame to be fused includes: Assign identifiers to each target in the fusion result corresponding to the frame to be fused in the current fusion cycle; Obtain the identifiers of each target in the fusion result corresponding to the frame to be fused in the previous fusion cycle; Based on the identifier, the fusion results of the two fusion cycles are associated, the track information corresponding to the previous fusion cycle is updated, and the track information corresponding to the current fusion cycle is obtained.

2. The method according to claim 1, characterized in that, The process of fusing and compensating the frames to be fused to obtain the localization result corresponding to the frames to be fused includes: At least one fusionable target is determined from at least one target in the visual frame to be fused and at least one target in the radar frame to be fused; the fusionable target includes a first target in the visual frame to be fused and a second target in the radar frame to be fused, and the first target and the second target satisfy a preset fusion condition; For each fusionable target, the fusion result of the fusionable target is determined based on a filtering algorithm; Based on the fusion result of the at least one fusionable target, the positioning result corresponding to the frame to be fused is determined.

3. The method according to claim 2, characterized in that, The method for determining the fusion result of the fusionable targets based on the filtering algorithm includes: Based on the perception data of the first target, determine the estimated value of the fusionable target at the input time of the second target; Based on the estimated value and the perceived data of the second target, determine the difference between the two; The fusion result of the fusionable target is determined based on the estimated value, the difference, and the Kalman gain.

4. The method according to claim 3, characterized in that, The step of fusing and compensating the frame to be fused to obtain the positioning result corresponding to the frame to be fused further includes: From at least one target in the visual frame to be fused and at least one target in the radar frame to be fused, at least one single-sensor target is determined; the single-sensor target is either a single visual target or a single radar target. For each of the at least one single-sensor target, an estimation result of the single-sensor target is generated based on a nonlinear motion model, according to the sensing data and compensation time of the single-sensor target. The step of determining the positioning result corresponding to the frame to be fused based on the fusion result of the at least one fusionable target includes: Based on the fusion result of the at least one fusionable target and the estimation result of the at least one single-sensor target, the positioning result corresponding to the frame to be fused is determined.

5. The method according to claim 1, characterized in that, The term refers to the assignment identifiers of each target in the fusion result corresponding to the frame to be fused in the current fusion period, including: Assign new identifiers to newly appearing targets in the fusion results of the frames to be fused in the current fusion cycle; For targets in the fusion result of the frame to be fused in the current fusion cycle that are the same as targets in the fusion result of the frame to be fused in the previous fusion cycle, the corresponding identifiers from the previous fusion cycle will continue to be used.

6. The method according to claim 1, characterized in that, The method of associating the fusion results from two fusion cycles based on the identifier includes: Identify targets with the same identifier among the targets in the fusion results corresponding to the frames to be fused in the current fusion cycle and the previous fusion cycle; The fusion results of targets with the same identifier in the previous fusion cycle and the fusion results in the current fusion cycle are associated.

7. The method according to any one of claims 1-4, characterized in that, The step of obtaining at least one visual frame and at least one radar frame corresponding to the current fusion cycle includes: Obtain at least one original visual frame and at least one original radar frame corresponding to the current fusion cycle; The at least one original visual frame and the at least one original radar frame are preprocessed to obtain at least one visual frame and at least one radar frame corresponding to the current fusion cycle.

8. The method according to claim 7, characterized in that, The preprocessing of the at least one original visual frame and the at least one original radar frame includes: Erroneous data in multiple original visual frames and multiple original radar frames are deleted.

9. A fusion positioning device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the fusion positioning method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the fusion positioning method as described in any one of claims 1 to 8.

11. An autonomous driving system, characterized in that, This includes the vehicle, as well as a single camera and a single radar sensor mounted on the vehicle; The single camera is used to acquire visual frames and send the visual frames to the vehicle, the visual frames including first perception data of at least one target; The single radar sensor is used to acquire radar frames and send the radar frames to the vehicle, the radar frames including second perception data of at least one target; The vehicle is configured to acquire at least one visual frame and at least one radar frame corresponding to the current fusion period based on a preset fusion period. The visual frame includes first perception data of at least one target obtained through a single camera; The radar frame includes second perception data of at least one target obtained through a single radar sensor; a frame to be fused is selected from at least one visual frame and at least one radar frame; the frame to be fused includes a visual frame to be fused and a corresponding radar frame to be fused; The frames to be fused are fused and compensated to obtain the positioning results corresponding to the frames to be fused; based on the positioning results corresponding to the frames to be fused, the trajectory information is determined, and autonomous driving decision information is generated based on the trajectory information; The step of determining the track information based on the positioning result corresponding to the frame to be fused includes: Assign identifiers to each target in the fusion result corresponding to the frame to be fused in the current fusion cycle; Obtain the identifiers of each target in the fusion result corresponding to the frame to be fused in the previous fusion cycle; Based on the identifier, the fusion results of the two fusion cycles are associated, the track information corresponding to the previous fusion cycle is updated, and the track information corresponding to the current fusion cycle is obtained.

Images