Method for radar and vision information fusion in holographic intersection scene

Abstract

The embodiment of the application provides a kind of radar and visual information fusion method under holographic intersection scene, comprising the following steps: S1, the intersection needing is aerial photography, to obtain aerial view;S2, visual calibration;S3, radar calibration;S4, target from radar and target from machine vision are mapped to overhead coordinate system;S5, set up radar and visual fusion area in overhead coordinate system;S6, point set fusion.The application uses overhead coordinate system as unified coordinate system, avoids obtaining camera internal parameter;Respectively calibrates space and time, and radar information and visual information are projected to overhead coordinate system;In space calibration, radar information projection uses affine transformation, and visual information projection uses thin plate transformation;Divide radar and visual fusion area, reduce the influence of inaccuracy in time and space calibration process on final effect, time alignment can have redundancy, avoid obtaining image and radar data at the same time.

Description

Technical Field

[0001] This application relates to the fields of millimeter-wave radar and visual measurement technology, and in particular to a method for fusing radar and visual information in holographic intersection scenarios. Background Technology

[0002] With the advancement of urbanization, traffic factors are having an increasingly significant impact on the further development of cities, making the refined management of urban roads more urgent. Holographic intersections based on radar-visual fusion digitally reconstruct elements such as pedestrians, motor vehicles, and non-motorized vehicles on urban roads, providing an integrated solution for the refined management of urban roads and further improving the efficiency of urban traffic governance.

[0003] In radar-visual fusion, the main processes involved are sensor calibration and alignment, and sensor fusion. Sensor calibration and alignment transforms data from multiple sensors so that their data are in the same spatiotemporal coordinate system, while also determining the error relationship under different usage conditions. This mainly involves spatial alignment and temporal alignment. Sensor fusion combines information such as position, speed, heading angle, latitude and longitude output by radar with information such as license plate, color, and vehicle type obtained through machine vision to provide all-time, multi-dimensional, and all-element perception.

[0004] Existing technologies often utilize camera calibration techniques to calculate the internal parameters of the camera and establish a camera coordinate model; based on the positional relationship between the camera and the radar, a coordinate correspondence between the camera and the radar in the world coordinate system is established; then, based on the coordinate correspondence, radar information and visual information are fused to realize the transformation of the 3D world coordinate system detected by the radar into the corresponding 2D image coordinates in the camera's visual image.

[0005] In practical applications of holographic intersections, cameras are often pre-installed, and obtaining their internal parameters is inconvenient. Existing technologies often rely on the camera's internal parameters and require simultaneous acquisition of radar and visual information. Summary of the Invention

[0006] To address one of the aforementioned technical shortcomings, this application provides a method for fusing radar and visual information in a holographic intersection scenario, which solves the problem that existing technologies often rely on camera internal parameters and require the acquisition of radar and visual information at the same time.

[0007] According to a first aspect of the embodiments of this application, a method for fusing radar and visual information in a holographic intersection scene is provided, comprising the following steps:

[0008] S1. Take aerial photos of the required intersections to obtain an aerial view. The aerial view should be clear, distortion-free, and cover the required road section.

[0009] S2, Visual calibration: Obtain the feature points of the aerial top view and the camera image and establish the correspondence. Use thin plate spline transformation to obtain the transformation equation, which is used to map the target obtained by machine vision in the camera pixel coordinate system to the top view coordinate system in real time.

[0010] S3. Radar Calibration: Match the points in the radar coordinate system with the points in the top view coordinate system, and then obtain the transformation matrix through affine transformation;

[0011] S4. Through steps S1, S2, and S3, the targets from the radar and the targets from the machine vision are mapped onto the top-down coordinate system;

[0012] S5. Set up a radar-visual fusion zone in the top-down coordinate system to ensure that the target from the radar and the target from the machine vision approximately overlap;

[0013] S6. Point Set Fusion: The Hungarian matching algorithm is used to match radar targets in the radar-visual fusion zone with targets from machine vision to establish a correspondence.

[0014] As a further improvement to this application, step S2 includes the following steps:

[0015] S21. The feature points in the top-view coordinate system correspond to the feature points in the camera pixel coordinate system to obtain feature point pairs;

[0016] S22. Calculate the thin plate spline transformation based on the feature point pairs in step S21 to obtain the transformation equation.

[0017] As a further improvement to this application, step S3 includes the following steps:

[0018] S31. When installing radar equipment, corner reflectors are set up at intersections. The coordinates of the point set in the top view coordinate system are obtained in the radar coordinate system through the corner reflectors, thus obtaining the point set pair.

[0019] S32. Based on the point set pair in step S31, calculate its affine transformation matrix to obtain the transformation matrix from the radar coordinate system to the top-view coordinate system, which is used to map the target obtained by the radar in the radar coordinate system to the top-view coordinate system in real time.

[0020] As a further improvement of this application, in step S3, it is assumed that the coordinates of the target point in the top view coordinate system in the radar coordinate system are as follows: (114.419858, 23.12206), (114.419902, 23.122015), (114.419877, 23.121952), (114.419828, 23.122006), (114.419834, 23.121971), (114.419842, 23.121926), (114.419693, 23.121966), (114.419701, 23.121939).

[0021] The corresponding coordinates in the top view coordinate system are: (2359,2456), (2404,2525), (2356,2601), (2309,2520), (2309,2569), (2308,2629), (2133,2542), (2137,2582).

[0022] The final transformation matrix is:

[0023]

[0024] As a further improvement to this application, the top-view coordinate system is an image coordinate system of aerial top view, with the upper left corner as the origin, the downward direction as the Y-axis, and the rightward direction as the X-axis, with the unit being pixels.

[0025] As a further improvement of this application, the radar coordinate system, wherein point O is the radar installation position, and the lower left corner is the radar installation position according to the radar installation position, that is, the upward direction is the Y-axis and the leftward direction is the X-axis.

[0026] As a further improvement of this application, the camera pixel coordinate system is the image output by the camera. In this coordinate system, the origin is the upper left corner of the image, the X-axis extends to the right, the Y-axis extends downward, and each pixel has a unique pixel coordinate.

[0027] The method for fusing radar and visual information in a holographic intersection scene provided in this application embodiment uses a top-down coordinate system as a unified coordinate system to avoid obtaining internal camera parameters.

[0028] Spatial and temporal coordinates are calibrated separately, and radar and visual information are projected onto a top-down coordinate system;

[0029] In spatial calibration, radar information projection uses affine transformation, while visual information projection uses plate transformation.

[0030] Dividing the radar-visual fusion zone reduces the impact of inaccuracies in the temporal and spatial calibration process on the final result. Temporal alignment can have redundancy to avoid acquiring images and radar data at the same time. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 A schematic diagram of the coordinate system for an aerial top view provided in an embodiment of this application.

[0033] Figure 2 A schematic diagram of the radar coordinate system provided in the embodiments of this application.

[0034] Figure 3 This is a schematic diagram of the camera pixel coordinate system provided in an embodiment of this application.

[0035] Figure 4 This is a visual calibration diagram provided for an embodiment of this application.

[0036] Figure 5 The image shows the visual calibration effect provided in the embodiments of this application.

[0037] Figure 6 This is a schematic diagram of radar calibration provided for an embodiment of this application.

[0038] Figure 7 The image shows the radar calibration effect provided in the embodiment of this application.

[0039] Figure 8 A top-view diagram showing the target points obtained by radar and vision, provided for an embodiment of this application.

[0040] Figure 9 This is a schematic diagram of the radar and vision fusion area provided in an embodiment of this application. Detailed Implementation

[0041] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0042] In the process of developing this application, the inventors discovered that in practical applications of holographic intersection scenarios, cameras are often pre-installed, and obtaining the internal parameters of the cameras is inconvenient. Existing technologies often rely on the internal parameters of the cameras and require the acquisition of radar and visual information at the same time.

[0043] To address the aforementioned issues, this application provides a method for fusing radar and visual information in a holographic intersection scenario, comprising the following steps:

[0044] S1. Take aerial photos of the required intersections to obtain an aerial view. The aerial view should be clear, distortion-free, and cover the required road section.

[0045] S2, Visual calibration: Obtain the feature points of the aerial top view and the camera image and establish the correspondence. Use thin plate spline transformation to obtain the transformation equation, which is used to map the target obtained by machine vision in the camera pixel coordinate system to the top view coordinate system in real time.

[0046] S3. Radar Calibration: Match the points in the radar coordinate system with the points in the top view coordinate system, and then obtain the transformation matrix through affine transformation;

[0047] S4. Through steps S1, S2, and S3, the targets from the radar and the targets from the machine vision are mapped onto the top-down coordinate system;

[0048] S5. Set up a radar-visual fusion zone in the top-down coordinate system to ensure that the target from the radar and the target from the machine vision approximately overlap;

[0049] S6. Point Set Fusion: The Hungarian matching algorithm is used to match radar targets in the radar-visual fusion zone with targets from machine vision to establish a correspondence.

[0050] As a further improvement to this application, step S2 includes the following steps:

[0051] S21. The feature points in the top-view coordinate system correspond to the feature points in the camera pixel coordinate system to obtain feature point pairs;

[0052] S22. Calculate the thin plate spline transformation based on the feature point pairs in step S21 to obtain the transformation equation.

[0053] As a further improvement to this application, step S3 includes the following steps:

[0054] S31. When installing radar equipment, corner reflectors are set up at intersections. The coordinates of the point set in the top view coordinate system are obtained in the radar coordinate system through the corner reflectors, thus obtaining the point set pair.

[0055] S32. Based on the point set pair in step S31, calculate its affine transformation matrix to obtain the transformation matrix from the radar coordinate system to the top-view coordinate system, which is used to map the target obtained by the radar in the radar coordinate system to the top-view coordinate system in real time.

[0056] As a further improvement of this application, in step S3, it is assumed that the coordinates of the target point in the top view coordinate system in the radar coordinate system are as follows: (114.419858, 23.12206), (114.419902, 23.122015), (114.419877, 23.121952), (114.419828, 23.122006), (114.419834, 23.121971), (114.419842, 23.121926), (114.419693, 23.121966), (114.419701, 23.121939).

[0057] The corresponding coordinates in the top view coordinate system are: (2359,2456), (2404,2525), (2356,2601), (2309,2520), (2309,2569), (2308,2629), (2133,2542), (2137,2582).

[0058] The final transformation matrix is:

[0059]

[0060] In this application, the top-view coordinate system is the same as the aerial top-view image coordinate system, with the origin at the top left corner, the Y-axis pointing downwards, and the X-axis pointing to the right. The unit is pixels. The aerial top-view coordinate system is as follows: Figure 1 As shown.

[0061] In this application, a radar coordinate system is used, where point O represents the radar installation position. Based on the radar installation position, the lower left corner is the radar installation location; that is, the Y-axis points upwards and the X-axis points to the left. The radar coordinate system is as follows: Figure 2 As shown, the unit is meters.

[0062] In this application, the camera pixel coordinate system is the image output by the camera. In this coordinate system, the origin is the upper left corner of the image, the X-axis extends to the right, and the Y-axis extends downwards. Each pixel has a unique pixel coordinate. The camera pixel coordinate system is as follows: Figure 3 As shown, the unit is pixels.

[0063] In this application, visual calibration involves matching the image obtained by the camera with points in the top-view coordinate system, and obtaining the transformation equation through thin-plate spline transformation. The specific process is as follows: 1. Feature points in the top-view coordinate system are mapped to feature points in the camera pixel coordinate system to obtain feature point pairs, such as... Figure 4 As shown; 2. The transformation equation is obtained by calculating the thin-plate spline transformation based on the feature points. By using the aerial view and the thin-plate spline transformation, the coordinates at the top view angle are obtained by solving for the camera's intrinsic and extrinsic parameters, and interference with normal traffic is avoided. The final calibration result is as follows. Figure 5 As shown。

[0064] In this application, the purpose of radar calibration is to match points in the radar coordinate system with points in the top-view coordinate system, and then obtain the transformation matrix through affine transformation. The specific steps are as follows: 1. Obtain the coordinate values ​​of the point set in the top-view coordinate system in the radar coordinate system through a corner reflector, and obtain point set pairs, such as... Figure 6 As shown; 2. Based on the point set pairs, calculate their affine transformation matrix to obtain the final transformation matrix from the radar coordinate system to the top view coordinate system. The final calibration result is as follows. Figure 7 As shown.

[0065] This method avoids using commonly used point set registration techniques. This is because, in practice, vehicles often move in straight lines, making it impossible to correspond to a straight line in two point sets. Furthermore, the radar detection area is much larger than the visual detection area, so the error introduced by this method is less than that of point set registration techniques.

[0066] The aforementioned visual and radar calibrations only address spatial alignment, ensuring that the distance between the radar target and the visual target is minimal in the top-down view at the same moment. In the holographic intersection scenario, temporal alignment is not strictly required; fixed parameters can be approximately set, and the final spatial and temporal alignment effect will be as follows: Figure 8 As shown.

[0067] In this application, the characteristics of radar and machine vision need to be considered when performing point set fusion: radar has a detection range of up to 100 meters and can provide accurate position and speed information; the vehicle's top-down coordinates obtained through machine vision are more accurate when close to the camera, but deviate significantly at greater distances. For example... Figure 8 As shown, circles represent radar target points and triangles represent visual target points. The vehicle is traveling from north to south, and both the radar and camera are installed on the north side. Visual target points closer to the north are closer to the radar target. Therefore, it is necessary to define an effective machine vision region. Within this effective region, the information provided by vision is primary, and the area where radar and vision fusion occurs should be included within the effective visual region. Theoretically, points obtained through image recognition and points detected by radar for the same vehicle should overlap in the top-view coordinate system. However, due to time errors between image frames and radar frames, radar vision calibration, and imperfections in machine vision technology, the two points may separate. It is necessary to find a region with minimal separation as the fusion region, such as... Figure 9 As shown.

[0068] In the fusion zone, the Hungarian matching algorithm is used to pair radar and visual information and save the matching results (the correspondence between radar targets and visual targets), thus completing point set fusion. In the radar-visual fusion zone and the radar ROI, the radar output with added visual information is used as the standard to achieve long-term tracking. Simultaneously, considering the jitter during handover, target points are filtered to smooth the trajectory.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as C, VHDL, Verilog, the object-oriented programming language Java, and the interpreted scripting language JavaScript.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0074] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for fusing radar and visual information in a holographic intersection scene, characterized in that, Includes the following steps: S1. Take aerial photos of the required intersections to obtain an aerial view. The aerial view should be clear, distortion-free, and cover the required road section. S2, Visual calibration: Obtain the feature points of the aerial top view and the camera image and establish the correspondence. Use thin plate spline transformation to obtain the transformation equation, which is used to map the target obtained by machine vision in the camera pixel coordinate system to the top view coordinate system in real time. S3. Radar Calibration: Match the points in the radar coordinate system with the points in the top view coordinate system, and then obtain the transformation matrix through affine transformation; S4. Through steps S1, S2, and S3, the targets from the radar and the targets from the machine vision are mapped onto the top-down coordinate system. S5. Set up a radar-visual fusion zone in the top-down coordinate system to ensure that the target from the radar and the target from the machine vision approximately overlap; S6. Point Set Fusion: The Hungarian matching algorithm is used to match radar targets in the radar-visual fusion zone with targets from machine vision to establish a correspondence.

2. The method for fusing radar and visual information in a holographic intersection scene according to claim 1, characterized in that, Step S2 includes the following steps: S21. The feature points in the top-view coordinate system correspond to the feature points in the camera pixel coordinate system to obtain feature point pairs; S22. Calculate the thin plate spline transformation based on the feature points in step S21 to obtain the transformation equation.

3. The method for fusing radar and visual information in a holographic intersection scene according to claim 1, characterized in that, Step S3 includes the following steps: S31. When installing radar equipment, corner reflectors are set up at intersections. The coordinates of the point set in the top view coordinate system are obtained in the radar coordinate system through the corner reflectors, thus obtaining the point set pair. S32. Based on the point set pair in step S31, calculate its affine transformation matrix to obtain the transformation matrix from the radar coordinate system to the top-view coordinate system, which is used to map the target obtained by the radar in the radar coordinate system to the top-view coordinate system in real time.

4. The method for fusing radar and visual information in a holographic intersection scene according to claim 3, characterized in that, In step S3, it is assumed that the coordinates of the target point in the top view coordinate system in the radar coordinate system are as follows: (114.419858, 23.12206), (114.419902, 23.122015), (114.419877, 23.121952), (114.419828, 23.122006), (114.419834, 23.121971), (114.419842, 23.121926), (114.419693, 23.121966), (114.419701, 23.121939). The corresponding coordinates in the top view coordinate system are: (2359,2456), (2404,2525), (2356,2601), (2309,2520), (2309,2569), (2308,2629), (2133,2542), (2137,2582). The final transformation matrix is:

5. The method for fusing radar and visual information in a holographic intersection scene according to claim 1, characterized in that, The top-view coordinate system is the image coordinate system of the aerial top view, with the origin at the top left corner, the Y-axis pointing downwards, and the X-axis pointing to the right, with the unit being pixels.

6. The method for fusing radar and visual information in a holographic intersection scene according to claim 1, characterized in that, The radar coordinate system is defined by point O, which represents the radar installation location. Based on the radar installation location, the lower left corner is the radar installation position, meaning the upward direction is the Y-axis and the leftward direction is the X-axis.

7. The method for fusing radar and visual information in a holographic intersection scene according to claim 1, characterized in that, The camera pixel coordinate system is the image output by the camera. In this coordinate system, the origin is the upper left corner of the image, the X-axis extends to the right, the Y-axis extends downward, and each pixel has a unique pixel coordinate.

Images