A radar data and video data fusion method and system based on perspective transformation

Abstract

The present invention discloses a method and system for fusing radar data and video data based on perspective transformation, which relates to the field of intelligent intersection monitoring, including: obtaining the coordinate perspective transformation coefficient of the monitored road area in advance, and obtaining the position coordinates of a preset number of vehicle targets on a bird's-eye view through the radar sensor and obtaining the pixel coordinates of the preset number of vehicle targets after perspective transformation to obtain position coordinate transformation parameters; when there are vehicle targets in the image of the camera monitoring area, the position information of the vehicle targets can be obtained in real time according to the pixel coordinates of the vehicle targets in the image, and according to the position information of the vehicle targets on the bird's-eye view obtained by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched, so that the actual data in meters monitored by the camera can be obtained in real time, thereby improving the accuracy and efficiency of intersection monitoring management based on radar and camera, and realizing the fusion of camera data and radar data.

Description

Technical Field

[0001] The present invention relates to the field of intelligent intersection monitoring, and in particular to a radar data and video data fusion method and system based on perspective transformation. Background Art

[0002] With the continuous development of intersection management technology, the current planning and management of lane lines at intersections usually involves identifying and analyzing images captured by cameras to obtain lane line positions. In the holographic image display and tracking of vehicles at intersections that are fused with radar data and video data, it is necessary to fuse the lane line position information in the video tracking data with the lane line position information in the radar data. However, since the data collected by the camera is projected onto the phase plane, the data of the overhead plane cannot be obtained. At the same time, since the data collected by the camera is in pixels, the actual data in meters monitored by the radar cannot be obtained, which greatly limits the use of the fusion algorithm when planning lane lines and tracking vehicle targets. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a radar data and video data fusion method and system based on perspective transformation, which can solve the problem that when lane line planning and management and vehicle target tracking management are currently performed based on multiple sensors, the actual data in meters monitored by the camera cannot be obtained, which leads to the problem that the use of the fusion algorithm is greatly limited when managing lane lines and tracking vehicle targets.

[0004] To achieve the above object, on the one hand, the present invention provides a radar data and video data fusion method based on perspective transformation, the method comprising:

[0005] Obtaining pixel coordinates after perspective transformation according to pixel coordinates of a preset number of position points in a preset rectangular area of ​​the monitored road and rectangular area position parameters corresponding to the preset rectangle in the image;

[0006] Obtaining a coordinate perspective transformation coefficient according to the pixel coordinates and the pixel coordinates after perspective transformation;

[0007] Generating a road bird's-eye view according to a preset width of the monitored road and configuring the monitoring target radar data of the monitored road on the bird's-eye view;

[0008] Acquire the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor and acquire the perspective-transformed pixel coordinates of the preset number of vehicle targets;

[0009] Acquire position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation, and the preset perspective transformation area height value; acquire position coordinate transformation parameters;

[0010] Determining the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target;

[0011] According to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched.

[0012] Furthermore, the step of obtaining coordinate perspective transformation coefficients according to the pixel coordinates and the pixel coordinates after perspective transformation includes:

[0013] Generate a first transformation matrix according to the perspective transformed pixel coordinates corresponding to a preset number of position points within a preset rectangular area of ​​the monitored road;

[0014] Generate a second transformation matrix according to the pixel coordinates corresponding to the preset number of position points within the preset rectangular area of ​​the monitored road and the pixel coordinates after perspective transformation;

[0015] The coordinate perspective transformation coefficients are obtained according to the first transformation matrix and the second transformation matrix.

[0016] Furthermore, before the step of generating a road bird's-eye view according to the preset width of the monitored road and configuring the monitoring target radar data of the monitored road on the bird's-eye view, the method further includes:

[0017] generating a perspective-transformed image according to the coordinate perspective transformation coefficients;

[0018] Determine whether each lane line in the perspective transformed image is parallel and perpendicular to the x-axis;

[0019] If so, it is confirmed that the coordinate perspective transformation coefficient meets the preset conditions.

[0020] Furthermore, before the step of determining the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target, the method further includes:

[0021] Acquire pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line;

[0022] Generate and display the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of a preset number of position points on the lane line;

[0023] Determining whether the lane line matches the corresponding initial lane line on the bird's-eye view;

[0024] If they match, it is confirmed that the position coordinate transformation parameters meet the preset conditions;

[0025] If there is no match, the position coordinate transformation parameter is adjusted according to the difference between the lane line and the corresponding initial lane line on the bird's-eye view.

[0026] Furthermore, the step of acquiring pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line includes:

[0027] If the lane line is a straight line, obtaining pixel coordinates of two endpoints on the lane line;

[0028] If the lane line is a broken line, obtaining pixel coordinates of two endpoints and each broken point on the lane line;

[0029] If the lane line is an arc, the pixel coordinates of a preset number of position points are obtained according to the arc of the lane line, and the number of the preset number of position points is proportional to the arc of the arc.

[0030] On the other hand, the present invention provides a radar data and video data fusion system based on perspective transformation, the system comprising: an acquisition unit, for acquiring pixel coordinates after perspective transformation according to pixel coordinates of a preset number of position points in a preset rectangular area of ​​a monitored road and rectangular area position parameters corresponding to the preset rectangle in an image;

[0031] The acquisition unit is further used to acquire a coordinate perspective transformation coefficient according to the pixel coordinates and the pixel coordinates after perspective transformation; generate a road bird's-eye view according to the preset width of the monitored road and configure the monitoring target radar data of the monitored road on the bird's-eye view; acquire the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor and acquire the pixel coordinates of the preset number of vehicle targets after perspective transformation; acquire position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation and the preset perspective transformation area height value;

[0032] The determining unit determines the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target;

[0033] A matching unit is used to match the time between the camera data and the radar data of each target in the monitored road area according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor.

[0034] Furthermore, the acquisition unit is specifically used to generate a first transformation matrix based on the perspective transformed pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitored road; generate a second transformation matrix based on the pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitored road and the perspective transformed pixel coordinates; and acquire the coordinate perspective transformation coefficients based on the first transformation matrix and the second transformation matrix.

[0035] Furthermore, the system also includes: a first verification unit; the first verification unit is used to generate a perspective transformed image based on the coordinate perspective transformation coefficient; determine whether the lane lines in the perspective transformed image are parallel and perpendicular to the x-coordinate axis; if so, confirm that the coordinate perspective transformation coefficient meets the preset conditions.

[0036] Further, the system further comprises: a second inspection unit;

[0037] The second verification unit is used to obtain pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line; generate and display the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line; determine whether the lane line matches the corresponding initial lane line on the bird's-eye view; if so, confirm that the position coordinate transformation parameters meet the preset conditions; if not, adjust the position coordinate transformation parameters according to the difference between the lane line and the corresponding initial lane line on the bird's-eye view.

[0038] Furthermore, the acquisition unit is specifically used to acquire the pixel coordinates of two endpoints from the lane line if the lane line is a straight line; acquire the pixel coordinates of two endpoints and each inflection point from the lane line if the lane line is a broken line; and acquire the pixel coordinates of a preset number of position points according to the curvature of the lane line if the lane line is an arc, wherein the number of the preset number of position points is proportional to the curvature of the arc.

[0039] The present invention provides a radar data and video data fusion method and system based on perspective transformation. First, according to the pixel coordinates of a preset number of position points in a preset rectangular area of ​​a monitored road and the rectangular area position parameters corresponding to the preset rectangle in the image, the pixel coordinates after perspective transformation are obtained, and then the coordinate perspective transformation coefficient is obtained according to the pixel coordinates and the pixel coordinates after perspective transformation, and the position coordinate transformation parameters are obtained according to the position coordinates of the preset number of vehicle targets obtained by the radar sensor, the pixel coordinates of the preset number of vehicle targets after perspective transformation, and the preset perspective transformation area height value; then, the position information of the vehicle target is determined according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target; finally, according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view obtained by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched. The present invention obtains the coordinate perspective transformation coefficient of the monitored road area in advance, obtains the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor, and obtains the pixel coordinates of the preset number of vehicle targets after perspective transformation to obtain the position coordinate transformation parameters. When there are vehicle targets in the image of the camera monitoring area, the position information of the vehicle targets can be obtained in real time according to the pixel coordinates of the vehicle targets in the image, and the time between the camera data and the radar data of each target in the monitored road area is matched according to the position information of the vehicle targets on the bird's-eye view obtained by the radar sensor. Therefore, when the intersection monitoring and management is performed based on multiple sensors, the actual data in meters monitored by the camera can be obtained in real time, which improves the accuracy and efficiency of the intersection monitoring and management based on radar and camera, and realizes the fusion of camera data and radar data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a radar data and video data fusion method based on perspective transformation provided by the present invention;

[0041] Figure 2 It is a structural schematic diagram of a radar data and video data fusion system based on perspective transformation provided by the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, a radar data and video data fusion method based on perspective transformation provided by an embodiment of the present invention includes the following steps:

[0044] 101. Obtain perspective-transformed pixel coordinates according to pixel coordinates of a preset number of position points in a preset rectangular area of ​​a monitored road and rectangular area position parameters corresponding to the preset rectangle in an image.

[0045] The preset number may be 3, 4, 5, etc., which is not limited in the embodiment of the present invention.

[0046] For the embodiment of the present invention, step 101 may specifically include: acquiring pixel coordinates after perspective transformation according to the pixel coordinates and the width and height of a rectangular area corresponding to the preset rectangle in the image.

[0047] Specifically, for example, the pixel coordinates of the four vertices are used counterclockwise to construct a rectangle, and then the pixel coordinates after perspective transformation are calculated according to the formulas Y=[1,1,w,w] and X=[1,h,h,1], where X and Y are the pixel coordinates after perspective transformation, w is the width of the rectangular area corresponding to the preset rectangle in the image, and h is the height of the rectangular area corresponding to the preset rectangle in the image.

[0048] 102. Obtain coordinate perspective transformation coefficients according to the pixel coordinates and the pixel coordinates after perspective transformation.

[0049] For the embodiment of the present invention, step 102 may specifically include: generating a first transformation matrix based on the perspective transformed pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitoring road; generating a second transformation matrix based on the pixel coordinates corresponding to the preset number of position points within the preset rectangular area of ​​the monitoring road and the perspective transformed pixel coordinates; and obtaining the coordinate perspective transformation coefficients based on the first transformation matrix and the second transformation matrix.

[0050] Specifically, for example, first according to the formula Generate the first transformation matrix, and then use the formula Generate the second transformation matrix, and finally according to the formula f = A - 1 B calculates the coordinate perspective transformation coefficient, where B is the first transformation matrix, A is the second transformation matrix, f is the coordinate perspective transformation coefficient, X1-X4, Y1-Y4 are the pixel coordinates of the four points in the rectangular area corresponding to the preset rectangle in the image after perspective transformation, and x1-x4, y1-y4 are the pixel coordinates of the four points in the preset rectangle.

[0051] For the embodiment of the present invention, in order to further improve the accuracy of the coordinate perspective transformation coefficient, before performing step 103, it also includes: generating a perspective transformed image based on the coordinate perspective transformation coefficient; determining whether the lane lines in the perspective transformed image are parallel and perpendicular to the x-coordinate axis; if so, confirming that the coordinate perspective transformation coefficient meets the preset conditions.

[0052] 103. Generate a road bird's-eye view according to the preset width of the monitored road and configure the monitoring target radar data of the monitored road on the bird's-eye view.

[0053] Among them, the preset width of the monitored road can be pre-acquired based on existing map tools.

[0054] 104. Obtain position coordinates of a preset number of vehicle targets on the bird's-eye view and obtain perspective-transformed pixel coordinates of the preset number of vehicle targets through the radar sensor.

[0055] The number of preset vehicles is usually two, and the two vehicles cannot be collinear on a horizontal line or a vertical line in a camera or radar coordinate system.

[0056] 105. Obtain position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation, and the preset perspective transformation area height value.

[0057] For the embodiment of the present invention, step 105 may specifically include: according to formula X m =k x Xb x and Y m =k y (-Y+h)+b y Get the position coordinate transformation parameters, where (X, Y) is the pixel coordinate after perspective transformation, (X m ,Y m ) is the position coordinate of a preset number of vehicle targets, H is the height of the known perspective transformation area, (k, b) is the position coordinate transformation parameter, the physical meaning of k is how many meters each pixel represents, and the physical meaning of b is the distance between the rectangular area selected by the perspective transformation and the x-axis and y-axis respectively. Specifically, the position coordinate transformation parameters can be calculated based on the position coordinates of the preset number of vehicle targets and the pixel coordinates after perspective transformation.

[0058] For an embodiment of the present invention, in order to further improve the accuracy of the position coordinate transformation parameters, before step 106, it may also include: obtaining pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line; generating and displaying the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line; determining whether the lane line matches the initial lane line corresponding to the bird's-eye view; if they match, confirming that the position coordinate transformation parameters meet the preset conditions; if they do not match, adjusting the position coordinate transformation parameters according to the difference between the lane line and the initial lane line corresponding to the bird's-eye view.

[0059] For the embodiment of the present invention, the step of obtaining pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line may specifically include: if the lane line is a straight line, obtaining pixel coordinates of two endpoints from the lane line; if the lane line is a broken line, obtaining pixel coordinates of two endpoints and each broken point from the lane line; if the lane line is an arc, obtaining pixel coordinates of a preset number of position points according to the curvature of the lane line, the number of the preset number of position points is proportional to the curvature of the arc, that is, the larger the curvature, the more position points are selected, thereby further improving the accuracy of lane line acquisition.

[0060] For an embodiment of the present invention, the step of generating and displaying the lane line on the image according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line may specifically include: obtaining the actual position coordinates of a preset number of position points on the lane line according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line, and then connecting each position point according to the actual position coordinates of the preset number of position points on the lane line to generate the lane line.

[0061] 106. Determine the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target.

[0062] For the embodiment of the present invention, step 106 may specifically include: constructing a vehicle target transformation matrix according to the pixel coordinates of the vehicle target; generating the pixel coordinates of the vehicle target after perspective transformation according to the vehicle target transformation matrix and the coordinate perspective transformation coefficients; obtaining the position information of the vehicle target according to the pixel coordinates of the vehicle target after perspective transformation and the position coordinate transformation parameters.

[0063] Specifically, first construct the vehicle target transformation matrix according to the formula c = [xcar ycar 1], and then according to the formula and Generate the pixel coordinates of the vehicle target after perspective transformation, and finally use the formula Y′car = k y (-Y car + H) + b v and X′ car = k x X car-b x Obtain the position information of the vehicle target, where cut=c·fa, c is the vehicle target transformation matrix, xcar, ycar are the pixel coordinates of the vehicle target, Xcar, Ycar are the pixel coordinates of the vehicle target after perspective transformation, cut is the spatial coordinate after perspective transformation, fa is the coordinate perspective transformation coefficient, and 1-3 respectively represent the first point to the third point of the preset rectangle.

[0064] 107. Match the time between the camera data and the radar data of each target in the monitored road area according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor.

[0065] A method for fusing radar data and video data based on perspective transformation is provided in an embodiment of the present invention. First, according to the pixel coordinates of a preset number of position points in a preset rectangular area of ​​a monitored road and the position parameters of a rectangular area corresponding to the preset rectangle in an image, the pixel coordinates after perspective transformation are obtained. Then, according to the pixel coordinates and the pixel coordinates after perspective transformation, the coordinate perspective transformation coefficient is obtained. According to the position coordinates of the preset number of vehicle targets obtained by the radar sensor, the pixel coordinates of the preset number of vehicle targets after perspective transformation and the height value of the preset perspective transformation area, the position coordinate transformation parameters are obtained. Then, according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters and the pixel coordinates of the vehicle target, the position information of the vehicle target is determined. Finally, according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view obtained by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched. The present invention obtains the coordinate perspective transformation coefficient of the monitored road area in advance, obtains the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor, and obtains the pixel coordinates of the preset number of vehicle targets after perspective transformation to obtain the position coordinate transformation parameters. When there are vehicle targets in the image of the camera monitoring area, the position information of the vehicle targets can be obtained in real time according to the pixel coordinates of the vehicle targets in the image, and the time between the camera data and the radar data of each target in the monitored road area is matched according to the position information of the vehicle targets on the bird's-eye view obtained by the radar sensor. Therefore, when the intersection monitoring and management is performed based on multiple sensors, the actual data in meters monitored by the camera can be obtained in real time, which improves the accuracy and efficiency of the intersection monitoring and management based on radar and camera, and realizes the fusion of camera data and radar data.

[0066] To implement the method provided by the embodiment of the present invention, the embodiment of the present invention provides a radar data and video data fusion system based on perspective transformation, such as Figure 2 As shown, the system includes: an acquisition unit 21, a determination unit 22, a matching unit 23, a first verification unit 24, and a second verification unit 25.

[0067] The acquisition unit 21 is used to acquire the pixel coordinates after perspective transformation according to the pixel coordinates of a preset number of position points in the preset rectangular area of ​​the monitored road and the rectangular area position parameters corresponding to the preset rectangle in the image.

[0068] Among them, since the rectangular road presented in the overhead view after the coordinate transformation is a rectangle rather than other quadrilaterals, four points of a rectangle on the road are selected for actual calibration measurement.

[0069] The acquisition unit 21 is also used to acquire coordinate perspective transformation coefficients according to the pixel coordinates and the pixel coordinates after perspective transformation; generate a road bird's-eye view according to the preset width of the monitored road and configure the monitoring target radar data of the monitored road on the bird's-eye view; acquire the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor and acquire the pixel coordinates of the preset number of vehicle targets after perspective transformation; acquire position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation and the preset perspective transformation area height value.

[0070] The determination unit 22 is used to determine the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target.

[0071] The matching unit 23 is used to match the time between the camera data and the radar data of each target in the monitored road area according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor.

[0072] Furthermore, the acquisition unit 21 is specifically used to generate a first transformation matrix based on the perspective transformed pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitored road; generate a second transformation matrix based on the pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitored road and the perspective transformed pixel coordinates; and acquire the coordinate perspective transformation coefficients based on the first transformation matrix and the second transformation matrix.

[0073] Furthermore, the first inspection unit 24 is used to generate a perspective transformed image based on the coordinate perspective transformation coefficient; determine whether the lane lines in the perspective transformed image are parallel and perpendicular to the x-coordinate axis; if so, confirm that the coordinate perspective transformation coefficient meets the preset conditions.

[0074] Furthermore, the second verification unit 25 is used to obtain pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line; generate and display the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line; determine whether the lane line matches the corresponding initial lane line on the bird's-eye view; if they match, confirm that the position coordinate transformation parameters meet the preset conditions; if they do not match, adjust the position coordinate transformation parameters according to the difference between the lane line and the corresponding initial lane line on the bird's-eye view.

[0075] Furthermore, the acquisition unit 21 is specifically used to acquire the pixel coordinates of the two endpoints from the lane line if the lane line is a straight line; acquire the pixel coordinates of the two endpoints and each inflection point from the lane line if the lane line is a broken line; and acquire the pixel coordinates of a preset number of position points according to the curvature of the lane line if the lane line is an arc, wherein the number of the preset number of position points is proportional to the curvature of the arc.

[0076] The present invention provides a radar data and video data fusion system based on perspective transformation. First, according to the pixel coordinates of a preset number of position points in a preset rectangular area of ​​a monitored road and the position parameters of a rectangular area corresponding to the preset rectangle in an image, the pixel coordinates after perspective transformation are obtained, and then the coordinate perspective transformation coefficients are obtained according to the pixel coordinates and the pixel coordinates after perspective transformation, and the position coordinate transformation parameters are obtained according to the position coordinates of the preset number of vehicle targets obtained by a radar sensor, the pixel coordinates of the preset number of vehicle targets after perspective transformation, and the height value of the preset perspective transformation area; then, the position information of the vehicle target is determined according to the coordinate perspective transformation coefficients, the position coordinate transformation parameters, and the pixel coordinates of the vehicle target; finally, according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view obtained by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched. The present invention obtains the coordinate perspective transformation coefficient of the monitored road area in advance, obtains the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor, and obtains the pixel coordinates of the preset number of vehicle targets after perspective transformation to obtain the position coordinate transformation parameters. When there are vehicle targets in the image of the camera monitoring area, the position information of the vehicle targets can be obtained in real time according to the pixel coordinates of the vehicle targets in the image, and the time between the camera data and the radar data of each target in the monitored road area is matched according to the position information of the vehicle targets on the bird's-eye view obtained by the radar sensor. Therefore, when the intersection monitoring and management is performed based on multiple sensors, the actual data in meters monitored by the camera can be obtained in real time, which improves the accuracy and efficiency of the intersection monitoring and management based on radar and camera, and realizes the fusion of camera data and radar data.

[0077] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0078] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0079] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0080] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0081] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0082] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic system, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing systems, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0083] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0084] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage systems, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and Blu-ray disk. Disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers. Combinations of the above may also be included in computer readable media.

[0085] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A radar data and video data fusion method based on perspective transformation, characterized in that: The method comprises: Obtaining pixel coordinates after perspective transformation according to pixel coordinates of a preset number of position points in a preset rectangular area of ​​the monitored road and rectangular area position parameters corresponding to the preset rectangle in the image; Obtaining a coordinate perspective transformation coefficient according to the pixel coordinates and the pixel coordinates after perspective transformation; Generating a road bird's-eye view according to a preset width of the monitored road and configuring the monitoring target radar data of the monitored road on the bird's-eye view; Acquire the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor and acquire the perspective-transformed pixel coordinates of the preset number of vehicle targets; Acquire position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation, and the preset perspective transformation area height value; Determining the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target; According to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor, the time between the camera data and the radar data of each target in the monitored road area is matched.

2. The radar data and video data fusion method based on perspective transformation according to claim 1, characterized in that: The step of obtaining the coordinate perspective transformation coefficient according to the pixel coordinates and the pixel coordinates after perspective transformation comprises: Generate a first transformation matrix according to the perspective transformed pixel coordinates corresponding to a preset number of position points within a preset rectangular area of ​​the monitored road; Generate a second transformation matrix according to the pixel coordinates corresponding to the preset number of position points within the preset rectangular area of ​​the monitored road and the pixel coordinates after perspective transformation; The coordinate perspective transformation coefficients are obtained according to the first transformation matrix and the second transformation matrix.

3. The radar data and video data fusion method based on perspective transformation according to claim 1 or 2, characterized in that: Before the step of generating a road bird's-eye view according to the preset width of the monitored road and configuring the monitoring target radar data of the monitored road on the bird's-eye view, the method further includes: generating a perspective-transformed image according to the coordinate perspective transformation coefficients; Determine whether each lane line in the perspective transformed image is parallel and perpendicular to the x-axis; If so, it is confirmed that the coordinate perspective transformation coefficient meets the preset conditions.

4. The radar data and video data fusion method based on perspective transformation according to claim 1, characterized in that: Before the step of determining the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target, the method further includes: Acquire pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line; Generate and display the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of a preset number of position points on the lane line; Determining whether the lane line matches the corresponding initial lane line on the bird's-eye view; If they match, it is confirmed that the position coordinate transformation parameters meet the preset conditions; If there is no match, the position coordinate transformation parameter is adjusted according to the difference between the lane line and the corresponding initial lane line on the bird's-eye view.

5. The method for fusing radar data and video data based on perspective transformation according to claim 4, characterized in that: The step of acquiring pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line comprises: If the lane line is a straight line, obtaining pixel coordinates of two endpoints on the lane line; If the lane line is a broken line, obtaining pixel coordinates of two endpoints and each broken point on the lane line; If the lane line is an arc, the pixel coordinates of a preset number of position points are obtained according to the arc of the lane line, and the number of the preset number of position points is proportional to the arc of the arc.

6. A radar data and video data fusion system based on perspective transformation, characterized in that: The system comprises: An acquisition unit, configured to acquire pixel coordinates after perspective transformation according to pixel coordinates of a preset number of position points in a preset rectangular area of ​​the monitored road and rectangular area position parameters corresponding to the preset rectangle in the image; The acquisition unit is further used to acquire a coordinate perspective transformation coefficient according to the pixel coordinates and the pixel coordinates after perspective transformation; generate a road bird's-eye view according to the preset width of the monitored road and configure the monitoring target radar data of the monitored road on the bird's-eye view; acquire the position coordinates of a preset number of vehicle targets on the bird's-eye view through the radar sensor and acquire the pixel coordinates of the preset number of vehicle targets after perspective transformation; acquire position coordinate transformation parameters according to the position coordinates of the preset number of vehicle targets, the pixel coordinates of the preset number of vehicle targets after perspective transformation and the preset perspective transformation area height value; A determination unit, configured to determine the position information of the vehicle target according to the coordinate perspective transformation coefficient, the position coordinate transformation parameter, and the pixel coordinates of the vehicle target; A matching unit is used to match the time between the camera data and the radar data of each target in the monitored road area according to the position information of the vehicle target and the position information of the vehicle target on the bird's-eye view acquired by the radar sensor.

7. The radar data and video data fusion system based on perspective transformation according to claim 6, characterized in that: The acquisition unit is specifically used to generate a first transformation matrix according to the perspective transformed pixel coordinates corresponding to a preset number of position points within the preset rectangular area of ​​the monitored road; generate a second transformation matrix according to the pixel coordinates corresponding to the preset number of position points within the preset rectangular area of ​​the monitored road and the perspective transformed pixel coordinates; and acquire coordinate perspective transformation coefficients according to the first transformation matrix and the second transformation matrix.

8. The radar data and video data fusion system based on perspective transformation according to claim 6, characterized in that: The system also includes: a first verification unit; the first verification unit is used to generate a perspective transformed image based on the coordinate perspective transformation coefficient; determine whether the lane lines in the perspective transformed image are parallel and perpendicular to the x-coordinate axis; if so, confirm that the coordinate perspective transformation coefficient meets the preset conditions.

9. The radar data and video data fusion system based on perspective transformation according to claim 6, characterized in that: The system further comprises: a second verification unit; The second verification unit is used to obtain pixel coordinates of a preset number of position points on the lane line according to the shape corresponding to the lane line; generate and display the lane line on the bird's-eye view according to the coordinate perspective transformation coefficient, the position coordinate transformation parameters, and the pixel coordinates of a preset number of position points on the lane line; determine whether the lane line matches the corresponding initial lane line on the bird's-eye view; if so, confirm that the position coordinate transformation parameters meet the preset conditions; if not, adjust the position coordinate transformation parameters according to the difference between the lane line and the corresponding initial lane line on the bird's-eye view.

10. The radar data and video data fusion system based on perspective transformation according to claim 6, characterized in that: The acquisition unit is specifically used to acquire the pixel coordinates of the two endpoints from the lane line if the lane line is a straight line; acquire the pixel coordinates of the two endpoints and each inflection point from the lane line if the lane line is a broken line; and acquire the pixel coordinates of a preset number of position points according to the curvature of the lane line if the lane line is an arc, wherein the number of the preset number of position points is proportional to the curvature of the arc.

Images