Roadside camera calibration method and equipment

Abstract

The invention relates to a calibration method of a roadside camera. The method comprises the following steps: receiving position information of a plurality of marks on a vehicle and corresponding timestamps; based on the timestamp, matching an image acquired by the roadside camera with the position information of the plurality of marks; and calculating parameters of the roadside camera based on the matched image and position information. The invention further relates to calibration equipment of the roadside camera, a roadside camera system, a computer readable storage medium, a computer program product and a vehicle.

Description

Technical Field

[0001] The present application relates to the field of calibration of roadside cameras, and more particularly, to a calibration method and device for roadside cameras, a roadside camera system, a computer-readable storage medium, a computer program product, and a vehicle. Background Art

[0002] Generally, roadside cameras need to be calibrated offline before they can be used. Since roadside cameras are installed by the roadside, traditional calibration processes often require blocking traffic flow and performing measurements and calibrations outdoors. This process is very complex and time-consuming. Moreover, even in bad weather, measurement personnel need to work outdoors.

[0003] In addition, on highways, the traffic flow is dense, and it is very dangerous to perform traditional calibration processes on roadside cameras without blocking the traffic flow.

[0004] Therefore, it is desirable to provide an improved calibration scheme for roadside cameras. Summary of the Invention

[0005] According to one aspect of the present application, there is provided a calibration method for a roadside camera, the method comprising: receiving position information of a plurality of markers on a vehicle and corresponding timestamps; based on the timestamps, matching an image acquired by the roadside camera with the position information of the plurality of markers; and calculating parameters of the roadside camera based on the matched image and position information.

[0006] As a supplement or replacement to the above solution, in the above method, receiving position information of a plurality of markers on a vehicle and corresponding timestamps includes: receiving the position information of the plurality of markers and corresponding timestamps from a positioning device installed on the vehicle, wherein the positions of the plurality of markers are fixed relative to the positioning device.

[0007] As a supplement or replacement to the above solution, in the above method, based on the timestamps, matching an image acquired by the roadside camera with the position information of the plurality of markers includes: acquiring an image containing the plurality of markers on the vehicle; calculating coordinate information of the plurality of markers in a pixel coordinate system, the coordinate information being timestamped; looking up position information of the plurality of markers with the same timestamp; and matching the position information of the plurality of markers with the same timestamp with the coordinate information of the plurality of markers in the pixel coordinate system.

[0008] As a supplement or replacement to the above solution, in the above method, the coordinate information of the multiple markers received is the coordinate information (x, y, z) in the world coordinate system, while the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and based on the matched image and position information, calculating the parameters of the roadside camera includes: determining the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

[0009] As a supplement or replacement to the above solution, in the above method, the parameters of the roadside camera are external parameters, and the external parameters include a translation matrix T and a rotation matrix R.

[0010] According to another aspect of the present application, there is provided a calibration device for a roadside camera, the device including: a receiving device configured to receive the position information of multiple markers on a vehicle and corresponding timestamps; a matching device configured to match the image acquired by the roadside camera with the position information of the multiple markers based on the timestamps; and a calculating device configured to calculate the parameters of the roadside camera based on the matched image and position information.

[0011] As a supplement or replacement to the above solution, in the above device, the receiving device is configured to: receive the position information of the multiple markers and corresponding timestamps from a positioning device installed on the vehicle, where the positions of the multiple markers are fixed relative to the positioning device.

[0012] As a supplement or replacement to the above solution, in the above device, the matching device is configured to: acquire an image including multiple markers on the vehicle; calculate the coordinate information of the multiple markers in the pixel coordinate system, where the coordinate information carries a timestamp; find the position information of the multiple markers with the same timestamp; and match the position information of the multiple markers with the same timestamp with the coordinate information of the multiple markers in the pixel coordinate system.

[0013] As a supplement or replacement to the above solution, in the above device, the coordinate information of the multiple markers received is the coordinate information (x, y, z) in the world coordinate system, while the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and the calculating device is configured to: determine the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

[0014] As a supplement or replacement to the above solution, in the above device, the parameters of the roadside camera are external parameters, and the external parameters include a translation matrix T and a rotation matrix R.

[0015] According to another aspect of the present application, a roadside camera system is provided. The roadside camera system includes a roadside camera and a roadside unit RSU that communicates with the roadside camera, wherein the roadside unit RSU includes the calibration device as described above.

[0016] According to another aspect of the present application, a computer-readable storage medium is provided. The medium includes instructions that, when running, execute the method as described above.

[0017] According to another aspect of the present application, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described above.

[0018] According to another aspect of the present application, a vehicle is provided. The vehicle includes an on-vehicle unit configured to communicate with the calibration device as described above, so that the roadside camera can perform online calibration.

[0019] As a supplement or replacement to the above solution, in the above vehicle, a plurality of markers are installed on the roof and / or body of the vehicle, and the relative positions between the markers and the positioning device installed on the vehicle are fixed.

[0020] As a supplement or replacement to the above solution, in the above vehicle, the vehicle is configured to travel on multiple lanes within the field of view of the roadside camera.

[0021] The calibration scheme of the roadside camera in the embodiment of the present application performs online calibration by receiving the position information of a plurality of markers on the vehicle and the corresponding timestamps, matching the images obtained by the roadside camera with the position information of the plurality of markers based on the timestamps, and finally calculating the parameters of the roadside camera based on the matched images and position information. The above solution completes online calibration by collecting measurement data of a plurality of markers on the vehicle without blocking the traffic flow. And the entire calibration work does not require marking any special points on the ground, nor does it require outdoor data collection. If a plurality of roadside cameras are located on the same road, the calibration work of the plurality of roadside cameras can be completed simultaneously by the vehicle traveling on the road multiple times. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] From the following detailed description in conjunction with the accompanying drawings, the above and other objects and advantages of the present application will become more fully clear, wherein the same or similar elements are denoted by the same reference numerals.

[0023] Figure 1 A flowchart showing the calibration method of a roadside camera according to an embodiment of the present application is shown;

[0024] Figure 2The structural schematic diagram of the calibration device for the roadside camera according to an embodiment of the present application is shown;

[0025] Figure 3 The structural schematic diagram of the roadside camera system according to an embodiment of the present application is shown;

[0026] Figure 4 The schematic diagram of the scenario for online calibration of the roadside camera according to an embodiment of the present application is shown; and

[0027] Figure 5 The schematic diagram of the vehicle with multiple markers according to an embodiment of the present application is shown. Detailed implementation manners

[0028] Hereinafter, the calibration scheme of the roadside camera according to the exemplary embodiments of the present application will be described in detail with reference to the drawings.

[0029] Figure 1 The flowchart of the calibration method 1000 for the roadside camera according to an embodiment of the present application is shown. As Figure 1 shown, the calibration method 1000 for the roadside camera includes:

[0030] In step S110, the position information of multiple markers on the vehicle and the corresponding timestamps are received;

[0031] In step S120, based on the timestamps, the images acquired by the roadside camera are matched with the position information of the multiple markers; and

[0032] In step S130, based on the matched images and position information, the parameters of the roadside camera are calculated.

[0033] In the context of the present application, the term "roadside camera" is also referred to as a road camera, which is installed on various roads (including intersections, highways, etc.) or around the roads. In one or more embodiments, classified by function, the roadside camera may include: a violation camera, a speed measurement camera, a parking violation capture camera, and a public security camera. Among them, the violation camera is generally used to detect violations such as running a red light, crossing the line, and going in the wrong direction, the speed measurement camera is used to capture the vehicle speeding behavior, the parking violation capture camera is used to capture the vehicle's illegal parking, and the public security camera is used by the public security department to monitor the public security of each road, such as being installed at various intersections, sections, communities, etc.

[0034] In step S110, the position information of multiple markers on the vehicle and the corresponding timestamps are received. In one or more embodiments, the multiple markers may be pre-installed on the roof and body of the vehicle so that the roadside camera can capture these markers. In one embodiment, the number of markers is 3 or more. Those skilled in the art can understand that the more the number of markers, the higher the accuracy of subsequent calibration of the roadside camera.

[0035] In one embodiment, the position information of the multiple markers and the corresponding timestamps are obtained by a positioning device (such as an RTK device). For example, step S110 may include: receiving the position information of the multiple markers and the corresponding timestamps from a positioning device (such as an RTK device) installed on the vehicle, where the positions of the multiple markers are fixed relative to the positioning device. Taking the RTK device as an example, the RTK device can obtain the precise position information of the RTK device through real-time kinematic carrier phase differential technology and assign very precise timestamps to the recorded position information, etc. Thus, based on the available precise position information of the RTK device, the precise position information of each marker can be easily deduced.

[0036] In step S120, based on the timestamps, the images obtained by the roadside camera are matched with the position information of the multiple markers. In one embodiment, step S120 includes: obtaining an image containing multiple markers (such as 3 or more colored markers) on the vehicle; calculating the coordinate information of the multiple markers in the pixel coordinate system, where the coordinate information has a timestamp (for example, the camera can perform precise time synchronization on its own system through the GPS system time service, so that each frame of image recorded by the camera can be assigned a precise timestamp); finding the position information of the multiple markers with the same timestamp; and matching the position information of the multiple markers with the same timestamp with the coordinate information of the multiple markers in the pixel coordinate system. For example, each frame of the image output by the camera can use this timestamp to find the marker position information corresponding to that time, similar to an operation of a lookup list.

[0037] In one or more embodiments, the received coordinate information of the multiple markers is the coordinate information (x, y, z) in the world coordinate system, while the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and step S130 includes: determining the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

[0038] In one embodiment, the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system satisfy the following conversion relationship:

[0039]

[0040] In the above formula, Zc is a scale factor (not zero), representing the effective focal length (the distance from the optical center to the image plane), which is the homogeneous coordinate of the image point in the image coordinate system. f x and f y are respectively called the normalized focal lengths on the x-axis and y-axis, and f x = f / dx, f y = f / dy, where f is the focal length of the camera in mm, dx and dy are the pixel sizes, and u 0 and v 0 are the image centers. R is the rotation matrix, and T is the translation matrix.

[0041] In addition, represents the internal parameters of the camera, while represents the external parameters of the camera. In one embodiment, the calibration method 1000 of the roadside camera is mainly used to calibrate the external parameters (assuming the internal parameters are known). In other words, the external parameters that need to be calibrated / calculated include the translation matrix T and the rotation matrix R. It should be noted that those skilled in the art can understand that the external parameters can also be represented in other forms besides the translation matrix T and the rotation matrix R, for example, represented by (x, y, z, Φ, θ, ψ), where Φ represents the roll angle, θ represents the pitch angle, and ψ represents the yaw angle.

[0042] In addition, it is easy for those skilled in the art to understand that the calibration method 1000 of the roadside camera provided by one or more embodiments of the present application can be implemented by a computer program. For example, the computer program is included in a computer program product, and when the computer program is executed by a processor, it implements the calibration method 1000 of the roadside camera of one or more embodiments of the present application. Another example is that when a computer-readable storage medium (such as a USB flash drive) storing the computer program is connected to a computer, running the computer program can execute the calibration method 1000 of the roadside camera of one or more embodiments of the present application.

[0043] Refer to Figure 2 , which shows a schematic structural diagram of a calibration device 2000 of a roadside camera according to an embodiment of the present application. As Figure 2As shown in the figure, the calibration device 2000 for roadside cameras includes a receiving device 210, a matching device 220, and a calculating device 230. Among them, the receiving device 210 is used to receive the position information of multiple markers on the vehicle and the corresponding timestamps; the matching device 220 is used to match the images obtained by the roadside camera with the position information of the multiple markers based on the timestamps; and the calculating device 230 is used to calculate the parameters of the roadside camera based on the matched images and position information.

[0044] The receiving device 210 is configured to receive the position information of multiple markers on the vehicle and the corresponding timestamps. In one or more embodiments, the multiple markers can be pre-installed on the roof or body of the vehicle so that the roadside camera can capture these markers. In one embodiment, the number of markers is 3 or more. Those skilled in the art can understand that the more the number of markers, the higher the accuracy of subsequent calibration of the roadside camera.

[0045] In one embodiment, the position information of the multiple markers and the corresponding timestamps are obtained through a positioning device (such as an RTK device). For example, the receiving device 210 is configured to: receive the position information of the multiple markers and the corresponding timestamps from the positioning device installed on the vehicle, where the positions of the multiple markers are fixed relative to the positioning device. Taking the RTK device as an example, the RTK device can obtain the precise position information of the RTK device through real-time kinematic carrier phase differential technology, and will assign very precise timestamps to the recorded position information, etc. In this way, based on the available precise position information of the RTK device (GPS antenna), the precise position information of each marker can be easily calculated.

[0046] The matching device 220 matches the images obtained by the roadside camera with the position information of the multiple markers based on the timestamps. In one embodiment, the matching device 220 is configured to: obtain an image containing multiple markers (such as 3 or more colored markers) on the vehicle; calculate the coordinate information of the multiple markers in the pixel coordinate system, and the coordinate information carries timestamps (for example, the camera can perform precise time synchronization on its own system through the GPS system time service, so that each frame of image recorded by the camera can be assigned an accurate timestamp); find the position information of the multiple markers with the same timestamp; and match the position information of the multiple markers with the same timestamp with the coordinate information of the multiple markers in the pixel coordinate system. For example, each frame of the image output by the camera can be used by the matching device 220 to find the marker position information corresponding to the corresponding time through this timestamp.

[0047] In one or more embodiments, the coordinate information of the received multiple markers is the coordinate information (x, y, z) in the world coordinate system, while the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and the computing device 230 is configured to: determine the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

[0048] In one embodiment, the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system satisfy the following conversion relationship:

[0049]

[0050] In the above formula, Zc is a scale factor (not zero), representing the effective focal length (the distance from the optical center to the image plane), which is the homogeneous coordinate of the image point in the image coordinate system. f x and f y are respectively called the normalized focal lengths on the x-axis and y-axis, f x = f / dx, f y = f / dy, where f is the focal length of the camera, in mm, dx and dy are the pixel sizes, u 0 and v 0 are the image centers. R is the rotation matrix, and T is the translation matrix.

[0051] Thus, according to the above conversion relationship, the computing device 230 can determine the parameters of the roadside camera. In one embodiment, the parameters of the roadside camera are external parameters, and the external parameters include the translation matrix T and the rotation matrix R.

[0052] The above calibration device 2000 for the roadside camera can be integrated in the roadside camera system in one embodiment. In one embodiment, referring Figure 3 , the roadside camera system 3000 may include: a roadside camera 310 and a roadside unit RSU 320 that communicates with the roadside camera 310. The roadside unit RSU 320 may include a calibration device according to one or more embodiments of the present application (for example, Figure 2 the calibration device 2000 shown).

[0053] Figure 4 shows a schematic diagram of the scenario of online calibration of the roadside camera 410 according to an embodiment of the present application. As Figure 4As shown, vehicle 422 is traveling in the first lane 420, while vehicle 424 is traveling in the second lane 430. Both vehicle 422 and vehicle 424 are within the field of view of the roadside camera 410. In one or more embodiments, vehicle 422 and vehicle 424 may represent the same vehicle, i.e., the same vehicle travels in different lanes at two different times.

[0054] Vehicles 422 and 424 carry multiple markers (e.g., 3 or more markers). Figure 5 A schematic diagram of a vehicle 5000 with multiple markers according to an embodiment of the present application is shown. As Figure 5 shown, markers 511, 512, and 513 are installed on the vehicle body, and their relative positions with respect to the antenna 520 of the positioning device (installed inside vehicle 5000) are fixed.

[0055] In one embodiment, a positioning device, such as an RTK device, is installed inside the vehicle. RTK devices are generally relatively inexpensive and easy to install. When an RTK device is installed in a vehicle, the movement accuracy of the RTK is below 10 cm. As Figure 5 shown, when an RTK device is installed inside the vehicle, since the relationship between the RTK antenna and the three markers installed on the vehicle top is fixed, the precise positions of the individual markers can be deduced from the RTK position. The entire calibration process does not require blocking the traffic flow.

[0056] Continuing to refer to Figure 4 , in one embodiment, when a vehicle (such as vehicle 422 and vehicle 424) with a positioning device and multiple markers travels in a lane in front of a roadside camera, the vehicle can record its own position and the positions of the multiple markers (as well as the precise GPS time). At the same time, the roadside camera will capture a video of the vehicle with high-precision GPS time. The roadside camera system (such as the roadside unit RSU) can calculate the pixel positions of the markers in the video and attach precise timestamps. Through a time synchronization method, the positioning device measurement results can be matched with the roadside camera measurement results according to the timestamps. In some embodiments, the vehicle also needs to travel in some other difficult lanes to cover a wide enough camera projection plane (field of view). For example, several lines with real RTK positions on the camera projection plane can be generated. The roadside camera system can use the above information to calculate the parameters of the camera, and the entire calculation (calibration) process is online (rather than offline). In this way, the test or calibration personnel only need to drive the vehicle with multiple markers back and forth several times on the road surface within the field of view of the roadside camera to complete the entire calibration process (even in bad weather) using the calibration scheme of the roadside camera of the embodiment of the present application, and there is no need to block the traffic flow.

[0057] In summary, the calibration scheme for the roadside camera according to the embodiments of the present application receives the position information of multiple markers on a vehicle and the corresponding timestamps, matches the images acquired by the roadside camera with the position information of the multiple markers based on the timestamps, and finally calculates the parameters of the roadside camera based on the matched images and position information for online calibration. The above scheme completes online calibration by collecting the measurement data of multiple markers on the vehicle without blocking the traffic flow. Moreover, the entire calibration work does not require marking any special points on the ground or conducting outdoor data collection. If multiple roadside cameras are located on the same road, the calibration work for the multiple roadside cameras can be completed simultaneously by having the vehicle drive on the road multiple times.

[0058] The above examples mainly illustrate the calibration scheme for the roadside camera according to the embodiments of the present application. Although only some of the embodiments of the present application have been described, those of ordinary skill in the art should understand that the present application can be implemented in many other forms without departing from its gist and scope. Therefore, the examples and embodiments shown are regarded as illustrative rather than restrictive, and the present application may cover various modifications and substitutions without departing from the spirit and scope of the present application as defined by the various claims.

Claims

1. A calibration method for a roadside camera, characterized in that, the method includes: receiving the position information of multiple markers on a vehicle and corresponding timestamps; based on the timestamps, matching the images acquired by the roadside camera with the position information of the multiple markers; and calculating the parameters of the roadside camera based on the matched images and position information.

2. The method according to claim 1, wherein, receiving the position information of multiple markers on a vehicle and corresponding timestamps includes: receiving the position information of the multiple markers and corresponding timestamps from a positioning device installed on the vehicle, wherein the positions of the multiple markers are fixed relative to the positioning device.

3. The method according to claim 1, wherein, based on the timestamps, matching the images acquired by the roadside camera with the position information of the multiple markers includes: acquiring an image containing multiple markers on the vehicle; calculating the coordinate information of the multiple markers in the pixel coordinate system, and the coordinate information is with timestamps; finding the position information of the multiple markers with the same timestamp; and matching the position information of the multiple markers with the same timestamp with the coordinate information of the multiple markers in the pixel coordinate system.

4. The method according to claim 3, wherein, the received coordinate information of the multiple markers is the coordinate information (x, y, z) in the world coordinate system, and the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and based on the matched images and position information, calculating the parameters of the roadside camera includes: determining the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

5. The method according to claim 4, wherein, the parameters of the roadside camera are external parameters, and the external parameters include a translation matrix T and a rotation matrix R.

6. A calibration device for a roadside camera, characterized in that, the device includes: a receiving device configured to receive the position information of multiple markers on a vehicle and corresponding timestamps; a matching device configured to match the images acquired by the roadside camera with the position information of the multiple markers based on the timestamps; and a calculating device configured to calculate the parameters of the roadside camera based on the matched images and position information.

7. The device according to claim 6, wherein, the receiving device is configured to: receive the position information of the multiple markers and corresponding timestamps from a positioning device installed on the vehicle, wherein the positions of the multiple markers are fixed relative to the positioning device.

8. The device according to claim 6, wherein, the matching device is configured to: acquire an image containing multiple markers on the vehicle; calculate the coordinate information of the multiple markers in the pixel coordinate system, and the coordinate information is with timestamps; find the position information of the multiple markers with the same timestamp; and match the position information of the multiple markers with the same timestamp with the coordinate information of the multiple markers in the pixel coordinate system.

9. The device according to claim 8, wherein, the coordinate information of the received multiple markers is the coordinate information (x, y, z) in the world coordinate system, and the coordinate information of the multiple markers in the pixel coordinate system is (u, v), and the computing device is configured to: determine the parameters of the roadside camera according to the conversion relationship between the coordinate information (x, y, z) in the world coordinate system and the coordinate information (u, v) in the pixel coordinate system.

10. The device according to claim 9, wherein, the parameters of the roadside camera are external parameters, and the external parameters include a translation matrix T and a rotation matrix R.

11. A roadside camera system, characterized in that, the roadside camera system includes a roadside camera and a roadside unit RSU communicating with the roadside camera, wherein the roadside unit RSU includes the device according to any one of claims 6 to 10.

12. A computer-readable storage medium, characterized in that, the medium includes instructions that, when running, execute the method according to any one of claims 1 to 5.

13. A computer program product, including a computer program, characterized in that, the computer program, when executed by a processor, implements the method according to any one of claims 1 to 5.

14. A vehicle, characterized in that, the vehicle includes an in-vehicle unit configured to communicate with the device according to any one of claims 6 to 10, so that the roadside camera can be calibrated online, wherein a plurality of markers are installed on the roof and / or the body of the vehicle and are fixed in the relative position with a positioning device installed on the vehicle.

Images