A roadside camera calibration method and device, and a target detection method and device

Abstract

The application discloses a roadside camera calibration method and device and a target detection method and device. The calibration method comprises the following steps: acquiring calibration images shot by a roadside camera under different preset light environments, wherein the preset light environments at least include a strong light environment and a weak light environment; selecting a plurality of feature points in the calibration images, acquiring pixel coordinates and world coordinates of the plurality of feature points; and acquiring a calibration model corresponding to the roadside camera under each preset light environment according to the pixel coordinates and the world coordinates of the plurality of feature points. The technical scheme of the application can overcome the influence of the light environment on the calibration accuracy to some extent, and improve the calibration accuracy of the roadside camera.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a roadside camera calibration method and device, and a target detection method and device. Background Technology

[0002] In Intelligent Transportation Systems (ITS), roadside cameras are deployed at various intersections in urban traffic for real-time monitoring. To monitor and analyze traffic conditions at intersections, such as vehicle speed and queue length, roadside cameras need to be calibrated. Furthermore, with the development of computer vision technology, new services based on roadside cameras are emerging rapidly, such as the reconstruction of static elements at digital intersections based on high-precision maps and the acquisition of dynamic vehicle trajectories in vehicle-road cooperative systems. These new services place higher demands on the calibration accuracy of roadside cameras.

[0003] In existing technologies, camera intrinsic and extrinsic parameters are typically calibrated. After calibration, target detection is performed based on these fixed parameters. Intrinsic parameters include the camera's focal length. In practical applications, the camera's focal length is affected by lighting conditions. When lighting conditions change, the camera automatically refocuses. In this situation, if target detection is performed based on the calibrated intrinsic and extrinsic parameters, the pixel positions of the target object will shift, leading to inaccurate detection results. Summary of the Invention

[0004] Based on the aforementioned problems in the prior art, this application provides a roadside camera calibration method and apparatus, and a target detection method and apparatus, to overcome to some extent the influence of lighting environment on the calibration accuracy of roadside cameras and improve the calibration accuracy of roadside cameras.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a roadside camera calibration method, the method comprising:

[0007] Acquire calibration images captured by roadside cameras under different preset lighting conditions, wherein the preset lighting conditions include at least strong lighting conditions and weak lighting conditions;

[0008] Multiple feature points are selected in the calibration image, and the pixel coordinates and world coordinates of the multiple feature points are obtained;

[0009] The calibration model of the roadside camera under each preset lighting environment is obtained based on the pixel coordinates and world coordinates of the multiple feature points.

[0010] Optionally, acquiring calibration images captured by the roadside camera under different preset lighting conditions includes:

[0011] Obtain the current ambient light intensity information;

[0012] Determine whether the current environment belongs to the preset lighting environment based on the current ambient illuminance information;

[0013] If the current environment is a preset lighting environment, the calibration image under the preset lighting environment is obtained by taking pictures of the road traffic signs by the roadside camera.

[0014] Optionally, the method further includes:

[0015] The calibration image is divided into regions to obtain the distant and near-field regions;

[0016] The step of selecting multiple feature points in the calibration image and obtaining the pixel coordinates and world coordinates of the multiple feature points includes:

[0017] Multiple first feature points are selected in the distant region of the calibration image captured under each preset lighting condition, and the pixel coordinates and world coordinates of the multiple first feature points are obtained; and multiple second feature points are selected in the near region of the calibration image captured under any preset lighting condition, and the pixel coordinates and world coordinates of the multiple second feature points are obtained.

[0018] The step of obtaining the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points includes:

[0019] The distant calibration parameters of the roadside camera under each preset lighting environment are obtained based on the pixel coordinates and world coordinates of the plurality of first feature points, and the close-range calibration parameters of the roadside camera are obtained based on the pixel coordinates and world coordinates of the plurality of second feature points.

[0020] Based on the close-up calibration parameters of the roadside camera and the distant calibration parameters of the roadside camera under each preset lighting condition, obtain the calibration model corresponding to the roadside camera under each preset lighting condition.

[0021] Optionally, the roadside camera captures images of road signs under each preset lighting condition, wherein the road signs are at a preset distance from the roadside camera, and the process of dividing the calibration image into distant and near-field regions includes:

[0022] Determine the pixel coordinates of the road markings in the calibration image;

[0023] The calibration image is divided into two regions based on the pixel coordinates of the road markings. The region corresponding to the shooting range between the road markings and the roadside camera is designated as the foreground region, and the other region is designated as the background region.

[0024] Optionally, the plurality of feature points is N feature points, and the step of obtaining the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the plurality of feature points includes:

[0025] Select K feature points from the N feature points to obtain C. N K There are 10 different combinations of feature points, where K is a positive integer less than N;

[0026] Perform perspective transformation on each combination of feature points to obtain the perspective transformation matrix corresponding to each combination of feature points.

[0027] Obtain the total projection error corresponding to each perspective transformation matrix, and determine the perspective transformation matrix with the smallest total projection error as the calibration model corresponding to the roadside camera under each preset lighting environment.

[0028] Optionally, the world coordinates of the plurality of feature points can be obtained through the following steps:

[0029] Obtain local high-precision maps based on the location information of roadside cameras;

[0030] Based on the received feature point selection instruction, the world coordinates of N feature points are obtained from the local high-precision map.

[0031] Secondly, embodiments of this application also provide a target detection method, the method comprising:

[0032] The target image captured by the roadside camera and the current lighting environment are acquired. The roadside camera is calibrated using the roadside camera calibration method described in the above embodiments.

[0033] The calibration model of the roadside camera is determined based on the current lighting environment;

[0034] The location information of the target object in the target image is detected based on the calibration model of the roadside camera.

[0035] Optionally, the current lighting environment can be obtained through the following steps:

[0036] Obtain the current ambient light intensity information;

[0037] The process of determining the calibration model of the roadside camera based on the current lighting environment includes:

[0038] Based on the current ambient light intensity information, determine the preset lighting environment to which the current environment belongs, and based on the preset lighting environment to which the current environment belongs, determine the calibration model of the roadside camera.

[0039] Thirdly, embodiments of this application also provide a roadside camera calibration device, the device comprising:

[0040] The calibration image acquisition unit is used to acquire calibration images captured by the roadside camera under different preset lighting conditions, wherein the preset lighting conditions include at least a strong lighting environment and a weak lighting environment.

[0041] A feature point processing unit is used to select multiple feature points in the calibration image and obtain the pixel coordinates and world coordinates of the multiple feature points;

[0042] The camera calibration unit is used to obtain the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points.

[0043] Fourthly, embodiments of this application also provide a target detection device, the device comprising:

[0044] The acquisition unit is used to acquire the target image captured by the roadside camera and the current lighting environment. The roadside camera is calibrated using the roadside camera calibration method in the above embodiment.

[0045] The model determination unit is used to determine the calibration model of the roadside camera based on the current lighting environment.

[0046] The target detection unit is used to detect the position information of the target object in the target image based on the calibration model of the roadside camera.

[0047] Fifthly, embodiments of this application also provide an electronic device, including:

[0048] Processor; and

[0049] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform a roadside camera calibration method or a target detection method.

[0050] Sixthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform a roadside camera calibration method or a target detection method.

[0051] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: In the camera calibration stage, this application embodiment acquires calibration images of the roadside camera under different preset lighting conditions, and obtains multiple feature points and their pixel coordinates and world coordinates from the calibration images. Then, based on the pixel coordinates and world coordinates of the multiple feature points, the calibration model corresponding to the roadside camera under each preset lighting condition is calculated. This can reduce the influence of the lighting environment on the calibration accuracy to a certain extent and improve the calibration accuracy. Furthermore, this application embodiment calculates the calibration model based on the pixel coordinates and world coordinates of the feature points. The calculation process does not require the use of performance parameters such as the distortion coefficient and focal length of the roadside camera, thus avoiding the influence of performance parameters on the calibration accuracy.

[0052] Furthermore, in the target detection stage of this application embodiment, when acquiring the target image captured by the roadside camera, the current lighting environment is also acquired, and the current calibration model of the roadside camera is determined based on the current lighting environment. The position information of the target object in the target image is then detected based on the determined calibration model, so as to avoid the position detection result of the target object being affected by the lighting environment and improve the detection accuracy. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0054] Figure 1 This is a flowchart illustrating a roadside camera calibration method in an embodiment of this application;

[0055] Figure 2 This is a schematic diagram of a calibration image shown in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of a visualized point cloud of a local high-precision map shown in an embodiment of this application;

[0057] Figure 4 This is a flowchart illustrating a target detection method in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram of the structure of a roadside camera calibration device shown in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of a target detection device shown in an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0063] The execution entity of the roadside camera calibration method provided in this application embodiment can be a roadside device (e.g., a roadside camera or a roadside computing device), a server, or a cloud control platform; alternatively, the execution entity of the method in this application embodiment can be software or hardware. Please refer to... Figure 1 , Figure 1 Taking roadside equipment as an example, this application provides a method for calibrating a roadside camera. Figure 1 As shown, the roadside camera calibration method provided in this application embodiment may include the following steps S110 and S130:

[0064] Step S110: Obtain calibration images captured by the roadside camera under different preset lighting conditions, wherein the preset lighting conditions include at least strong lighting conditions and weak lighting conditions.

[0065] As mentioned earlier, lighting conditions affect camera focal length. When changes in lighting conditions cause changes in camera focal length, the pixel positions of the same static object will shift. Since the physical distances of pixels in distant areas of an image are much greater than those in close-up areas—for example, when a distant object is approximately 150 meters from the roadside camera, the physical distance of one pixel of that distant object is approximately 1.2 meters, while when a close-up object is approximately 10 meters from the roadside camera, the physical distance of one pixel of that close-up object is several centimeters—the degree of pixel shift is related to the relative distance between the object and the roadside camera. The greater the relative distance between the object and the camera, the greater the pixel position shift and the greater the influence of lighting conditions. Conversely, the closer the object is to the camera, the smaller the pixel position shift and the smaller the influence of lighting conditions.

[0066] Based on this, this application utilizes the roadside camera to acquire calibration images under different preset lighting conditions after the roadside camera is mounted on a roadside pole. Research has found that under extreme lighting conditions, namely strong light and weak light, the pixel position shifts by approximately 3 to 4 pixels. It is evident that these two extreme lighting conditions cause a slight change in the camera's focal length, with a small number of pixels shifting. Therefore, presetting these two lighting conditions can, to some extent, overcome the impact of camera focal length changes on calibration accuracy. Of course, those skilled in the art can preset more levels of lighting conditions to further improve calibration accuracy.

[0067] It should be noted that the roadside camera in this application can be a bullet camera, and the acquired calibration image is a bullet-view image. A bullet-view image refers to an image that can be equivalently acquired by a bullet camera along a fixed directional angle, such as... Figure 2 As shown, the gun-view image can have the characteristic of near objects appearing larger and far objects smaller, but there is no obvious barrel distortion. The lane lines in the gun-view image are relatively straight and there is no obvious bending deformation, which makes it easy to obtain the world coordinates corresponding to the pixel coordinates of the feature points in the gun-view image.

[0068] Step S120: Select multiple feature points in the calibration image and obtain the pixel coordinates and world coordinates of the multiple feature points.

[0069] Pixel coordinates are the coordinates of the pixel where the feature point is located in the image. Pixel coordinates are two-dimensional. World coordinates in this application are the three-dimensional coordinates of each point in a geographic region. It can be understood that in the physical world, the same point can have different coordinate values ​​under different coordinate systems.

[0070] In this application, the world coordinates of a feature point can be coordinate values ​​in any coordinate system. For example, the world coordinates of a feature point can be three-dimensional coordinates composed of the longitude, latitude, and altitude corresponding to the feature point; it can also be three-dimensional coordinates composed of the X, Y, and Z coordinates in the natural coordinate system corresponding to the feature point; or it can be other forms of coordinates, as long as the coordinates can uniquely determine the location of the feature point in the geographic area. This application does not limit the specific form of coordinates used. Optionally, the world coordinates used in this application are world coordinates provided by high-precision map point cloud data, such as WGS84 coordinates.

[0071] Step S130: Obtain the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points.

[0072] like Figure 1As shown in the roadside camera calibration method, this embodiment acquires calibration images of the roadside camera under different preset lighting conditions, and obtains multiple feature points, pixel coordinates, and world coordinates of these feature points from the calibration images. Then, based on the pixel coordinates and world coordinates of the feature points, a calibration model corresponding to the roadside camera under each preset lighting condition is calculated. This can, to a certain extent, reduce the impact of the lighting environment on calibration accuracy and improve calibration accuracy. Furthermore, this embodiment calculates the calibration model based on the pixel coordinates and world coordinates of the feature points; the calculation process does not require the use of performance parameters such as the distortion coefficients and focal length of the roadside camera, thus avoiding the influence of performance parameters on calibration accuracy.

[0073] In some embodiments of this application, acquiring calibration images captured by roadside cameras under different preset lighting conditions includes:

[0074] Obtain the current ambient light intensity information;

[0075] Determine whether the current environment belongs to the preset lighting environment based on the current ambient illuminance information;

[0076] If the current environment is a preset lighting environment, the calibration image under the preset lighting environment is obtained by taking pictures of the road traffic signs by the roadside camera.

[0077] It should be noted that since changes in the lighting environment have a relatively weak impact on the camera's focal length, current focal length strategies cannot measure the amount of focal length change affected by the lighting environment. Therefore, this embodiment quantifies the preset lighting environment using illuminance. For example, the illuminance range corresponding to a strong lighting environment is set to [lx1, lx2), and the illuminance range corresponding to a weak lighting environment is set to [lx2, lx3). Thus, by obtaining the illuminance information of the current environment, it can be determined whether the current environment belongs to the preset lighting environment. In this embodiment, the illuminance information of the current environment can be obtained based on time period information and climate information, including seasonal information and weather information. For example, the time period from 10:00 AM to 3:00 PM during sunny summer weather is considered a strong lighting environment, the time period from 11:00 AM to 1:00 PM during cloudy weather is considered a strong lighting environment, and cloudy weather is considered a weak lighting environment, etc. Alternatively, this embodiment can also obtain the illuminance information of the current environment using an illuminance measurement device. For example, when roadside equipment integrates illuminance measurement devices, illuminance information can be obtained through the integrated illuminance measurement devices, and the roadside equipment can also obtain the current environmental illuminance information through the Internet.

[0078] It should be noted that in practical applications, multiple frames of images can be acquired under each preset lighting condition, and then filtered based on image clarity and the occlusion of road traffic signs within the images. Preferably, images with high clarity and no occlusion of road traffic signs are selected as calibration images. Selecting suitable calibration images through preliminary image filtering helps improve the accuracy of feature point pixel coordinates, thereby improving the calibration accuracy of the roadside camera.

[0079] As mentioned earlier, the degree of pixel offset is related to the relative distance between the target object and the roadside camera. The farther the relative distance between the target object and the roadside camera, the greater the degree of pixel offset and the greater the influence of the lighting environment. Conversely, the closer the relative distance between the target object and the roadside camera, the smaller the degree of pixel offset and the less the influence of the lighting environment.

[0080] It is known that the lighting environment has a significant impact on the pixel offset of distant objects, while its impact on the pixel offset of near objects is negligible. Based on this, some embodiments of this application can perform partitioning of the calibration image. Under different preset lighting conditions, the near-field region can use the same calibration parameters, while the distant region needs to use separate calibration parameters, thereby improving calibration efficiency.

[0081] Specifically, in some embodiments of this application, the above... Figure 1 The roadside camera calibration method also includes:

[0082] The calibrated image is divided into regions to obtain a distant region and a near region;

[0083] Multiple first feature points are selected in the distant region of the calibration image captured under each preset lighting condition, and the pixel coordinates and world coordinates of the multiple first feature points are obtained; and multiple second feature points are selected in the near region of the calibration image captured under any preset lighting condition, and the pixel coordinates and world coordinates of the multiple second feature points are obtained; that is, multiple first feature points are selected in the distant region of the calibration image captured under strong light and weak light conditions respectively, and multiple second feature points are selected in the near region of the calibration image captured under strong light or weak light conditions.

[0084] The distant calibration parameters of the roadside camera under each preset lighting environment are obtained based on the pixel coordinates and world coordinates of the plurality of first feature points, and the close-range calibration parameters of the roadside camera are obtained based on the pixel coordinates and world coordinates of the plurality of second feature points.

[0085] Based on the close-range calibration parameters of the roadside camera and the distant calibration parameters of the roadside camera under each preset lighting condition, a calibration model corresponding to the roadside camera under each preset lighting condition is obtained. That is, the close-range calibration parameters and the distant calibration parameters under each preset lighting condition are combined to form a calibration model for that preset lighting condition, so that close-range targets share a common set of calibration parameters, while distant targets use corresponding calibration parameters based on lighting conditions.

[0086] It should be noted that the calculation methods for the far-view calibration parameters and the near-view calibration parameters in this embodiment can refer to the method for obtaining the calibration model based on the projection matrix P introduced in subsequent embodiments, and will not be repeated here.

[0087] The calibration image partitioning method of this embodiment can be implemented in various ways. If the roadside camera captures the road sign under each preset lighting condition when taking calibration images, the road sign needs to be placed in a designated position beforehand, i.e., the road sign and the roadside camera have a preset distance. When the distance between the target object and the roadside camera is greater than the preset distance between the road sign and the roadside camera, the lighting environment has a significant impact on the pixel offset of the target object. When the distance between the target object and the roadside camera is less than the preset distance between the road sign and the roadside camera, the impact of the lighting environment on the pixel offset of the target object can be ignored. The preset distance between the road sign and the roadside camera is related to factors such as the field of view, field of view range, and installation posture of the roadside camera. Optionally, the preset distance is 80 meters. Of course, those skilled in the art can also set the position of the road sign according to experiments.

[0088] The partitioning scheme for the calibrated image at this time is:

[0089] Determine the pixel coordinates of the road markings in the calibration image;

[0090] The calibration image is divided into two regions based on the pixel coordinates of the road markings;

[0091] The area corresponding to the shooting range between the road markings and the roadside camera is designated as the foreground area, and the other area is designated as the background area.

[0092] This embodiment requires manually setting road markings. The boundary between the foreground and background areas is determined by the pixel coordinates of the road markings in the calibration image, and the foreground and background areas are then divided based on this boundary. In other implementations, road markings may not be set up; the aforementioned boundary line in the calibration image can be determined using a high-precision map.

[0093] In some embodiments of this application, the calibration image includes road traffic signs, i.e., the calibration image is obtained by a roadside camera capturing road traffic signs under different preset lighting conditions. The calibration image may include multiple road traffic signs. Multiple road traffic signs are advantageous in two ways: firstly, they facilitate the selection of multiple non-collinear feature points, which improves the accuracy of the calibration model; secondly, they help determine a sufficient number of feature points to meet the feature point requirements for calculating the calibration model.

[0094] The road traffic signs in this embodiment are mainly used to control and guide traffic. They are usually composed of various lines marked on the road surface (stop lines, pedestrian crossings, lane dividing lines, etc., dotted or solid lines), arrow lines (turn left, turn right, turn left and go straight, turn right and go straight, etc.), text, delineators, etc.

[0095] In some embodiments of this application, N feature points are selected from the calibration image through the following steps:

[0096] Obtain the line segments included in the road traffic signs in the calibration image;

[0097] The center point of each line segment included in the road traffic sign is selected as the feature point.

[0098] The accuracy of the calibration model is related to the number of feature points. To improve the accuracy of the calibration model, the number of selected feature points can be increased appropriately, for example, N∈[15, 40]. Of course, without considering computational resources and efficiency, more feature points can also be selected.

[0099] Since this application obtains the world coordinates of feature points from point cloud data of high-precision maps, and point cloud data is composed of unordered clusters of points, feature points such as corners, intersections, and centers, which are easily selected in images, are not easily obtained in point cloud data of high-precision maps. However, the center points of line segments are easily selected in point cloud data of high-precision maps. Based on this, this embodiment divides road traffic signs in the calibration image into multiple line segments, and selects the center point of each line segment as a feature point. For example, in... Figure 2 In the lane dividing dashed lines shown, the center point of the short side of each line segment constituting the dashed line is selected as the feature point.

[0100] It should be noted that when the road traffic signs in the calibration image are solid lane lines, these lines can be divided into equal segments, and the center point of the shorter side of each segment can be selected as the feature point. Similarly, when the road traffic signs in the calibration image are turning arrows, the center points of the two hypotenuses forming the arrow can be used as feature points, and the solid turning line can be divided into equal segments, with the center point of the shorter side of each segment selected as the feature point. Equal-spacing division facilitates locating the corresponding position points in the point cloud data of the high-precision map.

[0101] After selecting N feature points in the calibration image, the pixel coordinates of the feature points can be obtained according to the received feature point selection instructions. For example, the calibration image can be opened using a drawing tool or pixel extraction software, and feature point selection instructions can be generated based on the user's touch or click operations, enabling the user to independently select feature points, improving the flexibility of feature point selection. Moreover, user-selected feature points can reduce the computational load on the computer and improve the calibration efficiency of roadside cameras.

[0102] During their research, the inventors discovered that the selection of feature points has a significant impact on calibration accuracy. To improve the calibration accuracy of roadside cameras, it is best that feature points are not all located on the same straight line, as this results in the loss of some dimensional coordinate distortion information, reducing the accuracy of the calibration model. In the preferred embodiment of this application, since the calibration image is a bullet-view image, the distortion in the central region of the image is relatively small. Therefore, feature points are preferably selected within the central region of the calibration image. Thus, the feature points are preferably located in the central region of the calibration image and are distributed in a planar pattern.

[0103] In some embodiments of this application, the world coordinates of N feature points are obtained through the following steps:

[0104] Obtain local high-precision maps based on the location information of roadside cameras;

[0105] Based on the received feature point selection instruction, the world coordinates of N feature points are obtained from the local high-precision map.

[0106] The high-definition map (HD map) used in this embodiment can be derived from high-definition maps designed for vehicle driving, especially those designed for autonomous vehicles. The HD map can contain elements for vehicle navigation, such as road information, intersection information, traffic signal information, and lane rule information.

[0107] By importing point cloud data into visualization software, such as QGIS, one can obtain... Figure 3 The visualized high-precision map shown is, in such Figure 3 In the high-definition map shown, when the high-definition map is zoomed in to a certain extent, Figure 3 The lines, arrows, and other elements shown are represented as a point cloud with a disordered arrangement.

[0108] When obtaining the world coordinates of feature points from a high-precision map, this embodiment can obtain a local high-precision map based on the location information of the roadside camera, and generate feature point selection instructions based on the user's touch or click operations, thereby reducing the amount of calculation and improving the efficiency of camera calibration.

[0109] In this embodiment, when obtaining the world coordinates of N feature points according to the feature point selection instruction, it is determined whether there is a point cloud on the local high-precision map corresponding to the feature point selection instruction; if there is, the world coordinates of the feature point are obtained according to the position information of the point cloud; if there is no point cloud, the feature point cloud corresponding to the feature point selection instruction is obtained, and the world coordinates of the feature point are obtained according to the position information of the feature point cloud.

[0110] The system generates a feature point selection command based on the user's touch or click operation on the center position of a point cloud segment. The visualized high-precision map is then zoomed in to see if a point cloud exists at the center position. If a point cloud exists, the world coordinates of the feature point can be determined based on the latitude and longitude information of the point cloud. If no point cloud exists, the latitude and longitude information of the point cloud near the center position can be interpolated to determine the world coordinates of the feature point.

[0111] In practical applications, when the pixel coordinates and corresponding world coordinates of N feature points in each calibration image are obtained, the pixel coordinates can be saved as a first file using the index of the N feature points. Similarly, the world coordinates of the N feature points can be saved as a second file. The indexes of the corresponding feature points in the first and second files are integrated so that the pixel coordinates and world coordinates in the first and second files correspond one-to-one with the feature point indexes, thus obtaining N feature point pairs.

[0112] Next, select K feature points from the N feature points to obtain... There are 3 different combinations of feature points, where K is a positive integer less than N and greater than 3;

[0113] Perform perspective transformation on each combination of feature points to obtain the perspective transformation matrix corresponding to each combination of feature points.

[0114] Obtain the total projection error corresponding to each perspective transformation matrix, and determine the perspective transformation matrix with the smallest total projection error as the calibration model corresponding to the roadside camera under each preset lighting environment.

[0115] In some possible implementations of this embodiment, obtaining the total projection error corresponding to each perspective transformation matrix includes:

[0116] The K-point perspective imaging algorithm is used to perform perspective transformation on each combination of feature points to obtain the perspective transformation matrix corresponding to each combination of feature points.

[0117] Obtain the remaining feature points corresponding to each combination of feature points;

[0118] Based on the perspective transformation matrix corresponding to each feature point combination and the pixel coordinates of each remaining feature point, obtain the calculated world coordinate value of each remaining feature point;

[0119] The total projection error corresponding to each perspective transformation matrix is ​​obtained by summing the world coordinates of all remaining feature points and their differences.

[0120] For example, by selecting any four feature point pairs from the above N feature point pairs, we can obtain For each type of point pair combination:

[0121] Calculate the corresponding 3×3 perspective transformation matrix P respectively, and then we can calculate... A perspective transformation matrix P is calculated; then, the world coordinates of the remaining N-4 feature points are calculated using the calculated perspective transformation matrix P, specifically according to the formula longtitude_latitude=P×pixel. T The homogeneous values ​​of the world coordinates are calculated and then normalized. For example, this is done by simultaneously dividing the 3×1 latitude and longitude coordinates by the third dimension, resulting in the final calculated world coordinate values. Here, pixel = [u, v, 1]. T Let longtitude_latitude be the homogeneous form of pixel coordinates, where longtitude_latitude = [x, y, z]. T Let the world coordinate system be used. Subtract the calculated world coordinates of the remaining N-4 feature points from their world coordinates to obtain the projection errors of the remaining N-4 feature points. Add them together to obtain the total projection error. Use the perspective transformation matrix P with the smallest error value as the calibration model.

[0122] The calibration method described above in the embodiments of this application can be used to calibrate roadside cameras, thereby increasing the calibration range from 65-200 meters to 65-450 meters. The calibration accuracy is improved to about 1 meter within the range of 65-220 meters, to within 1.5 meters within the range of 220-280 meters, and to within 3 meters within the range of 280-450 meters, which can meet the calibration accuracy requirements of intelligent transportation systems.

[0123] This application also provides a target detection method. The target detection method provided in this application can be executed by a roadside device (e.g., a roadside camera or roadside computing device), a server, or a cloud control platform; alternatively, the method can be executed by software or hardware. Please refer to... Figure 4 , Figure 4 Taking roadside equipment as an example, this application provides a target detection method according to its embodiments. Figure 4 As shown, an embodiment of this application provides a target detection method that may include the following steps S410 and S430:

[0124] S410 acquires target images captured by roadside cameras and the current lighting conditions.

[0125] The roadside camera in this embodiment is calibrated using the roadside camera calibration method described in the above embodiments, and the calibration process will not be repeated here.

[0126] When the roadside camera captures the target image, it also acquires the current lighting environment. If the current lighting environment is a strong lighting environment, the calibration model corresponding to the strong lighting environment is used as the current calibration model of the roadside camera; if the current lighting environment is a weak lighting environment, the calibration model corresponding to the weak lighting environment is used as the current calibration model of the roadside camera.

[0127] S420, determine the calibration model of the roadside camera based on the current lighting environment.

[0128] S430, Detect the position information of the target object in the target image according to the calibration model of the roadside camera.

[0129] In this embodiment, when acquiring the target image captured by the roadside camera, the current lighting environment is also acquired. Based on the current lighting environment, the current calibration model of the roadside camera is determined. The position information of the target object in the target image is then detected based on the determined calibration model, so as to avoid the position detection result of the target object being affected by the lighting environment and improve the detection accuracy.

[0130] In some embodiments of this application, the current lighting environment is obtained through the following steps:

[0131] The illuminance information of the current environment is obtained. The method for obtaining the illuminance information of the current environment can be referred to the description of the relevant embodiments above, and will not be repeated here.

[0132] Accordingly, the calibration model of the roadside camera is determined based on the current lighting environment, including:

[0133] Based on the current ambient light intensity information, determine the preset lighting environment to which the current environment belongs, and based on the preset lighting environment to which the current environment belongs, determine the calibration model of the roadside camera.

[0134] This application also provides a roadside camera calibration device 500, such as... Figure 5 The diagram shows a schematic representation of a roadside camera calibration device according to an embodiment of this application. The device 500 is applied to roadside equipment and includes: a calibration image acquisition unit 510, a feature point processing unit 520, and a camera calibration unit 530, wherein:

[0135] The calibration image acquisition unit 510 is used to acquire calibration images captured by the roadside camera under different preset lighting conditions, wherein the preset lighting conditions include at least a strong lighting environment and a weak lighting environment.

[0136] The feature point processing unit 520 is used to select multiple feature points in the calibration image and obtain the pixel coordinates and world coordinates of the multiple feature points.

[0137] The camera calibration unit 530 is used to obtain the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points.

[0138] In one embodiment of this application, the calibration image acquisition unit 510 is used to acquire the illuminance information of the current environment; determine whether the current environment belongs to a preset lighting environment based on the illuminance information of the current environment; if the current environment belongs to the preset lighting environment, then acquire the calibration image of the road traffic signs under the preset lighting environment by taking pictures of them with the roadside camera.

[0139] In one embodiment of this application, the device 500 further includes a region division unit;

[0140] A region division unit is used to divide the calibration image into regions to obtain a distant region and a near region.

[0141] The feature point processing unit 520 is used to select multiple first feature points in the distant area of ​​a calibration image captured under each preset lighting condition, and obtain the pixel coordinates and world coordinates of the multiple first feature points; and to select multiple second feature points in the near area of ​​a calibration image captured under any preset lighting condition, and obtain the pixel coordinates and world coordinates of the multiple second feature points.

[0142] The camera calibration unit 530 is used to obtain the distant calibration parameters of the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the plurality of first feature points; to obtain the close-range calibration parameters of the roadside camera based on the pixel coordinates and world coordinates of the plurality of second feature points; and to obtain the calibration model corresponding to the roadside camera under each preset lighting environment based on the close-range calibration parameters of the roadside camera and the distant calibration parameters of the roadside camera under each preset lighting environment.

[0143] In one embodiment of this application, the roadside camera captures images of road signs under each preset lighting condition. The road signs are at a preset distance from the roadside camera. A region division unit is specifically used to determine the pixel coordinates of the road signs in the calibration image. Based on the pixel coordinates of the road signs, the calibration image is divided into two regions: the region corresponding to the shooting range between the road signs and the roadside camera is designated as the near-field region, and the other region is designated as the far-field region.

[0144] In one embodiment of this application, the plurality of feature points are N feature points, and the camera calibration unit 530 is used to select K feature points from the N feature points to obtain Different combinations of feature points are used, where K is a positive integer less than N; perspective transformation is performed on each combination of feature points to obtain the perspective transformation matrix corresponding to each combination of feature points; the total projection error corresponding to each perspective transformation matrix is ​​obtained, and the perspective transformation matrix with the smallest total projection error is determined as the calibration model corresponding to the roadside camera under each preset lighting environment.

[0145] In one embodiment of this application, the feature point processing unit 520 is further configured to acquire a local high-precision map based on the location information of the roadside camera; and acquire the world coordinates of N feature points from the local high-precision map based on the received feature point selection instruction.

[0146] It is understood that the roadside camera calibration device described above can realize each step of the roadside camera calibration method provided in the foregoing embodiments. The relevant explanations of the roadside camera calibration method are applicable to the roadside camera calibration device, and will not be repeated here.

[0147] This application embodiment also provides a target detection device 600, such as Figure 6 The diagram shows a target detection device according to an embodiment of this application. The device 600 is applied to roadside equipment and includes: an acquisition unit 610, a model determination unit 620, and a target detection unit 630, wherein:

[0148] The acquisition unit 610 is used to acquire the target image captured by the roadside camera and the current lighting environment;

[0149] The roadside camera in this embodiment is calibrated using the roadside camera calibration method described in the above embodiments, and the calibration process will not be repeated here.

[0150] The model determination unit 620 is used to determine the calibration model of the roadside camera based on the current lighting environment;

[0151] The target detection unit 630 is used to detect the position information of the target object in the target image based on the calibration model of the roadside camera.

[0152] Optionally, the acquisition unit 610 is used to acquire the illuminance information of the current environment;

[0153] The model determination unit 620 is used to determine the preset lighting environment to which the current environment belongs based on the current environment's illuminance information, and to determine the calibration model of the roadside camera based on the preset lighting environment to which the current environment belongs.

[0154] It is understood that the target detection device described above can implement each step of the target detection method provided in the foregoing embodiments. The relevant explanations of the target detection method are applicable to the target detection device and will not be repeated here.

[0155] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0156] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0157] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0158] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a roadside camera calibration device or target detection device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0159] Acquire calibration images captured by roadside cameras under different preset lighting conditions, wherein the preset lighting conditions include at least strong lighting conditions and weak lighting conditions;

[0160] Multiple feature points are selected in the calibration image, and the pixel coordinates and world coordinates of the multiple feature points are obtained;

[0161] The calibration model of the roadside camera under each preset lighting environment is obtained based on the pixel coordinates and world coordinates of the multiple feature points.

[0162] Alternatively, the processor executes the program stored in memory and specifically performs the following operations:

[0163] Acquire target images captured by roadside cameras and the current lighting environment;

[0164] The calibration model of the roadside camera is determined based on the current lighting environment;

[0165] The location information of the target object in the target image is detected based on the calibration model of the roadside camera.

[0166] The above is as stated in this application. Figure 1 The method or embodiment disclosed herein is performed by the roadside camera calibration device. Figure 4 The method executed by the target detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads the information from the memory and, in conjunction with its hardware, completes the steps of the aforementioned roadside camera calibration method or target detection method.

[0167] The electronic device can also perform Figure 1The method executed by the roadside camera calibration device and the function of the roadside camera calibration device in the embodiment shown in 1 are described above. Alternatively, the electronic device may also perform... Figure 4 The method executed by the target detection device in the present application is to implement the function of the target detection device in the embodiment shown in 4. The embodiments of the present application will not be described again here.

[0168] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the roadside camera calibration device in the illustrated embodiment, or the method executed by... Figure 4 The method executed by the target detection device in the illustrated embodiment will not be described again in this application.

[0169] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0174] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0175] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for calibrating a roadside camera, characterized in that, The method includes: Acquire calibration images captured by roadside cameras under different preset lighting conditions, wherein the preset lighting conditions include at least strong lighting conditions and weak lighting conditions; Multiple feature points are selected in the calibration image, and the pixel coordinates and world coordinates of the multiple feature points are obtained; Based on the pixel coordinates and world coordinates of the multiple feature points, obtain the calibration model corresponding to the roadside camera under each preset lighting environment; The method further includes: The calibrated image is divided into regions to obtain a distant region and a near region; The step of selecting multiple feature points in the calibration image and obtaining the pixel coordinates and world coordinates of the multiple feature points includes: Multiple first feature points are selected in the distant region of the calibration image captured under each preset lighting condition, and the pixel coordinates and world coordinates of the multiple first feature points are obtained; and multiple second feature points are selected in the near region of the calibration image captured under any preset lighting condition, and the pixel coordinates and world coordinates of the multiple second feature points are obtained. The step of obtaining the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points includes: The distant calibration parameters of the roadside camera under each preset lighting environment are obtained based on the pixel coordinates and world coordinates of the plurality of first feature points, and the close-range calibration parameters of the roadside camera are obtained based on the pixel coordinates and world coordinates of the plurality of second feature points. Based on the close-up calibration parameters of the roadside camera and the distant calibration parameters of the roadside camera under each preset lighting condition, obtain the calibration model corresponding to the roadside camera under each preset lighting condition.

2. The roadside camera calibration method as described in claim 1, characterized in that, The acquisition of calibration images captured by roadside cameras under different preset lighting conditions includes: Obtain the current ambient light intensity information; Determine whether the current environment belongs to the preset lighting environment based on the current ambient illuminance information; If the current environment is a preset lighting environment, the calibration image under the preset lighting environment is obtained by taking pictures of the road traffic signs by the roadside camera.

3. The roadside camera calibration method as described in claim 1, characterized in that, The roadside camera captures images of road signs under various preset lighting conditions. The road signs are at a preset distance from the roadside camera. The calibration image is divided into regions to obtain a distant view region and a near view region, including: Determine the pixel coordinates of the road markings in the calibration image; The calibration image is divided into two regions based on the pixel coordinates of the road markings; The area corresponding to the shooting range between the road markings and the roadside camera is designated as the foreground area, and the other area is designated as the background area.

4. The roadside camera calibration method as described in claim 1, characterized in that, The plurality of feature points refers to N feature points. The step of obtaining the calibration model corresponding to the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the plurality of feature points includes: Select K feature points from the N feature points to obtain There are 10 different combinations of feature points, where K is a positive integer less than N; Perform perspective transformation on each combination of feature points to obtain the perspective transformation matrix corresponding to each combination of feature points. Obtain the total projection error corresponding to each perspective transformation matrix, and determine the perspective transformation matrix with the smallest total projection error as the calibration model corresponding to the roadside camera under each preset lighting environment.

5. The roadside camera calibration method as described in claim 1, characterized in that, The world coordinates of the multiple feature points are obtained through the following steps: Obtain local high-precision maps based on the location information of roadside cameras; Based on the received feature point selection instruction, the world coordinates of N feature points are obtained from the local high-precision map.

6. A target detection method, characterized in that, The method includes: Acquire the target image captured by the roadside camera and the current lighting environment, wherein the roadside camera is calibrated using the roadside camera calibration method as described in any one of claims 1 to 5; The calibration model of the roadside camera is determined based on the current lighting environment; The location information of the target object in the target image is detected based on the calibration model of the roadside camera.

7. The target detection method as described in claim 6, characterized in that, Obtain the current lighting environment by following these steps: Obtain the current ambient light intensity information; The process of determining the calibration model of the roadside camera based on the current lighting environment includes: Based on the current ambient light intensity information, determine the preset lighting environment to which the current environment belongs, and based on the preset lighting environment to which the current environment belongs, determine the calibration model of the roadside camera.

8. A roadside camera calibration device, characterized in that, The device includes: The calibration image acquisition unit is used to acquire calibration images captured by the roadside camera under different preset lighting conditions, wherein the preset lighting conditions include at least a strong lighting environment and a weak lighting environment. A feature point processing unit is used to select multiple feature points in the calibration image and obtain the pixel coordinates and world coordinates of the multiple feature points; The camera calibration unit is used to obtain the calibration model of the roadside camera under each preset lighting environment based on the pixel coordinates and world coordinates of the multiple feature points. The device further includes a region division unit, the region division unit being used for: The calibrated image is divided into regions to obtain a distant region and a near region; The feature point processing unit is specifically used for: Multiple first feature points are selected in the distant region of the calibration image captured under each preset lighting condition, and the pixel coordinates and world coordinates of the multiple first feature points are obtained; and multiple second feature points are selected in the near region of the calibration image captured under any preset lighting condition, and the pixel coordinates and world coordinates of the multiple second feature points are obtained. The camera calibration unit is specifically used for: The distant calibration parameters of the roadside camera under each preset lighting environment are obtained based on the pixel coordinates and world coordinates of the plurality of first feature points, and the close-range calibration parameters of the roadside camera are obtained based on the pixel coordinates and world coordinates of the plurality of second feature points. Based on the close-up calibration parameters of the roadside camera and the distant calibration parameters of the roadside camera under each preset lighting condition, obtain the calibration model corresponding to the roadside camera under each preset lighting condition.

9. A target detection device, characterized in that, The device includes: An acquisition unit is used to acquire the target image captured by the roadside camera and the current lighting environment, wherein the roadside camera is calibrated using the roadside camera calibration method as described in any one of claims 1 to 5. The model determination unit is used to determine the calibration model of the roadside camera based on the current lighting environment. The target detection unit is used to detect the position information of the target object in the target image based on the calibration model of the roadside camera.

10. An electronic device, characterized in that, include: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the roadside camera calibration method as described in any one of claims 1 to 5 or the target detection method as described in any one of claims 6 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform the roadside camera calibration method as described in any one of claims 1 to 5 or the target detection method as described in any one of claims 6 to 7.

Images