Camera calibration method and device based on Leiye matching and storage medium

Abstract

The invention discloses a camera calibration method and device based on Leiye matching and a storage medium, and the method comprises the steps: carrying out the radar detection and image recognition of a to-be-detected region, obtaining the radar data and image data of a target object in the to-be-detected region, and enabling the radar data to comprise the radar position and the image position; the image data comprises pixel positions; performing synchronous processing on the radar position and the pixel position to obtain a position matching pair; the position matching pair comprises a target radar position and a target pixel position of the target object; projecting the target radar position to a pixel coordinate system where the target pixel position is located according to the obtained initial external reference and a pre-calibrated camera internal reference, and obtaining a projection position of the radar position in the pixel coordinate system; and performing iterative adjustment on the initial external parameter according to the distance between the projection position and the target pixel position to obtain a target external parameter. According to the scheme, the camera calibration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of parameter calibration, and in particular to a camera calibration method, device and storage medium based on radar matching. Background Art

[0002] Camera intrinsic parameters and camera extrinsic parameters are both very important parameters of the camera. The three-dimensional coordinate points in the world coordinate system can be projected onto the camera image by relying on the camera intrinsic and extrinsic parameter matrices. Similarly, the pixel points on the image can be projected into the world coordinate system by relying on the camera intrinsic and extrinsic parameter matrices and pixel depth information to obtain the three-dimensional coordinates of the pixel points.

[0003] At present, camera intrinsic and extrinsic parameters have been widely used in various fields of computer vision, such as 3D visual reconstruction, 3D target detection, radar data fusion, etc. However, how to effectively determine the intrinsic and extrinsic parameters of a camera is a very important issue.

[0004] For camera intrinsic parameters, they can generally be calibrated by Zhang’s calibration method when the camera leaves the factory. For camera extrinsic parameters, since the calibration of camera extrinsic parameters depends on the selection of the world reference coordinate system, there are different calibration processes for different application scenarios of the camera. Specifically, the extrinsic parameter calibration can only be achieved after the camera is installed, which affects the camera calibration efficiency. Summary of the invention

[0005] The present application at least provides a camera calibration method, apparatus, device and computer-readable storage medium based on radar matching.

[0006] In a first aspect, the present application provides a camera calibration method based on radar-visual matching, comprising: performing radar detection and image recognition processing on an area to be detected respectively, obtaining radar data and image data of a target object in the area to be detected, wherein the radar data includes a radar position of the target object in the area to be detected, and the image data includes a pixel position of the target object in the area to be detected; synchronously processing the radar position and the pixel position to obtain a position matching pair; the position matching pair includes a target radar position and a target pixel position of the target object; projecting the target radar position to a pixel coordinate system where the target pixel position is located according to an acquired initial extrinsic parameter and a pre-calibrated camera intrinsic parameter, to obtain a projection position of the target radar position in the pixel coordinate system; iteratively adjusting the initial extrinsic parameter according to a distance between the projection position and the target pixel position, to obtain a target extrinsic parameter.

[0007] In one embodiment, performing radar detection and image recognition processing on the area to be detected respectively to obtain radar data and image data of the target object in the area to be detected includes: performing radar detection on the area to be detected to obtain the radar position of the target object; performing image recognition processing on the area to be detected to obtain the detection area of ​​the target object; and determining the pixel position of the target object based on the pixel information of the detection area.

[0008] In one embodiment, before performing radar detection and image recognition processing on the area to be detected respectively to obtain radar data and image data of the target object in the area to be detected, the method also includes: acquiring a regional image of the area to be detected; performing lane line detection on the regional image to obtain lane line information.

[0009] In one embodiment, after performing lane line detection on the area image to obtain the lane line information, the method further includes: in response to the lane line information representing that there are at least two lanes in the area to be detected, performing radar detection and image recognition processing on each lane respectively, and determining at least one target object from each lane.

[0010] In one embodiment, the synchronous processing of the radar position and the pixel position to obtain a position matching pair includes: in response to the consistency of frame rates between the radar detection and the image recognition processing, obtaining a radar timestamp of the radar data and an image timestamp of the image data; and time-aligning each radar position and each pixel position according to the radar timestamp and the image timestamp to obtain the position matching pair.

[0011] In one embodiment, the step of synchronously processing the radar position and the pixel position to obtain a position matching pair includes: In response to the frame rate inconsistency between the radar detection and the image recognition processing, the radar data and the image data are subjected to frame extraction processing to obtain target radar data and target image data; the time difference between the target radar data and the target image data is less than a preset time difference threshold; according to the radar timestamp of the target radar data and the image timestamp of the target image data, the target radar position of each target radar data and the target pixel position of each target image data are time-aligned to obtain the position matching pair.

[0012] In one embodiment, projecting the target radar position to the pixel coordinate system where the target pixel position is located according to the acquired initial extrinsic parameters and the pre-calibrated camera intrinsic parameters to obtain the projected position of the target radar position in the pixel coordinate system includes: determining an initial data pair earlier than the current time sequence from each position matching pair according to the timing information of each position matching pair, and determining the initial extrinsic parameters according to the initial data pair; performing homogenization processing on the target radar position in each position matching pair to obtain a homogenized radar position; projecting the homogenized radar position to the pixel coordinate system according to the initial extrinsic parameters and the camera intrinsic parameters to obtain a projected radar position; and performing dehomogenization processing on the projected radar position to obtain the projected position.

[0013] In one embodiment, the iterative adjustment of the initial extrinsic parameter according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter includes: iteratively adjusting the initial extrinsic parameter according to the Cartesian distance between the projection position and the pixel position through a preset optimization algorithm so that the Cartesian distance is less than or equal to a preset distance threshold or the number of iterative adjustments is greater than a preset number threshold, so as to obtain the target extrinsic parameter.

[0014] In a second aspect, the present application provides a camera calibration device based on radar-visual matching, comprising: a detection module, used to perform radar detection and image recognition processing on a to-be-detected area respectively, to obtain radar data and image data of a target object in the to-be-detected area, wherein the radar data includes the radar position of the target object in the to-be-detected area, and the image data includes the pixel position of the target object in the to-be-detected area; a synchronization module, used to synchronize the radar position and the pixel position to obtain a position matching pair; the position matching pair includes a target radar position and a target pixel position of the target object; a projection module, used to project the target radar position to a pixel coordinate system where the target pixel position is located according to an acquired initial external parameter and a pre-calibrated camera internal parameter, to obtain a projection position of the target radar position in the pixel coordinate system; an adjustment module, used to iteratively adjust the initial external parameter according to the distance between the projection position and the target pixel position, to obtain a target external parameter.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned camera calibration method based on radar matching.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the above-mentioned camera calibration method based on radar matching is implemented.

[0017] The above scheme can obtain radar data and image data of the target object in the area to be detected, including the radar position and pixel position of the target object, by performing radar detection and image recognition processing on the area to be detected respectively. The radar position and pixel position are processed synchronously to obtain a position matching pair, and the radar position and pixel position in the synchronized position matching pair are the target radar position and target pixel position of the target object. According to the acquired initial external parameters and the pre-calibrated camera internal parameters, the target radar position is projected to the pixel coordinate system where the target pixel position is located to obtain the projection position of the target radar position in the pixel coordinate system, but the uncalibrated initial external parameters may not be accurate enough to accurately project the target radar position. Therefore, according to the distance between the projection position and the target pixel position, the initial external parameters are iteratively adjusted to obtain the target external parameters. In this way, the calibration of the camera external parameters is achieved, and the camera calibration efficiency is improved.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of a camera calibration method based on radar matching of the present application; Figure 2 is a schematic diagram of an exemplary application scenario in the camera calibration method based on radar matching of the present application; Figure 3 is a block diagram of a camera calibration device based on radar matching shown in an exemplary embodiment of the present application; Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] For ease of understanding, one of the applicable scenarios of the present application is now exemplarily described.

[0025] The camera's internal and external parameters are very important parameters of the camera. Based on the camera's internal and external parameters, the three-dimensional coordinate points in the world coordinate system can be projected onto the camera image. In addition, if the depth information of each pixel on the camera image is obtained through methods such as depth estimation, the camera's internal and external parameter matrix can be used to project the pixels on the image into the world coordinate system to obtain the three-dimensional coordinates of the pixels.

[0026] The camera intrinsics describe the conversion relationship between the coordinates in the camera coordinate system and the pixel coordinates in the image. It mainly includes two parts. To the camera image coordinate system The conversion relationship (the camera coordinate system is a three-dimensional coordinate system, the origin is at the camera optical center, the X-axis is parallel to the horizontal axis of the camera image, facing right, the Y-axis is parallel to the vertical axis of the camera image, facing downward, and the Z-axis conforms to the right-hand coordinate system rule; the image coordinate system is a two-dimensional plane coordinate system, with the intersection of the camera principal optical axis and the image plane as the origin, the X-axis is parallel to the X-axis of the camera coordinate system, facing right, and the Y-axis is parallel to the Y-axis of the camera coordinate system, facing downward). Based on the pinhole imaging principle, the positions of objects in the image coordinate system and the camera coordinate system are in equal proportion. Based on this equal proportion, the coordinates in the camera coordinate system can be projected to the influence coordinate system. This projection relationship is related to the focal length of the camera and can be expressed by a matrix, assuming that The second part describes the image coordinate system coordinates To the camera image pixel coordinate system coordinates The conversion relationship (the origin of the image pixel coordinate system is the upper left corner of the image, The axis is parallel to the horizontal axis of the image and points to the right. The axis is parallel to the vertical axis of the image and points downward). This part is related to the physical size of the camera pixel and the image resolution. This projection relationship can be represented by a matrix, assuming that . Relying on the projection matrix The camera's intrinsic matrix can be obtained , relying on this projection matrix, the pixel coordinates of the camera image in the camera coordinate system can be easily calculated.

[0027] The camera extrinsic parameter describes the conversion relationship between the coordinates in the world coordinate system and the coordinates in the camera coordinate system, which is the transformation relationship between the rigid body in two positions in the world coordinate system. It mainly includes two parts: 1. The rotation amount around the three coordinate axes from the world coordinate system posture to the camera coordinate system posture, which can be expressed by the rotation matrix Representation; 2. The translation amount from the world coordinate system position to the camera coordinate system position along the three coordinate axes can be expressed by the translation matrix These transformation relationships can be expressed by an augmented 4 by 4 matrix express:

[0028] The camera intrinsic parameters are mainly determined by factors such as camera focal length, pixel physical size, image resolution, distortion parameters, etc. These parameters can be obtained after the camera is manufactured. Therefore, the camera intrinsic parameters can be calibrated when the camera leaves the factory. This application mainly focuses on how to obtain the camera's extrinsic parameters, that is, the transformation matrix that converts coordinates in the world coordinate system to coordinates in the camera coordinate system. Due to different camera usage scenarios and different selected world coordinate systems, the camera extrinsic parameters are generally obtained by calibration after the camera is installed.

[0029] Therefore, how to effectively calibrate the internal and external parameters of the camera is a very important issue. For the camera's internal parameters, they can generally be calibrated by Zhang's calibration method when the camera leaves the factory. However, for the camera's external parameters, due to the uncertainty of the camera installation scene, the camera's external parameter calibration needs to rely on the selection of the world reference coordinate system. Therefore, at present, technicians are usually required to perform on-site calibration after the camera is installed and deployed. The mathematical expression of the calibration process can be:

[0030] Among them, the coordinates of the target object in the radar coordinate system are , the coordinates of the target object in the pixel coordinate system , K is the camera's intrinsic parameter matrix (which can be directly obtained because it has been calibrated at the factory), T is the extrinsic parameter matrix to be determined, are the coefficients used for dimensionality transformation.

[0031] See also Figure 1 , Figure 1 This is a flowchart of an exemplary embodiment of the camera calibration method based on radar matching of the present application. Specifically, it may include the following steps: Step S110, performing radar detection and image recognition processing on the area to be detected respectively, to obtain radar data and image data of the target object in the area to be detected, wherein the radar data includes the radar position of the target object in the area to be detected, and the image data includes the pixel position of the target object in the area to be detected.

[0032] It should be noted that the present application can perform radar detection and image recognition processing by using an integrated radar and camera, or a separately installed radar and camera. Among them, the radar of the present application may include but is not limited to millimeter wave radar, laser radar, etc. The camera of the present application may include but is not limited to common visible light cameras, infrared cameras, etc.

[0033] The target object of the present application may be one or more, and the multiple target objects may be the same or different. Furthermore, the multiple target objects may appear in the area to be detected at the same time or at different times, which is not limited here.

[0034] The radar can perform radar detection on the first area, and the camera can collect images on the second area. The area to be detected refers to the common area (intersection area) of radar detection and image recognition processing (image detection), which is equivalent to that if the first area and the second area are the same, the first area or the second area can be the area to be detected (or part of the first area or the second area is the area to be detected); if the first area and the second area are different, the intersection area of ​​the first area and the second area is determined as the area to be detected (or part of the intersection area can be the area to be detected).

[0035] For example, taking a traffic scene as an example, when installing a camera in a traffic scene, the camera can be mounted on a beam beside the road, on a traffic gantry, etc., and the camera field of view can be made as wide as possible. When installing a radar in a traffic scene, the radar can also be installed in the same way or in other locations, and the radar must have a certain field of view. Among them, the radar can be installed close to the camera, and it is ensured that it has a similar observation angle, so that the external parameter calibration process is more accurate.

[0036] Specifically, in this step, radar data refers to data determined by radar, and image data refers to data determined by camera. Radar data may include but is not limited to the position, speed, direction, and time of the target object in the area to be detected. Similarly, image data may also include but is not limited to the position, speed, direction, and time of the target object in the area to be detected. Among them, the position information of the present application can be represented in the form of coordinates, and can also be represented in the form of azimuth and distance, which is not limited here. The following text mainly uses examples in the form of coordinates.

[0037] For example, when a target object enters the detection area where the radar and camera are cross-sensed, the detection frame (detection area) of the target object can be obtained through image recognition technology based on the data collected by the camera. ,in, Indicates the pixel coordinates of the upper left corner of the detection box. represents the pixel coordinates of the lower right corner of the detection box, and the mathematical expression of the center coordinates of the target object can be The center coordinates of the target object can be used to summarize the overall position information of the target object and obtain its pixel position. Similarly, the position and speed information of the target object can be obtained from the data collected by the radar, which can be expressed as , where x, y, and z represent the radar coordinates of the target object (there are many specific calculation methods, such as taking its centroid coordinates or other point cloud coordinates, or calculating the coordinates of multiple point clouds, etc. For details, please refer to the existing radar positioning technology, which will not be described here), and v represents its speed. Indicates the heading of the target object.

[0038] Step S120, synchronizing the radar position and the pixel position to obtain a position matching pair; the position matching pair includes a target radar position and a target pixel position of the target object.

[0039] As described in conjunction with the above steps, during the implementation of the present application, the area to be detected can be continuously detected.

[0040] For example, in the time period If a target object passes through the detection area, the image data captured by the camera and the point cloud data detected by the radar can be frame synchronized so that the pixel coordinates of the target object in the image data and the radar coordinates in the radar data can be successfully matched. Frame data, through the target detection algorithm in image recognition, the target object in the time period can be obtained Inside Pixel detection box data in frame data: Similarly, the target detection algorithm detected by radar can obtain the target object in the radar coordinate system. Frame position, speed, and heading information: Since the image data and radar data have been frame aligned, and the target object in the image data and the target object in the radar data are judged to be the same target. Frame data, we can obtain N sets of target radar coordinates and target pixel coordinates that match each other (take the center point pixel of the image target detection frame) to form a set of position matching pairs Among them, the target radar position and the target pixel position refer to the radar position and pixel position that are synchronized and matched successfully through the above method.

[0041] It should be noted that the camera extrinsic parameters have 6 degrees of freedom. In the above calibration process, a set of position matching pairs can provide two constraints. Therefore, the camera extrinsic calibration process requires at least 3 sets of non-collinear position matching pairs. They can be from the same target object or from different target objects, which is not limited here. In practical applications, a larger number of position matching pairs (for example, 10 sets or more) are usually used to participate in the calibration process of the camera extrinsic parameters to improve the robustness and accuracy of the calibration results.

[0042] Step S130, projecting the target radar position to the pixel coordinate system where the target pixel position is located according to the acquired initial external parameters and the pre-calibrated camera internal parameters, to obtain the projection position of the target radar position in the pixel coordinate system.

[0043] The initial external parameter refers to the initial external parameter matrix of the camera The initial external parameter matrix can be obtained by randomly initializing according to a preset external parameter matrix template (randomly filling values ​​into the external parameter matrix template to initialize the matrix); or the initial external parameter matrix can be determined according to the first few position matching pairs in the position matching pair set. The camera intrinsic parameter refers to the camera's intrinsic parameter matrix (Factory calibration).

[0044] Therefore, according to the initial external parameter matrix of the camera and the internal parameter matrix A coordinate transformation function is constructed to project the target radar position in the position matching pair to the pixel coordinate system where the target pixel position is located, and obtain the projection position of the target radar position in the pixel coordinate system.

[0045] It should be noted that since the initial external parameter matrix is ​​randomly generated or based on a small amount of reference data, the accuracy may not be high. Therefore, there may be some deviation between the radar projection position of the target object and the actual target pixel position of the target object in the image.

[0046] Step S140, iteratively adjusting the initial extrinsic parameter according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter.

[0047] Combined with the above steps, multiple sets of position matching pairs can be obtained as the detection of the area to be detected continues. For the coordinate change function constructed by the projection process of the above steps, although its initial external parameter matrix may not be accurate enough, the application can adjust the initial external parameter matrix in the process of importing the target radar position in the multiple sets of position matching pairs into the coordinate change function to obtain the corresponding projection position. Then, analyze the change of the projection position, thereby determining the optimized external parameter matrix and obtaining the target external parameter.

[0048] For example, in the present application, the initial extrinsic parameters may be iteratively adjusted according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameters. The process of iteratively adjusting the initial extrinsic parameters in the present application may include but is not limited to using a gradient-free global optimization algorithm to minimize the objective function to obtain a camera target extrinsic parameter matrix with better effect (for example, when the projection position and the target pixel position are closest). As a result, the entire camera calibration process can be fully automated without human intervention, thus improving the efficiency of camera calibration.

[0049] Among them, the distance between the projection position and the target pixel position of the present application may include but is not limited to the Cartesian distance between the projection position coordinates and the target pixel position coordinates in the image space; or the vector distance between the feature vector of the projection position (or the feature vector of the projection position and its neighborhood position) and the feature vector of the target pixel position (or the feature vector of the target pixel position and its neighborhood position) in the vector space; or the vector distance between the feature vector of the target object corresponding to the projection position and the feature vector of the target object corresponding to the target pixel position; it can also be the result of weighted summation of the above at least two distances according to preset weights as the final distance representation, which is not limited here. For example, the average value of the Cartesian distance and the feature vector distance is used as the final distance between the two positions. It should be noted that the feature distance of the present application may be but is not limited to the appearance features, semantic features, etc. of the target object, which are not limited here.

[0050] It can be seen that the present application can obtain radar data and image data of the target object in the area to be detected, including the radar position and pixel position of the target object, by performing radar detection and image recognition processing on the area to be detected respectively. The radar position and pixel position are processed synchronously to obtain a position matching pair, and the radar position and pixel position in the synchronized position matching pair are the target radar position and target pixel position. According to the acquired initial external parameters and the pre-calibrated camera internal parameters, the target radar position is projected to the pixel coordinate system where the target pixel position is located to obtain the projection position of the target radar position in the pixel coordinate system, but the uncalibrated initial external parameters may not be accurate enough to accurately project the target radar position. Therefore, according to the distance between the projection position and the target pixel position, the initial external parameters are iteratively adjusted to obtain the target external parameters. This achieves the calibration of the camera external parameters and improves the camera calibration efficiency.

[0051] Based on the above embodiments, the embodiment of the present application describes the steps of performing radar detection and image recognition processing on the area to be detected, and obtaining radar data and image data of the target object in the area to be detected. Specifically, the method of this embodiment includes the following steps: Perform radar detection on the detection area to obtain the radar position of the target object; perform image recognition processing on the detection area to obtain the detection area of ​​the target object; determine the pixel position of the target object based on the pixel information of the detection area.

[0052] Among them, the radar position of the target object can be obtained by using the radar sensor to perform radar detection on the detection area. The specific method can refer to the existing radar positioning technology, for example, the radar position of the target object can be determined based on the reflected millimeter wave received by the millimeter wave radar, and the radar position of the target object can be determined based on the laser point cloud received by the laser radar, etc., which will not be elaborated here.

[0053] A camera is used to capture an image of the area to be detected, and image recognition processing is performed on the area to be detected to obtain a detection area (such as a detection box or a boundary box, etc.) of the target object. According to the method of the aforementioned embodiment, the pixel position of the target object (such as the coordinates of the center point of the detection box) can be determined based on the pixel coordinates of the corner points of the detection box in the image.

[0054] Based on the above embodiments, the embodiment of the present application describes the steps before radar detection and image recognition processing are performed on the area to be detected, and radar data and image data of the target object in the area to be detected are obtained. Specifically, the method of this embodiment includes the following steps: Collect a regional image of the area to be detected; perform lane line detection on the regional image to obtain lane line information.

[0055] In conjunction with the above-mentioned embodiments, the camera calibration method based on radar matching of the present application can calibrate cameras in various indoor and outdoor scenes (such as traffic roads, enterprise parks, etc.). For traffic scenes, when calibrating cameras in traffic scenes, lane lines in the traffic scenes can be used as reference.

[0056] For example, reference may be made to Figure 2 As shown, Figure 2 This is a schematic diagram of an exemplary application scenario in the camera calibration method based on radar matching of the present application. By collecting the regional image of the area to be detected by the camera, the lane line detection can be performed on the regional image using image recognition technology to obtain the lane line information in the regional image. Alternatively, the regional image can be displayed by manually drawing lane lines, and the lane lines in the regional image are generated in response to the received drawing instructions, which is not limited here. The lane line information can be shared by image data and radar data.

[0057] Based on the above embodiment, the embodiment of the present application describes the steps after performing lane line detection on the regional image and obtaining lane line information. Specifically, the method of this embodiment includes the following steps: In response to the lane line information indicating that there are at least two lanes in the area to be detected, radar detection and image recognition processing are performed on each lane respectively, and at least one target object is determined from each lane.

[0058] Combined with the above embodiments, Figure 2 In the application environment shown, the number of lanes in the area to be detected can be determined based on the lane line information. If there is one lane in the area to be detected, radar detection and image detection are directly performed on the lane to determine at least one target object. If there are at least two lanes in the area to be detected, each lane needs to be subjected to radar detection and image recognition processing respectively, and at least one target object is determined from each lane. Among them, the target objects in each lane are not necessarily detected at the same time. Optionally, if not every lane has a target object with successfully paired radar coordinates, radar detection and image recognition processing can be continued until the conditions are met. Or if the target object cannot be detected in one or more lanes for a long time, subsequent processing can still be performed based on the target objects detected in other lanes. Selecting at least one target object from each lane can ensure the accuracy of the parameter calibration process.

[0059] This embodiment mainly uses Figure 2 The three lanes shown in the figure are used as an example to illustrate how to determine the lane where the target object is located in the image. The three lanes are represented from left to right as , four straight lines are drawn in the figure to represent . Where a, b and c are coefficients. For example, when a vehicle passes through the area to be detected and appears in the image, the target detection algorithm of the image recognition technology can obtain the pixel bounding box of the vehicle on the image, which is generally expressed as ,in Indicates the pixel coordinates of the upper left corner of the target pixel box. Indicates the coordinates of the lower right corner of the target pixel frame. Generally, the center coordinates of the target pixel frame can be To calculate the lane this target is in. The pseudo code of the calculation method can be as follows:

[0060] The return value of the above pseudo code can represent the lane where the target object is located. If -1 is returned, it means that the lane calculation failed and relevant prompt information can be output. Therefore, this application can use the above method to understand the number of lanes in the perceived environment, and each time a vehicle passes by, the lane number to which the vehicle belongs can be calculated. When a vehicle enters the intersection area of ​​the camera and radar perception environment, the target pixel frame can be obtained on the image. , on the radar, the position, speed and heading of the target can be obtained, expressed as .

[0061] Based on the above embodiment, the embodiment of the present application describes the steps of synchronizing the radar position and the pixel position to obtain a position matching pair. Specifically, the method of this embodiment includes the following steps: In response to the consistent frame rate between radar detection and image recognition processing, a radar timestamp of radar data and an image timestamp of image data are obtained; and each radar position and each pixel position are time-aligned according to the radar timestamp and the image timestamp to obtain a position matching pair.

[0062] In conjunction with the above-mentioned embodiment, since the acquisition frequency of the radar and the acquisition frequency of the camera may be different, the camera calibration method of the present application needs to be calibrated based on the radar data and camera data (image data) that are matched and aligned with each other. Therefore, in the process of determining the position matching pair (matching point), it is necessary to first determine whether the frame rate of the radar detection and the frame rate of the image recognition are consistent.

[0063] In response to the consistent frame rate between radar detection and image recognition processing (or the radar acquisition frame rate is consistent with the camera acquisition frame rate), time alignment processing can be performed based on the timestamp of the radar data and the timestamp of the image data to match the radar data and the image data in time, thereby obtaining at least three sets of position matching pairs (if no position matching pair exists, relevant prompt information is generated).

[0064] Specifically, each radar data may have corresponding radar timestamp information, and each image data may also have corresponding image timestamp information. Therefore, the radar data and image data with the same radar timestamp and image timestamp may be determined as a set of position matching pairs (matching points). Among them, the position matching pairs obtained by successful matching include the target radar data and the target image data.

[0065] For example, in the traffic application scenario of the above-mentioned embodiment, since the image and radar data have been frame aligned, and the image and radar are identified as the same target, therefore, the acquired Frame data, we can get The matching point pairs of radar coordinates and image pixel coordinates (the pixels at the center of the bounding box of the target detection image) are obtained. In order to improve the accuracy of the subsequent numerical optimization estimation of the camera extrinsic parameters, a target object can be selected in each lane to obtain the matching point pairs of its radar coordinates and image pixel coordinates. For example, the matching pair is obtained in lane 0. , get the matching pair in lane 1 , get the matching pair in lane 2 .

[0066] From the above, we can see that for each lane in the intersection area of ​​the camera and radar perception environment, some matching pairs between image pixel coordinates and radar coordinates can be automatically obtained. These coordinate matching pair data can be expressed as , indicating that a total of coordinate matching pairs.

[0067] Based on the above embodiment, the embodiment of the present application describes the steps of synchronizing the radar position and the pixel position to obtain a position matching pair. Specifically, the method of this embodiment includes the following steps: In response to the frame rate inconsistency between radar detection and image recognition processing, radar data and image data are subjected to frame extraction processing to obtain target radar data and target image data; the time difference between the target radar data and the target image data is less than a preset time difference threshold; according to the radar timestamp of the target radar data and the image timestamp of the target image data, the target radar position of each target radar data and the target pixel position of each target image data are time-aligned to obtain a position matching pair.

[0068] In conjunction with the above-mentioned embodiment, in response to the inconsistent frame rate between radar detection and image recognition processing (or the inconsistent acquisition frame rate of the radar and the acquisition frame rate of the camera), if the radar data and the camera data (image data) are to be aligned, it is necessary to perform frame extraction processing on the radar data and / or the image data to obtain the target radar data and / or the target image data. The purpose of the frame extraction processing is to align the radar data and the image data in terms of quantity and time. Therefore, according to the actual radar frame rate and camera frame rate, the radar data can be selectively subjected to frame extraction processing, or the image data can be subjected to frame extraction processing, or the radar data and the image data can be subjected to frame extraction processing, which is not limited here. The data obtained after the frame extraction processing is the target radar data and the target image data, wherein the time difference between the target radar data and the target image data should be less than the preset time difference threshold, thereby being able to more accurately characterize the matching relationship between the radar data and the image data. For example, if the camera frame rate is inconsistent with the radar frame rate, a frame extraction strategy can be adopted to synchronize the data acquisition frame rates of the camera and radar to ensure that the time difference between the camera image data and the radar point cloud data is less than 100 milliseconds for each frame of data collected.

[0069] Similarly, after obtaining the target radar data and the target image data, we can still refer to the description of the aforementioned embodiment, and time-align the target radar position of each target radar data and the target pixel position of each target image data according to the radar timestamp of the target radar data and the image timestamp of the target image data to obtain a position matching pair.

[0070] Based on the above embodiment, the embodiment of the present application describes the steps of projecting the target radar position to the pixel coordinate system where the target pixel position is located according to the acquired initial external parameters and the pre-calibrated camera internal parameters to obtain the projection position of the target radar position in the pixel coordinate system. Specifically, the method of this embodiment includes the following steps: According to the timing information of each position matching pair, an initial data pair earlier than the current timing is determined from each position matching pair, and the initial extrinsic parameter is determined according to the initial data pair; the target radar position in each position matching pair is homogenized to obtain the homogenized radar position; the homogenized radar position is projected to the pixel coordinate system according to the initial extrinsic parameter and the camera intrinsic parameter to obtain the projected radar position; the projected radar position is dehomogenized to obtain the projected position.

[0071] Each position matching pair has its timing information, which can be determined based on the timestamp of the radar data and / or the timestamp of the image data in the matching pair, which will not be described in detail here. is the first frame matching pair, is the second frame matching pair. The timing of the first frame matching pair is earlier than that of the second frame matching pair. If the current timing is the Nth frame, the current matching pair is . An initial data pair (initial matching pair) earlier than the current time sequence can be selected, such as the first frame matching pair and the second frame matching pair, and the coordinate transformation function provided in the above embodiment can be substituted to calculate the initial external parameter matrix. The radar coordinates in the obtained coordinate matching pair are homogenized, and then the camera internal and external parameter matrix is ​​used to project the homogenized radar coordinates to the image pixel coordinate system, and then the projected coordinates are dehomogenized to obtain the pixel coordinates of the radar coordinates after the camera internal and external parameter projection, that is, the projection position (projection coordinates). The mathematical expression of this process can be:

[0072] in, Represents the dehomogenization operation on the projected coordinates. Indicates radar coordinates The pixel coordinates (projection coordinates) obtained after projecting the camera's internal and external parameters.

[0073] Based on the above embodiment, the embodiment of the present application describes the steps of iteratively adjusting the initial extrinsic parameter according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter. Specifically, the method of this embodiment includes the following steps: According to the Cartesian distance between the projection position and the pixel position, the initial extrinsic parameter is iteratively adjusted through a preset optimization algorithm so that the Cartesian distance is less than or equal to a preset distance threshold or the number of iterative adjustments is greater than a preset number threshold, thereby obtaining the target extrinsic parameter.

[0074] In combination with the above-mentioned embodiment, after the projection position is obtained, the initial extrinsic parameters can be iteratively adjusted according to the Cartesian distance between the projection position and the pixel position, and the objective function of the numerical optimization is constructed, so as to solve the optimal camera extrinsic parameter matrix. It can make all the collected pixel coordinates and radar coordinates match each other as much as possible. , the radar coordinates Pixel coordinates obtained by projecting to the pixel coordinate system and The distance is the closest.

[0075] Specifically, the mathematical expression of the objective function of this embodiment can be:

[0076] in, Represents the Cartesian distance between two coordinates. After constructing the above objective function, we can use a gradient-free global optimization algorithm to minimize the objective function to obtain a better camera extrinsic parameter matrix estimate. The entire process can be fully automated without human involvement, improving the efficiency of camera calibration.

[0077] Exemplarily, the optimization algorithm may refer to but is not limited to using a particle swarm optimization algorithm to solve the optimal camera extrinsic matrix .For example: 1. Initialize optimization algorithm parameters: set the number of particles to is 50, and the maximum number of iterations of the algorithm is is 10000, and the inertia weight is is 0.3, and the cognitive factor is is 0.2, and the social factor is is 0.3, and the search range is ,in Is a 12-dimensional vector, representing the external parameter matrix There are 12 parameters to be optimized. Set to -100000.0, Set to 100000.0.

[0078] 2. Initialize particles: Randomly initialize the position of each particle and speed :

[0079] in , which means the first Particles, , indicating the parameters to be optimized No. dimension. Indicates that the value range is Randomly select a number from a uniform distribution. Set the value to 100.0.

[0080] Record the current best position of each particle and the global optimal position :

[0081] The above formula indicates that in the current iteration 0, the initialized particle position is recorded as the optimal position, and the global optimal position is the particle position among all particles that minimizes the value of the optimization objective function.

[0082] 3. Iterative optimization: For each iteration : 3.1 Update speed:

[0083] in and is a random number between [0,1]).

[0084] 3.2 Update location:

[0085] 3.3 Evaluate the objective function value of the particle:

[0086] 3.4 Update the optimal position of a single particle:

[0087] That is, if the current particle position loss function is smaller than the historical best position loss function value of this particle, the current particle position is recorded as the best position of the particle; otherwise, the historical best position is recorded as the current particle best position.

[0088] 3.5 Update the global best position:

[0089] Among the best positions of all particles, the position that minimizes the loss function is recorded as the global best position.

[0090] 4. Output the optimal estimation result: According to the above iterative optimization steps, when the maximum number of iterations is reached After that, the optimal solution of the camera extrinsic matrix is ​​output for: , the loss function at this time is In addition to setting the algorithm termination condition to the maximum number of iterations, you can also set it to stop optimizing and solving the minimum value of the objective function. If the algorithm to be optimized meets the set termination condition, you can stop optimizing and solve, output the camera extrinsic parameter estimation, and obtain the target extrinsic parameter.

[0091] On the basis of the above embodiments, it should be noted that in the embodiments of the present application, in the specific implementation process of the present application, commonly used image target detection algorithms can be used for image recognition processing, such as the YOLO series and other algorithms, which can identify the pixel coordinates of the bounding box of the target object in the image, and the coordinates are the pixel coordinates of the upper left and lower right corners of the bounding box. In traffic application scenarios, you can focus on identifying vehicle targets in images. In addition, an image target tracking algorithm can be used so that the same target object detected by the above method can maintain the same ID number (unique identifier). For radar detection, a radar point cloud target detection algorithm can be used to obtain information about target objects moving in the environment. And a radar target tracking algorithm is used so that the same target object detected by the radar in the above method can maintain the same ID number.

[0092] In summary, since both the image and radar data are processed by their own tracking algorithms, the same target object can keep the same ID number. Therefore, a moving target can theoretically be matched only once, and the image pixel coordinates and radar coordinates of the aligned timestamp of this target can be taken out to form a coordinate matching pair.

[0093] Through the method provided in the above embodiment, the present application can automatically extract the matching pair of radar coordinates without manual participation, with a high degree of automation and improved calibration efficiency. The camera extrinsic matrix is ​​solved based on the numerical optimization method. The optimization objective function can be defined according to the task requirements, which is highly flexible.

[0094] It should be further explained that the execution subject of the camera calibration method based on radar visual matching can be a camera calibration device based on radar visual matching, for example, the camera calibration method based on radar visual matching can be executed by a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the camera calibration method based on radar visual matching can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0095] Figure 3 FIG. 1 is a block diagram of a camera calibration device based on radar matching, which is an exemplary embodiment of the present application. Figure 3 As shown, the exemplary camera calibration device 300 based on radar matching includes: a detection module 310, a synchronization module 320, a projection module 330 and an adjustment module 340. Specifically: The detection module 310 is used to perform radar detection and image recognition processing on the area to be detected, respectively, to obtain radar data and image data of the target object in the area to be detected, wherein the radar data includes the radar position of the target object in the area to be detected, and the image data includes the pixel position of the target object in the area to be detected.

[0096] The synchronization module 320 is used to synchronize the radar position and the pixel position to obtain a position matching pair; the position matching pair includes a target radar position and a target pixel position of the target object.

[0097] The projection module 330 is used to project the target radar position to the pixel coordinate system where the target pixel position is located according to the acquired initial external parameters and the pre-calibrated camera internal parameters, so as to obtain the projection position of the target radar position in the pixel coordinate system.

[0098] The adjustment module 340 is used to iteratively adjust the initial extrinsic parameter according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter.

[0099] In this exemplary camera calibration device based on radar vision matching, by performing radar detection and image recognition processing on the area to be detected respectively, radar data and image data of the target object in the area to be detected can be obtained, including the radar position and pixel position of the target object. The radar position and pixel position are synchronously processed to obtain a position matching pair, and the radar position and pixel position in the synchronized position matching pair are the target radar position and target pixel position. According to the acquired initial external parameters and the pre-calibrated camera internal parameters, the target radar position is projected to the pixel coordinate system where the target pixel position is located, and the projection position of the target radar position in the pixel coordinate system is obtained, but the uncalibrated initial external parameters may not be enough to accurately project the target radar position. Therefore, according to the distance between the projection position and the target pixel position, the initial external parameters are iteratively adjusted to obtain the target external parameters. In this way, the calibration of the camera external parameters is achieved, and the camera calibration efficiency is improved.

[0100] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0101] The functions of each module can be found in the embodiment of the camera calibration method based on radar matching, which will not be described in detail here.

[0102] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned camera calibration method embodiments based on radar matching. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0103] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned camera calibration method embodiments based on radar matching. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0104] In this exemplary electronic device, by performing radar detection and image recognition processing on the area to be detected respectively, radar data and image data of the target object in the area to be detected can be obtained, including the radar position and pixel position of the target object. The radar position and pixel position are synchronously processed to obtain a position matching pair, and the radar position and pixel position in the synchronized position matching pair are the target radar position and target pixel position. According to the acquired initial external parameters and the pre-calibrated camera internal parameters, the target radar position is projected to the pixel coordinate system where the target pixel position is located, and the projection position of the target radar position in the pixel coordinate system is obtained, but the uncalibrated initial external parameters may not be accurate enough to accurately project the target radar position. Therefore, according to the distance between the projection position and the target pixel position, the initial external parameters are iteratively adjusted to obtain the target external parameters. In this way, the calibration of the camera external parameters is achieved, and the camera calibration efficiency is improved.

[0105] See also Figure 5 , Figure 5The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned camera calibration method embodiments based on radar matching.

[0106] In this exemplary storage medium, by running the program instructions in the storage medium, radar detection and image recognition processing are performed on the detection area respectively, and radar data and image data of the target object in the detection area can be obtained, including the radar position and pixel position of the target object. The radar position and pixel position are synchronized to obtain a position matching pair, and the radar position and pixel position in the synchronized position matching pair are the target radar position and target pixel position. According to the acquired initial external parameters and the pre-calibrated camera internal parameters, the target radar position is projected to the pixel coordinate system where the target pixel position is located, and the projection position of the target radar position in the pixel coordinate system is obtained, but the uncalibrated initial external parameters may not be accurate enough to accurately project the target radar position. Therefore, according to the distance between the projection position and the target pixel position, the initial external parameters are iteratively adjusted to obtain the target external parameters. In this way, the calibration of the camera external parameters is achieved, and the camera calibration efficiency is improved.

[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0108] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0110] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A camera calibration method based on radar matching, characterized in that: The method comprises: Performing radar detection and image recognition processing on the area to be detected respectively to obtain radar data and image data of the target object in the area to be detected, wherein the radar data includes the radar position of the target object in the area to be detected, and the image data includes the pixel position of the target object in the area to be detected; The radar position and the pixel position are synchronously processed to obtain a position matching pair; the position matching pair includes a target radar position and a target pixel position of the target object; The target radar position is projected to the pixel coordinate system where the target pixel position is located according to the acquired initial external parameters and the pre-calibrated camera internal parameters, so as to obtain the projection position of the target radar position in the pixel coordinate system; The initial extrinsic parameter is iteratively adjusted according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter.

2. The method according to claim 1, characterized in that The radar detection and image recognition processing are respectively performed on the area to be detected to obtain radar data and image data of the target object in the area to be detected, including: Performing radar detection on the area to be detected to obtain the radar position of the target object; Performing image recognition processing on the area to be detected to obtain a detection area of ​​the target object; The pixel position of the target object is determined according to the pixel information of the detection area.

3. The method according to claim 1, characterized in that Before respectively performing radar detection and image recognition processing on the area to be detected to obtain radar data and image data of the target object in the area to be detected, the method further includes: Acquiring a regional image of the area to be detected; Perform lane line detection on the area image to obtain lane line information.

4. The method according to claim 3, characterized in that After performing lane line detection on the regional image to obtain the lane line information, the method further includes: In response to the lane line information indicating that there are at least two lanes in the area to be detected, radar detection and image recognition processing are performed on each lane respectively, and at least one target object is determined from each lane.

5. The method according to claim 1, characterized in that The step of synchronously processing the radar position and the pixel position to obtain a position matching pair includes: In response to the frame rates between the radar detection and the image recognition processing being consistent, acquiring a radar timestamp of the radar data and an image timestamp of the image data; According to the radar timestamp and the image timestamp, each radar position and each pixel position are time-aligned to obtain the position matching pair.

6. The method according to claim 1, characterized in that The step of synchronously processing the radar position and the pixel position to obtain a position matching pair includes: In response to the frame rate inconsistency between the radar detection and the image recognition processing, performing frame extraction processing on the radar data and the image data to obtain target radar data and target image data; the time difference between the target radar data and the target image data is less than a preset time difference threshold; According to the radar timestamp of the target radar data and the image timestamp of the target image data, the target radar position of each target radar data and the target pixel position of each target image data are time-aligned to obtain the position matching pair.

7. The method according to claim 1, characterized in that The method of projecting the target radar position to the pixel coordinate system where the target pixel position is located according to the acquired initial external parameters and the pre-calibrated camera internal parameters to obtain the projection position of the target radar position in the pixel coordinate system includes: According to the timing information of each position matching pair, determine an initial data pair earlier than the current timing from each position matching pair, and determine the initial external parameter according to the initial data pair; Performing homogenization processing on the target radar positions in each position matching pair to obtain the homogenized radar positions; Projecting the homogenized radar position to the pixel coordinate system according to the initial external parameters and the camera internal parameters to obtain the projected radar position; The projected radar position is subjected to inverse homogenization processing to obtain the projected position.

8. The method according to claim 1, characterized in that The iteratively adjusting the initial extrinsic parameter according to the distance between the projection position and the target pixel position to obtain the target extrinsic parameter comprises: According to the Cartesian distance between the projection position and the pixel position, the initial extrinsic parameter is iteratively adjusted through a preset optimization algorithm so that the Cartesian distance is less than or equal to a preset distance threshold or the number of iterative adjustments is greater than a preset number threshold, thereby obtaining the target extrinsic parameter.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images