Calibration method and device for vehicle-mounted surround view camera external parameters, computer device and medium

Abstract

This invention provides a method, apparatus, computer device, and medium for calibrating the extrinsic parameters of a vehicle-mounted surround-view camera, relating to the field of camera parameter calibration technology. The method includes: acquiring video of all targets around a vehicle; generating a target map composed of the targets based on the position information of each target in the video; determining the vehicle's center coordinate system based on the position information of targets placed at relevant positions along the vehicle's central axis in the target map; converting the target map to the vehicle center coordinate system; and determining the extrinsic parameters of each vehicle-mounted surround-view camera based on the position information of the camera-positioned targets in the images acquired by each camera and their position information in the target map. This approach improves the flexibility and accuracy of the calibration results.

Description

Technical Field

[0001] This invention relates to the field of camera parameter calibration technology, and in particular to a method, apparatus, computer equipment, and medium for calibrating the extrinsic parameters of a vehicle-mounted surround-view camera. Background Technology

[0002] Existing extrinsic calibration techniques either cannot avoid human measurement errors or require the construction of costly calibration sites. More similar solutions include:

[0003] Currently, the existing methods for calibrating the external parameters of vehicle surround view systems include the following:

[0004] A checkerboard pattern is used for extrinsic parameter calibration of the vehicle surround view system. However, because the checkerboard pattern position parameters still need to be measured, and calibration only supports four vehicle types, issues remain regarding calibration in non-standard environments, where accuracy may be insufficient. In non-standard environments, the calibrator needs to place and measure the checkerboard calibration cloth, and the regularity of placement and measurement accuracy are uncontrollable depending on the calibrator, potentially introducing measurement errors and affecting calibration accuracy. Furthermore, the limited support for only four vehicle types restricts application flexibility.

[0005] Another method involves deploying large and small QR code calibration boards. The large labels obtain bird's-eye view attitude images from each camera, while the small labels infer attitude transitions between adjacent cameras. This approach reduces the measurement of the target, but still requires measuring the distance from the vehicle's coordinate system to the target's coordinate system. Therefore, measurement errors are still introduced, affecting calibration accuracy. Furthermore, the required QR code size is too large, making it inconvenient for technical support personnel to carry and resulting in poor ease of use.

[0006] Other possible solutions include calibration based on a high-precision pre-set site and online calibration based on special environments (such as lane lines). However, these solutions are highly dependent on the site and have poor application flexibility. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a calibration method for the extrinsic parameters of a vehicle-mounted surround-view camera to solve the technical problems of low accuracy and poor flexibility in calibration results in the prior art. The method includes:

[0008] Video of all targets is captured around the vehicle. The targets are coded graphic marker boards. Multiple targets are placed around the vehicle body, and the targets are placed at relevant positions along the central axis of the vehicle body.

[0009] Based on the location information of each target in the video, a target map composed of each target is generated;

[0010] Based on the position information of the target placed at a relevant position on the vehicle's central axis in the target map, the vehicle's center coordinate system is determined, and the vehicle's center point is the origin of the vehicle's center coordinate system.

[0011] Transform the target map to the vehicle center coordinate system;

[0012] For each of the vehicle-mounted surround view cameras, the targets included in the images captured by each of the vehicle-mounted surround view cameras are regarded as camera positioning targets for each of the vehicle-mounted surround view cameras. Based on the position information of the camera positioning targets in the images captured by each of the vehicle-mounted surround view cameras and the position information in the target map, the extrinsic parameters of each of the vehicle-mounted surround view cameras are determined.

[0013] This invention also provides a calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera, to solve the technical problems of low accuracy and poor flexibility in calibration results in the prior art. The device includes:

[0014] A video acquisition module is used to capture video of all targets around the vehicle. The targets are coded graphic marker boards. Multiple targets are placed around the vehicle body, and the targets are placed at relevant positions along the central axis of the vehicle body.

[0015] The map generation module is used to generate a target map composed of the targets based on the location information of each target in the video;

[0016] The vehicle body coordinate system generation module is used to determine the vehicle center coordinate system based on the position information of the target placed at a relevant position on the vehicle body centerline in the target map, wherein the vehicle body center point is the origin of the vehicle center coordinate system.

[0017] The coordinate system transformation module is used to transform the target map to the vehicle center coordinate system;

[0018] The calibration module is used to, for each of the vehicle-mounted surround view cameras, regard the targets included in the images captured by each of the vehicle-mounted surround view cameras as camera positioning targets for each of the vehicle-mounted surround view cameras, and determine the extrinsic parameters of each of the vehicle-mounted surround view cameras based on the position of the camera positioning targets in the images captured by each of the vehicle-mounted surround view cameras and the position information in the target map.

[0019] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned calibration method for any of the extrinsic parameters of the vehicle surround-view camera, thereby solving the technical problems of low accuracy and poor flexibility of calibration results in the prior art.

[0020] This invention also provides a computer-readable storage medium storing a computer program that performs any of the above-described calibration methods for the extrinsic parameters of a vehicle surround-view camera, in order to solve the technical problems of low accuracy and poor flexibility of calibration results in the prior art.

[0021] Compared with the prior art, the beneficial effects that the above-mentioned at least one technical solution adopted in the embodiments of this specification can achieve include at least the following: A target is proposed for calibrating the extrinsic parameters of the vehicle surround-view camera. The target is an coded graphic marker board, which is placed around the vehicle body. Simultaneously, targets are placed at relevant positions along the vehicle's central axis. This allows for the capture of video of all targets around the vehicle. Based on the position information of each target in the video, a target map composed of the targets is generated. The vehicle's center coordinate system is determined based on the position information of the targets placed at relevant positions along the vehicle's central axis in the target map. After converting the target map to the vehicle center coordinate system, the extrinsic parameters of each vehicle surround-view camera can be determined based on the position information of the camera positioning targets in the images captured by each vehicle surround-view camera and their position information in the target map. The calibration of extrinsic parameters of vehicle-mounted surround-view cameras based on a target has been achieved. The target does not require the layout of a checkerboard or measurement, thus avoiding the introduction of measurement errors. There are no special requirements for the calibration site, which improves the flexibility of calibration and also helps to improve the accuracy of calibration results. In addition, there are no requirements for the size of the target, and large-sized targets are not needed, which helps to improve the ease of operation of calibration. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a method for calibrating the extrinsic parameters of a vehicle-mounted surround-view camera according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of vehicle body positioning provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of a target map provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a top view stitching result provided in an embodiment of the present invention;

[0027] Figure 5 This is a flowchart illustrating a method for calibrating the extrinsic parameters of a vehicle-mounted surround-view camera, as provided in an embodiment of the present invention.

[0028] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of the present invention;

[0029] Figure 7 This is a structural block diagram of a calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera provided in an embodiment of the present invention. Detailed Implementation

[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In this embodiment of the invention, a method for calibrating the extrinsic parameters of an in-vehicle surround-view camera is provided, such as... Figure 1 As shown, the method includes:

[0033] Step S101: Collect video of all targets around the vehicle. The targets are coded graphic marker boards. Multiple targets are placed around the vehicle body, and the targets are placed at relevant positions on the vehicle body's central axis.

[0034] Step S102: Generate a target map composed of the targets based on the position information of each target in the video;

[0035] Step S103: Based on the position information of the target placed at the relevant position on the center line of the vehicle body in the target map, determine the vehicle center coordinate system, where the center point of the vehicle body is the origin of the vehicle center coordinate system.

[0036] Step S104: Convert the target map to the vehicle center coordinate system;

[0037] Step S105: For each vehicle-mounted surround view camera on the vehicle, the target included in the image captured by each vehicle-mounted surround view camera is regarded as the camera positioning target of each vehicle-mounted surround view camera. Based on the position information of the camera positioning target in the image captured by each vehicle-mounted surround view camera and the position information in the target map, the extrinsic parameters of each vehicle-mounted surround view camera are determined.

[0038] Depend on Figure 1 As shown in the process, in this embodiment of the invention, the calibration of the extrinsic parameters of the vehicle-mounted surround-view camera is realized based on a target. The placement of the target does not require the layout of a checkerboard or measurement, thus avoiding the introduction of measurement errors. There are no special requirements for the calibration site, thereby improving the flexibility of calibration and also helping to improve the accuracy of calibration results. In addition, there are no requirements for the size of the target, and large-sized targets are not required, which helps to improve the ease of operation of calibration.

[0039] In practice, the aforementioned target is an coded graphic marker board, and the coding type is Aruco or AprilTag.

[0040] In practice, the aforementioned coded graphic marking board is square in shape and has a fixed size.

[0041] In practice, when calibrating the extrinsic parameters of the vehicle surround-view camera, the aforementioned multiple targets can be placed around the vehicle body. There is no need to set up a checkerboard or take measurements; only a flat surface is required. To further improve the accuracy of vehicle positioning, the targets can be placed at relevant positions along the vehicle's centerline. This allows the vehicle's center point to be quickly and accurately determined based on the targets placed at relevant positions along the vehicle's centerline, thereby establishing the vehicle's center coordinate system.

[0042] In practice, the relevant positions of the vehicle's centerline include at least two of the following:

[0043] Targets can be placed on the front of the vehicle body, the rear of the vehicle body, the left parallel line parallel to the vehicle body centerline, and the right parallel line parallel to the vehicle body centerline. Targets can also be placed on the left parallel line parallel to the vehicle body centerline and the right parallel line parallel to the vehicle body centerline, at positions that are close to the vehicle body or tires.

[0044] Specifically, depending on the vehicle body positioning, 2 / 3 of the targets can be placed in specific positions on the vehicle body (i.e., the relevant positions of the vehicle body's centerline). The targets placed in the relevant positions of the vehicle body's centerline can be called vehicle body positioning targets. Figure 2 Two strategies are provided for placing target pairs at relevant positions along the vehicle's centerline. One is, as... Figure 2As shown in Figure a, a target is placed at the front and rear of the vehicle, close to the vehicle's centerline; another method is... Figure 2 As shown in Figure b, a target is placed on the front side of the vehicle and on the left and right parallel lines along the vehicle's centerline, attached to either the vehicle body or the tires. The advantages of these two placement strategies are: first, the clear centering marks at the front and rear of the vehicle allow for precise visual positioning of the centerline; second, the left and right tires being on the ground also facilitate quick and accurate location of the positioning points. Using these positioning targets, the vehicle's center coordinate system can be accurately located, thus avoiding complex, time-consuming, and error-prone measurement methods.

[0045] In practice, the process of collecting videos of all targets around the vehicle can be achieved using a USB low-distortion camera. The USB low-distortion camera is connected to a portable PC and placed around the vehicle to collect images or videos of the targets. It is necessary to capture images of all camera-positioned targets and vehicle-positioned targets. The targets included in the images collected by the vehicle-mounted surround-view camera are camera-positioned targets, while the targets placed at relevant positions on the vehicle's central axis are vehicle-positioned targets.

[0046] In specific implementation, after obtaining videos of all targets around the vehicle body, to further improve the calibration accuracy, this embodiment provides a method for constructing a target map. For example, based on the position information of each target in the video, a target map composed of the targets is generated, including:

[0047] Identify the target in each frame of the video and extract the pixel coordinates of the corner points of the target in each frame;

[0048] Keyframe images of the video are determined, wherein the keyframe images are two adjacent images, and the two adjacent images include at least one identical target;

[0049] The keyframe image is input into a graph optimization model, and the graph optimization model outputs a transformation matrix that transforms the coordinate system of each target to the coordinate system of the target target. The target target is one of the multiple targets, and each target has its own coordinate system. The keyframe image includes the pixel coordinates of the corner points of the target.

[0050] Based on the transformation matrix, the coordinate system of each target is transformed to the coordinate system of the target target to generate the target map.

[0051] In practice, the pixel coordinates of the corner points of the target are the pixel coordinates of the top corners of the target shape. For example, if the target is square, then the pixel coordinates of the corner points of the target are the pixel coordinates of the four top corners of the square target.

[0052] In practice, in order to improve the accuracy of the target map, in this embodiment, after determining the key frame (i.e., there is at least one identical target in two consecutive frames) images, a graph optimization model is established to optimize all key frame images. The optimization objects include camera (i.e., the camera that captures video) parameters (including intrinsic and extrinsic parameters), the position and orientation of the target in the image.

[0053] Specifically, considering a target in a single frame, the graph optimization model is shown in Equation (1), and the reprojection error is calculated.

[0054]

[0055] c j Given a known quantity, denoted as the three-dimensional coordinates of the lower corner point j of target i in its own target coordinate system;

[0056] γ i Let be the quantity to be estimated, and let represent the homogeneous transformation matrix from the target coordinate system to the world coordinate system;

[0057] γ t Let be the quantity to be estimated, and let represent the homogeneous transformation matrix from the world coordinate system to the camera coordinate system, i.e., the extrinsic parameters of the camera (i.e., the camera that captures video).

[0058] δ is the quantity to be estimated, representing the intrinsic parameter matrix of the camera (i.e., the camera that captures video).

[0059] function Ψ(δ,γ) t ,γ i ,c j ) represents the projection of the j-th corner point of target i from its 3D spatial coordinates to its pixel coordinates. It first projects the corner point c j Transform to the world coordinate system, then to the camera coordinate system, and finally to the pixel coordinate system.

[0060] u i t ,j For observation, let represent the actual pixel coordinates of the j-th corner point of target i as observed in the t-th keyframe image.

[0061] e i t This represents the error in reprojection and observation of the i-th target in the t-th keyframe image.

[0062] Considering all targets in all keyframe images, the graph optimization model is shown in Equation (2), which is the model that minimizes the total reprojection error.

[0063]

[0064] Where M represents the total number of targets and N represents the total number of keyframe images.

[0065] For solving the graph optimization model, existing graph optimization model techniques were used, which treat each target as a node, calculate the optimal pose transformation between any two nodes, and then find the optimal path that minimizes the sum of the reprojection errors of all edges that make up the path.

[0066] After optimization using the graph optimization model, the position and attitude information of all targets is obtained. This involves transforming the coordinate systems of each target to the coordinate system of the target target using a transformation matrix. Then, based on this transformation matrix, the coordinate systems of each target are transformed back to the target target's coordinate system, thus generating a target map, also known as a QR code map. Figure 3 As shown.

[0067] In practice, after obtaining the target map, the vehicle body can be located, and then the vehicle center coordinate system can be determined. For example, based on the ID of the target placed at the relevant position on the vehicle's center axis, the three-dimensional coordinates of the target placed at the relevant position on the vehicle's center axis can be obtained from the target map.

[0068] The coordinates of the vehicle's center point are determined based on the three-dimensional coordinates.

[0069] The front of the vehicle body is defined as the positive y-axis direction of the vehicle center coordinate system, the right side of the vehicle body is defined as the positive x-axis direction of the vehicle center coordinate system, and the direction perpendicular to the ground surface of the vehicle is defined as the positive z-axis direction of the vehicle center coordinate system. The y-axis vector is determined based on the three-dimensional coordinates, and the x-axis vector and z-axis vector are calculated based on the y-axis vector.

[0070] In practice, each target has a unique ID, which can be a distinguishable identifier such as letters, characters, or numbers, for example... Figure 3 As shown, each target has a unique ID, which is a unique number.

[0071] In practice, the coordinates of the vehicle's center point can be determined based on the three-dimensional coordinates of each target placed at a relevant position on the vehicle's central axis on a target map. For example, using... Figure 2Taking the target placement strategy shown in Figure a as an example, a target is placed at the front and rear of the vehicle body, close to the vehicle's centerline. This allows us to infer the vector representations of the vehicle's center coordinates, lateral and longitudinal coordinates in the QR code map, thus determining the vehicle's center coordinate system. Subsequently, we can calculate the pose transformation from the QR code map coordinate system to the vehicle's center coordinate system. Based on this pose transformation, the QR code map is converted to the vehicle's center coordinate system, enabling the QR code map to be used by the surround-view calibration system.

[0072] Specifically, after constructing the QR code map (i.e., the target map mentioned above), the pixel coordinates of all target corner points are known. Figure 2 Taking the target placement strategy shown in Figure a as an example, the coordinates of the vehicle's center point can be determined based on the center coordinates of the targets positioned close to the front and rear of the vehicle on the QR code map. The center coordinates of the front and rear targets on the QR code map are p and p, respectively. F p R Then the coordinates of the center point of the vehicle body are Let the directions at the front, right, and perpendicular to the ground plane of the vehicle body be the y, x, and z axes of the vehicle center coordinate system, respectively. Then the unit vectors in the x, y, and z directions are x = v(v(p)). F -p R )×(0,0,1) T ), y = v(p F -p R z = v(v(p) F -p R )×(0,0,1) T )×v(p F -p R ), where v(·) denotes the normalization of the vector, that is, the transformation from the QR code map coordinate system to the vehicle center coordinate system: rotation matrix. Translation vector t cm =-R cm O m .

[0073] In specific implementation, after converting the target map (i.e., the coordinate system of the QR code map) to the vehicle center coordinate system, the extrinsic parameters of each vehicle-mounted surround-view camera can be determined based on the position information of the camera positioning target in the images captured by each of the vehicle-mounted surround-view cameras and the position information in the target map. For example, the 2D coordinate information of the corner point of the camera positioning target is identified in the images captured by each of the vehicle-mounted surround-view cameras, and the 3D coordinate information of the corner point of the camera positioning target is obtained according to the converted target map. The 2D coordinate information and the 3D coordinate information of the corner point of the camera positioning target are input into each of the vehicle-mounted surround-view cameras, and each vehicle-mounted surround-view camera obtains its own extrinsic parameters through automatic calibration.

[0074] In specific implementation, before each of the vehicle-mounted surround-view cameras performs automated calibration based on 2D and 3D coordinate information, an omnidirectional model or fisheye model can be used to correct the 2D and 3D coordinate information of the corner points of the camera positioning target. Then, during the automated calibration process, an optimization model that minimizes the reprojection error is established using the corrected 2D and 3D coordinate information. The optimization object is the pose of each of the vehicle-mounted surround-view cameras, and the Levenberg-Marquardt algorithm is used for iterative optimization to solve for the extrinsic parameters of each of the vehicle-mounted surround-view cameras.

[0075] In practice, after obtaining the extrinsic parameters of each of the vehicle-mounted surround-view cameras, the bird's-eye view transformation can be inferred based on these parameters. The images captured by each camera are then converted into top-view images, which are then stitched together to complete the stitching and fusion process. The stitching result is as follows: Figure 4 As shown.

[0076] In practice, the above-mentioned method for calibrating the extrinsic parameters of the vehicle-mounted surround-view camera can be implemented using a portable host computer (PC), a USB low-distortion camera, a set of QR code markers (i.e., the aforementioned targets), a group of vehicle-mounted surround-view cameras (e.g., three or more), and a terminal device. The USB low-distortion camera surrounds the vehicle to collect video of all targets, which is then input into the portable host computer to construct a QR code map. This QR code map is then input into the terminal device, which uses the markers at the relevant positions of the vehicle's centerline on the QR code map to... The target's coordinates determine the vehicle's center coordinate system, and the QR code map is converted to the vehicle's center coordinate system. Finally, for each of the vehicle-mounted surround-view cameras, the 2D coordinate information of the corner points of the camera positioning target is identified in the images captured by each camera. The 3D coordinate information of the corner points of the camera positioning target is obtained based on the converted target map. The 2D and 3D coordinate information of the corner points of the camera positioning target are then input into each of the vehicle-mounted surround-view cameras, enabling each camera to automatically calibrate and obtain its own extrinsic parameters.

[0077] In specific implementation, the following should be combined Figure 5 The process of implementing the above-mentioned calibration method for the extrinsic parameters of the vehicle-mounted surround-view camera includes the following steps:

[0078] 1. Site layout

[0079] (1) Distribute the QR code targets appropriately around the vehicle body. More targets can be placed in the central area of ​​each vehicle surround view camera, and fewer targets can be placed in the edge area. The targets within the field of view of the vehicle surround view camera can be called camera positioning targets;

[0080] (2) Depending on the vehicle body positioning strategy, 2 / 3 of the targets need to be placed in special positions on the vehicle body (i.e., the relevant positions of the vehicle body centerline mentioned above). The targets placed in these positions can be called vehicle body positioning targets. Figure 2 Two strategies for placing vehicle body positioning targets are provided.

[0081] (3) The targets should generally surround the vehicle body, and the distance between any two targets should be controlled within 1 meter.

[0082] 2. Low-distortion camera data acquisition

[0083] Connect a USB low-distortion camera to a portable PC. The USB low-distortion camera will circle the vehicle body to collect images or videos of all targets. It is necessary to capture images of all camera-positioned targets and vehicle-positioned targets.

[0084] 3. Low-distortion image target detection

[0085] The QR code detection algorithm is used to identify QR code targets in each frame of the video and extract the pixel coordinates of the corner points of the QR code targets.

[0086] 4. QR code map construction

[0087] Keyframe images are extracted using the IDs of the identified QR code targets (i.e., at least one identical target exists between consecutive frames). A graph optimization model is established, and the keyframe images are input into the model to optimize all keyframe images. The optimization objects include camera (USB interface low distortion camera) parameters (including intrinsic and extrinsic parameters), the position and pose of the targets in the images (i.e., the transformation matrix that transforms the coordinate systems of each target to the target's coordinate system), γ1,...,γ M ,γ 1 ,...,γ N δ represents the output of the image optimization model in Formula 2. Each target is a directed QR code, i.e., it has a fixed Cartesian coordinate system, and the positions of other targets are represented by one of the targets (i.e., the target mentioned above).

[0088] Considering a target in a single frame, the image optimization model is shown in Equation (1), and the reprojection error is calculated.

[0089]

[0090] c j Given a known quantity, denoted as the three-dimensional coordinates of the lower corner point j of target i in the target coordinate system;

[0091] γ iLet be the quantity to be estimated, and let represent the homogeneous transformation matrix from the target coordinate system to the world coordinate system;

[0092] γ t Let be the quantity to be estimated, and let represent the homogeneous transformation matrix from the world coordinate system to the camera coordinate system, i.e., the camera's extrinsic parameters.

[0093] δ is the quantity to be estimated, representing the intrinsic parameter matrix of the camera.

[0094] function Ψ(δ,γ) t ,γ i ,c j This represents the projection of the corner point j of target i from its 3D spatial coordinates to its pixel coordinates. It first projects the corner point c... j Transform to the world coordinate system, then to the camera coordinate system, and finally to the pixel coordinate system.

[0095] For observation, let represent the actual pixel coordinates of the j-th corner point of target i as observed in frame t.

[0096] This represents the error between the reprojection and observation of target i in frame t.

[0097] Considering all targets in all frames, the image optimization model is shown in Equation (2), which is the model that minimizes the total reprojection error.

[0098]

[0099] For solving the image optimization model, existing graph optimization model techniques were used, which treat each target as a node, calculate the optimal pose transformation between any two nodes, and then find the optimal path that minimizes the sum of the reprojection errors of all edges that make up the path.

[0100] After optimization, the position and attitude information of all targets are obtained, and the coordinate system of each target is transformed into the coordinate system of the target target to generate a target map, also known as a QR code map. The QR code map is then imported into the vehicle terminal device.

[0101] 5. Data acquisition from vehicle-mounted surround view cameras

[0102] Images from four in-vehicle surround-view cameras (four cameras are used as an example) are collected on the vehicle terminal. The images contain QR code targets within the field of view.

[0103] 6. Target detection in panoramic images

[0104] The algorithm uses a QR code detection algorithm to identify QR code targets in images captured by a vehicle-mounted surround-view camera and extracts the 2D coordinate information of the corner points of the QR code targets.

[0105] 7. Vehicle positioning

[0106] This solution uses the vehicle's center coordinate system as a reference for camera extrinsic parameters. Before using the QR code map, the vehicle body needs to be located to determine the vehicle's center coordinate system, and then the QR code map needs to be converted to the vehicle's center coordinate system. Specifically, using... Figure 2 Taking the target placement strategy shown in Figure a as an example, a target is placed at the front and rear of the vehicle body, close to the vehicle's centerline. This allows us to infer the vector representations of the vehicle's center coordinates, lateral and longitudinal coordinates in the QR code map, thus determining the vehicle's center coordinate system. Subsequently, we can calculate the pose transformation from the QR code map coordinate system to the vehicle's center coordinate system. Based on this pose transformation, the QR code map is converted to the vehicle's center coordinate system, enabling the QR code map to be used by the surround-view calibration system.

[0107] 8. Calibration of external parameters of vehicle-mounted surround view camera

[0108] The 3D coordinate information of the corner points of the camera positioning target is obtained based on the converted QR code map. The 2D and 3D coordinate information of the corner points of the camera positioning target are corrected using an omnidirectional model or a fisheye model. Each of the vehicle-mounted surround view cameras is automatically calibrated using the corrected 2D and 3D coordinate information to obtain its own external parameters.

[0109] 9. Top-down view transformation, splicing, etc.

[0110] By using the extrinsic parameters of each vehicle-mounted surround view camera obtained from the solution, the bird's-eye view transformation is inferred, the images captured by each vehicle-mounted surround view camera are converted into top views, and then the top views of each vehicle-mounted surround view camera are stitched together to complete the stitching and fusion work.

[0111] As can be seen, compared with the scheme of using a checkerboard calibration cloth as the target, the above-mentioned vehicle surround view camera extrinsic parameter calibration method uses QR code targets that do not require regular placement or any measurement, which greatly saves time and costs. The placement and measurement of the checkerboard calibration cloth is essentially a manual map creation, but it contains human error that cannot be eliminated and is difficult to optimize. The above-mentioned vehicle surround view camera extrinsic parameter calibration method utilizes the directionality and absolute scale information of QR codes to obtain an optimized QR code map under monocular camera conditions, thus achieving better global accuracy than the latter.

[0112] Compared to existing technologies that use two different sizes of QR codes for the targets, the above-mentioned calibration method for the extrinsic parameters of vehicle surround-view cameras uses a unified QR code size, which is simpler in form. The two-size QR code scheme has a limit on the number of targets, and cannot add more targets to improve calibration accuracy, while the above-mentioned calibration method for the extrinsic parameters of vehicle surround-view cameras does not have this limitation. The two-size QR code scheme improves calibration accuracy by increasing the size of the targets, but the excessively large size is inconvenient for transportation and difficult to implement in practice for technical support services.

[0113] The aforementioned calibration method for the extrinsic parameters of vehicle-mounted surround-view cameras avoids issues such as safety concerns related to lane marking schemes and traffic violation problems. Compared to other high-precision calibration methods, the above calibration method for the extrinsic parameters of vehicle-mounted surround-view cameras, if implemented with the construction of a calibration facility, does not require consideration of target placement accuracy.

[0114] In this embodiment, a computer device is provided, such as... Figure 6 As shown, it includes a memory 601, a processor 602, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned calibration method for any of the extrinsic parameters of the vehicle surround-view camera.

[0115] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0116] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described calibration methods for the extrinsic parameters of a vehicle surround-view camera.

[0117] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0118] Based on the same inventive concept, this invention also provides a calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera, as described in the following embodiments. Since the principle of the calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera is similar to that of the calibration method for the extrinsic parameters of a vehicle-mounted surround-view camera, the implementation of the calibration device can refer to the implementation of the calibration method for the extrinsic parameters of a vehicle-mounted surround-view camera, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0119] Figure 7 This is a structural block diagram of a calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera according to an embodiment of the present invention, such as... Figure 7 As shown, the device includes:

[0120] The video acquisition module 701 is used to capture video of all targets around the vehicle. The targets are coded graphic markers. Multiple targets are placed around the vehicle body, and the targets are placed at relevant positions on the vehicle body's central axis.

[0121] The map generation module 702 is used to generate a target map composed of the targets based on the position information of each target in the video;

[0122] The vehicle body coordinate system generation module 703 is used to determine the vehicle center coordinate system based on the position information of the target placed at the relevant position on the vehicle body centerline in the target map, wherein the vehicle body center point is the origin of the vehicle center coordinate system.

[0123] The coordinate system transformation module 704 is used to transform the target map to the vehicle center coordinate system;

[0124] The calibration module 705 is used to, for each of the vehicle-mounted surround view cameras on the vehicle, regard the targets included in the images captured by each of the vehicle-mounted surround view cameras as camera positioning targets for each of the vehicle-mounted surround view cameras, and determine the extrinsic parameters of each of the vehicle-mounted surround view cameras based on the position of the camera positioning targets in the images captured by each of the vehicle-mounted surround view cameras and the position information in the target map.

[0125] In one embodiment, a map generation module is configured to identify the targets in each frame of the video and extract the pixel coordinates of the corner points of the targets in each frame; determine keyframe images of the video, wherein the keyframe images are two adjacent frames, and the two adjacent frames include at least one identical target; input the keyframe images into a graph optimization model, the graph optimization model outputs a transformation matrix that transforms the coordinate system of each target to the coordinate system of a target target, wherein the target target is one of a plurality of targets, each target has its own coordinate system, the keyframe images include the pixel coordinates of the corner points of the targets, and according to the transformation matrix, the coordinate systems of each target are transformed to the coordinate system of the target target to generate the target map.

[0126] In one embodiment, the vehicle body coordinate system generation module is used to obtain the three-dimensional coordinates of the target placed at the relevant position on the vehicle body centerline in the target map according to the ID of the target placed at the relevant position on the vehicle body centerline; determine the coordinates of the vehicle body center point according to the three-dimensional coordinates; determine the front of the vehicle body as the positive y-axis direction of the vehicle center coordinate system, determine the right side of the vehicle body as the positive x-axis direction of the vehicle center coordinate system, determine the direction perpendicular to the surface where the vehicle is located and upward as the positive z-axis direction of the vehicle center coordinate system, determine the unit vector of the y-axis according to the three-dimensional coordinates, and calculate the unit vector of the x-axis and the unit vector of the z-axis according to the vector of the y-axis.

[0127] In one embodiment, the calibration module is used to identify the 2D coordinate information of the corner points of the camera positioning target in the images acquired by each of the vehicle-mounted surround view cameras, obtain the 3D coordinate information of the corner points of the camera positioning target according to the converted target map, input the 2D coordinate information and the 3D coordinate information of the corner points of the camera positioning target into each of the vehicle-mounted surround view cameras, and each of the vehicle-mounted surround view cameras obtains its own extrinsic parameters through automatic calibration.

[0128] In one embodiment, it also includes:

[0129] The stitching module is used to convert the images captured by each of the vehicle surround view cameras into a top view based on the extrinsic parameters of each camera, and then stitch the top views of each camera together.

[0130] The embodiments of the present invention achieve the following technical effects: they enable the calibration of extrinsic parameters of vehicle-mounted surround-view cameras based on targets. The placement of the targets does not require the layout of a checkerboard or measurement, thus avoiding the introduction of measurement errors. There are no special requirements for the calibration site, thereby improving the flexibility of calibration and also helping to improve the accuracy of calibration results. In addition, there are no requirements for the size of the targets, and large-sized targets are not required, which helps to improve the ease of operation of calibration.

[0131] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for calibrating an external parameter of a surround view camera mounted on a vehicle, characterized in that, include: Video of all targets is captured around the vehicle. The targets are coded graphic markers, and multiple targets are placed around the vehicle's body. Targets are positioned at locations corresponding to the vehicle's central axis, which includes at least two of the following: The front of the vehicle body on the vehicle body centerline, the rear of the vehicle body on the vehicle body centerline, the left parallel line of the vehicle body parallel to the vehicle body centerline, and the right parallel line of the vehicle body parallel to the vehicle body centerline; Based on the location information of each target in the video, a target map composed of each target is generated; Based on the position information of the target placed at a relevant position on the vehicle's central axis in the target map, the vehicle's center coordinate system is determined, and the vehicle's center point is the origin of the vehicle's center coordinate system. Transform the target map to the vehicle center coordinate system; For each of the vehicle-mounted surround view cameras, the target included in the image captured by each of the vehicle-mounted surround view cameras is regarded as the camera positioning target of each of the vehicle-mounted surround view cameras. Based on the position information of the camera positioning target in the image captured by each of the vehicle-mounted surround view cameras and the position information in the target map, the external parameters of each of the vehicle-mounted surround view cameras are determined. Based on the position information of the target placed at a relevant position on the vehicle's centerline in the target map, the vehicle's center coordinate system is determined, including: Based on the ID of the target placed at a relevant position on the vehicle's centerline, obtain the three-dimensional coordinates of the target placed at a relevant position on the vehicle's centerline in the target map; The coordinates of the vehicle's center point are determined based on the three-dimensional coordinates. The front of the vehicle body is defined as the positive y-axis direction of the vehicle center coordinate system, the right side of the vehicle body is defined as the positive x-axis direction of the vehicle center coordinate system, and the direction perpendicular to the ground surface of the vehicle is defined as the positive z-axis direction of the vehicle center coordinate system. The unit vector of the y-axis is determined based on the three-dimensional coordinates, and the unit vectors of the x-axis and z-axis are calculated based on the vector of the y-axis. If the center coordinates of the targets at the front and rear of the vehicle on the vehicle's centerline in the target map are respectively , Then the coordinates of the center point of the vehicle body are Let the directions at the front, right, and perpendicular to the surface where the vehicle is located be the y, x, and z axes of the vehicle center coordinate system, respectively. Then the unit vectors in the x, y, and z directions are: , , ,in, The normalization of the vectors is used to obtain the transformation from the target map coordinate system to the vehicle center coordinate system: rotation matrix. Translation vector . 2.The method of claim 1, wherein, Based on the location information of each target in the video, a target map composed of the targets is generated, including: Identify the target in each frame of the video and extract the pixel coordinates of the corner points of the target in each frame; Keyframe images of the video are determined, wherein the keyframe images are two adjacent images, and the two adjacent images include at least one identical target; The keyframe image is input into a graph optimization model, and the graph optimization model outputs a transformation matrix that transforms the coordinate system of each target to the coordinate system of the target target. The target target is one of the multiple targets, and each target has its own coordinate system. The keyframe image includes the pixel coordinates of the corner points of the target. Based on the transformation matrix, the coordinate system of each target is transformed to the coordinate system of the target target to generate the target map.

3. The calibration method for the extrinsic parameters of a vehicle-mounted surround-view camera as described in any one of claims 1 to 2, characterized in that, Based on the position information of the camera positioning target in the images acquired by each of the vehicle-mounted surround-view cameras and its position information on the target map, the extrinsic parameters of each of the vehicle-mounted surround-view cameras are determined, including: In the images captured by each of the vehicle-mounted surround-view cameras, the 2D coordinate information of the corner points of the camera positioning target is identified. The 3D coordinate information of the corner points of the camera positioning target is obtained according to the converted target map. The 2D coordinate information and the 3D coordinate information of the corner points of the camera positioning target are input into each of the vehicle-mounted surround-view cameras. Each of the vehicle-mounted surround-view cameras obtains its own external parameters through automatic calibration.

4. The method of calibrating vehicle exterior orientation parameters of surround view cameras according to any one of claims 1 to 2, wherein, The target is a QR code marker.

5. The calibration method for the extrinsic parameters of a vehicle-mounted surround-view camera as described in any one of claims 1 to 2, characterized in that, Also includes: Based on the extrinsic parameters of each of the vehicle-mounted surround view cameras, the images captured by each of the vehicle-mounted surround view cameras are converted into top views, and the top views of each of the vehicle-mounted surround view cameras are stitched together.

6. A calibration device for the extrinsic parameters of a vehicle-mounted surround-view camera, characterized in that, include: A video acquisition module is used to capture video of all targets around the vehicle. The targets are coded graphic marker boards, and multiple targets are placed around the vehicle's body. Targets are positioned at locations corresponding to the vehicle's central axis, which includes at least two of the following: The front of the vehicle body on the vehicle body centerline, the rear of the vehicle body on the vehicle body centerline, the left parallel line of the vehicle body parallel to the vehicle body centerline, and the right parallel line of the vehicle body parallel to the vehicle body centerline; The map generation module is used to generate a target map composed of the targets based on the location information of each target in the video; The vehicle body coordinate system generation module is used to determine the vehicle center coordinate system based on the position information of the target placed at a relevant position on the vehicle body centerline in the target map, wherein the vehicle body center point is the origin of the vehicle center coordinate system. A coordinate system transformation module is used to transform the target map to the vehicle center coordinate system; The calibration module is used to, for each of the vehicle-mounted surround view cameras on the vehicle, regard the targets included in the images captured by each of the vehicle-mounted surround view cameras as camera positioning targets for each of the vehicle-mounted surround view cameras, and determine the extrinsic parameters of each of the vehicle-mounted surround view cameras based on the position of the camera positioning targets in the images captured by each of the vehicle-mounted surround view cameras and the position information in the target map. Based on the position information of the target placed at a relevant position on the vehicle's centerline in the target map, the vehicle's center coordinate system is determined, including: Based on the ID of the target placed at a relevant position on the vehicle's centerline, obtain the three-dimensional coordinates of the target placed at a relevant position on the vehicle's centerline in the target map; The coordinates of the vehicle's center point are determined based on the three-dimensional coordinates. The front of the vehicle body is defined as the positive y-axis direction of the vehicle center coordinate system, the right side of the vehicle body is defined as the positive x-axis direction of the vehicle center coordinate system, and the direction perpendicular to the ground surface of the vehicle is defined as the positive z-axis direction of the vehicle center coordinate system. The unit vector of the y-axis is determined based on the three-dimensional coordinates, and the unit vectors of the x-axis and z-axis are calculated based on the vector of the y-axis. If the center coordinates of the targets at the front and rear of the vehicle on the vehicle's centerline in the target map are respectively , Then the coordinates of the center point of the vehicle body are Let the directions at the front, right, and perpendicular to the surface where the vehicle is located be the y, x, and z axes of the vehicle center coordinate system, respectively. Then the unit vectors in the x, y, and z directions are: , , ,in, The normalization of the vectors is used to obtain the transformation from the target map coordinate system to the vehicle center coordinate system: rotation matrix. Translation vector .

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the calibration method for the extrinsic parameters of the vehicle surround-view camera as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the calibration method for the extrinsic parameters of the vehicle surround-view camera according to any one of claims 1 to 5.

Images