Camera calibration method, image stitching method, device, medium, camera and vehicle

Abstract

The present disclosure relates to a camera calibration method, an image stitching method, an apparatus, a medium, a camera, and a vehicle. The calibration method includes: obtaining a first image captured by a camera to be calibrated; the first image has an original image resolution; performing distortion correction on the first image and determining a target area parameter and a target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution; determining a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution; obtaining a second image captured by the camera to be calibrated, and converting reference position information in the second image into target position information corresponding to the target image resolution; and determining an extrinsic parameter of the camera to be calibrated based on the target intrinsic parameter and the target position information. Thus, by setting a larger target image resolution, a larger effective area can be reserved for the camera to be calibrated, thereby facilitating improvement of the stitching blind area problem.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and in particular to a camera calibration method, an image stitching method, a device, a medium, a camera, and a vehicle. Background Art

[0002] With the development of vehicle electronic technology, people have an increasing demand for assisted driving functions. Among them, the vehicle surround view function is a basic function in vehicle assisted driving and can be realized using the vehicle's panoramic surround view system.

[0003] In a vehicle-mounted panoramic surround view system, specific scene calibration patterns are typically used to provide reference points for calibrating the extrinsic parameters (i.e., external parameters) of the system's cameras. This allows for registration of video images captured by four to eight cameras (i.e., cameras) installed around the vehicle body, achieving the goal of panoramic surround view stitching. However, when this vehicle-mounted panoramic surround view system is used in a vehicle with larger dimensions, the limitations of the intrinsic and extrinsic parameters of each camera result in a relatively limited effective area. In this case, the stitched image obtained based on the distortion-corrected image and camera parameters, including both intrinsic and extrinsic parameters, may contain missing information, resulting in blind spots in the resulting stitched image. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a camera calibration method, image stitching method, device, medium, camera and vehicle that can improve the stitching blind area problem.

[0005] The present disclosure provides a camera calibration method, which includes:

[0006] Acquire a first image captured by the camera to be calibrated; the first image has an original image resolution;

[0007] Performing distortion correction on the first image and determining target region parameters and target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution;

[0008] Based on the target area parameters and the target image resolution, determining the target intrinsic parameters of the camera to be calibrated corresponding to the target image resolution;

[0009] Acquire a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to the target image resolution;

[0010] Based on the target intrinsic parameters and target position information, the extrinsic parameters of the camera to be calibrated are determined.

[0011] In some embodiments, before performing distortion correction on the first image and determining target region parameters and target image resolution for distortion correction, the method further includes:

[0012] Get the intrinsic parameters and distortion parameters of the camera to be calibrated;

[0013] The step of performing distortion correction on the first image and determining target area parameters and target image resolution for the distortion correction includes:

[0014] Based on the intrinsic parameters and the distortion parameters, the pixels on the first image are converted into pixels after distortion correction;

[0015] Determining a target area for distortion correction based on a reference area corresponding to the pixel points after distortion correction; the target area is a rectangular area within the reference area defined by the target area parameters;

[0016] Performing distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution;

[0017] The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter, and the first direction intersects the second direction.

[0018] In some embodiments, performing distortion correction on the first image corresponding to the target area includes:

[0019] Using an open image library to perform distortion correction on the first image corresponding to the target area;

[0020] The vertex attributes of the computer graphics corresponding to the first image include vertex coordinates and texture coordinates; the vertex coordinates are (m / W-1.0, n / H-1.0), and the texture coordinates are (x / X, y / Y). The texture coordinates corresponding to the first image are input into a built-in function of the open image library, and the rendered image is used as output, which is the distortion-corrected image.

[0021] Wherein, (x, y) represents any pixel in the first image, (m, n) represents the pixel (x, y) after distortion correction using intrinsic parameters and distortion parameters, X×Y represents the original image resolution, and W and H represent half of the side length of the target area along the first direction and the second direction, respectively.

[0022] In some embodiments, the target image resolution is M×N; the method further includes:

[0023] Adjusting the target image resolution and / or adjusting the target area parameter so that the ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter includes:

[0024] At least one of M, N, W, and H is adjusted so that M:N=W:H.

[0025] In some embodiments, determining a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution includes:

[0026] Based on the target image resolution, determining the target image center point of the camera to be calibrated corresponding to the target image resolution;

[0027] Based on the target area parameters and the target image resolution, combined with the focal length of the camera to be calibrated, the target focal length of the camera to be calibrated corresponding to the target image resolution is determined;

[0028] Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated;

[0029] Where, Fx = fx / (W / M), Fy = fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, and Fx and Fy represent the target focal length of the camera to be calibrated;

[0030] The method also includes:

[0031] Change the distortion parameter to 0.

[0032] In some embodiments, converting the reference position information in the second image into target position information corresponding to the target image resolution includes:

[0033] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the reference position information is corrected for distortion;

[0034] Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution;

[0035] Among them, the target location information is the target location coordinates:

[0036] (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H));

[0037] Wherein, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction.

[0038] In some embodiments, the second image includes a checkerboard, and the reference position information is the coordinates of the corner points of the checkerboard;

[0039] Converting the reference position information in the second image into target position information corresponding to the target image resolution includes:

[0040] identifying coordinates of corner points of the checkerboard in the second image;

[0041] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the coordinates of the corner points are corrected for distortion;

[0042] The coordinates of the corner points after distortion correction are scaled according to the ratio between the target area parameters and the target image resolution.

[0043] In some embodiments, the target region parameter and the target image resolution satisfy the following relationship: M≤W, N≤H.

[0044] The present disclosure also provides a camera calibration device, which includes:

[0045] A first acquisition module is used to acquire a first image captured by the camera to be calibrated; the first image has an original image resolution;

[0046] a distortion correction module, configured to perform distortion correction on the first image and determine target region parameters and target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution;

[0047] An internal parameter determination module is used to determine the target internal parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution;

[0048] a position conversion module, configured to obtain a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to the target image resolution;

[0049] The extrinsic parameter solving module is used to determine the extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and target position information.

[0050] In some embodiments, the apparatus further comprises:

[0051] Auxiliary acquisition module, used to obtain the intrinsic parameters and distortion parameters of the camera to be calibrated;

[0052] The distortion correction module includes:

[0053] A pixel conversion submodule, configured to convert pixels on the first image into distortion-corrected pixels based on an intrinsic parameter and a distortion parameter;

[0054] The region determination submodule is used to determine the target region for distortion correction based on the reference region corresponding to the pixel points after distortion correction; the target region is a rectangular region within the reference region defined by the target region parameters;

[0055] An image correction submodule, configured to perform distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution;

[0056] The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter; and the first direction intersects the second direction.

[0057] In some embodiments, the image correction submodule is used to perform distortion correction on the first image corresponding to the target area, specifically including:

[0058] Using an open image library to perform distortion correction on the first image corresponding to the target area;

[0059] The vertex attributes of the computer graphics corresponding to the first image include vertex coordinates and texture coordinates; the vertex coordinates are (m / W-1.0, n / H-1.0), and the texture coordinates are (x / X, y / Y). The texture coordinates corresponding to the first image are input into a built-in function of the open image library, and the rendered image is used as output, which is the distortion-corrected image.

[0060] Wherein, (x, y) represents any pixel in the first image, (m, n) represents the pixel (x, y) after distortion correction using intrinsic parameters and distortion parameters, X×Y represents the original image resolution, and W and H represent half of the side length of the target area along the first direction and the second direction, respectively.

[0061] In some embodiments, the target image resolution is M×N; the apparatus further comprises:

[0062] The parameter adjustment submodule is used to adjust the target image resolution and / or the target area parameters so that the ratio of the first direction pixel data to the second direction pixel data in the target image resolution is equal to the ratio of the first direction pixel data to the second direction pixel data in the target area parameters, and specifically includes:

[0063] At least one of M, N, W, and H is adjusted so that M:N=W:H.

[0064] In some embodiments, the internal parameter determination module includes:

[0065] A center point determination submodule is used to determine the center point of the target image of the camera to be calibrated corresponding to the target image resolution based on the target image resolution;

[0066] A focal length determination submodule is used to determine the target focal length of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution, combined with the focal length of the camera to be calibrated;

[0067] Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated;

[0068] Where, Fx = fx / (W / M), Fy = fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, and Fx and Fy represent the target focal length of the camera to be calibrated;

[0069] The device also includes:

[0070] The distortion parameter determination submodule is used to change the distortion parameter to 0.

[0071] In some embodiments, the position conversion module is configured to convert the reference position information in the second image into target position information corresponding to the target image resolution, specifically comprising:

[0072] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the reference position information is corrected for distortion;

[0073] Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution;

[0074] Among them, the target location information is the target location coordinates:

[0075] (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H));

[0076] Wherein, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction.

[0077] In some embodiments, the second image includes a checkerboard, and the reference position information is the coordinates of the corner points of the checkerboard;

[0078] The position conversion module is used to convert the reference position information in the second image into target position information corresponding to the target image resolution, specifically including:

[0079] identifying coordinates of corner points of the checkerboard in the second image;

[0080] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the coordinates of the corner points are corrected for distortion;

[0081] The coordinates of the corner points after distortion correction are scaled according to the ratio between the target area parameters and the target image resolution.

[0082] In some embodiments, the target region parameter and the target image resolution satisfy the following relationship: M≤W, N≤H.

[0083] The present disclosure also provides an image stitching method, which includes:

[0084] Acquiring images to be stitched using at least two cameras;

[0085] Calibrate the at least two cameras using any of the above methods;

[0086] Based on the calibrated camera parameters, the images to be stitched are stitched.

[0087] The present disclosure further provides an image stitching device, the image stitching device comprising:

[0088] An image acquisition module, configured to acquire images to be stitched using at least two cameras;

[0089] A camera calibration module, configured to calibrate the at least two cameras using any one of the above methods;

[0090] The image stitching module is used to stitch the images to be stitched based on the calibrated camera parameters.

[0091] The present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of any of the above methods.

[0092] The present disclosure also provides a surround-view camera, which includes a processor and a memory;

[0093] The processor is configured to execute the steps of any of the above-mentioned camera calibration methods by calling the program or instruction stored in the memory to calibrate the camera;

[0094] Alternatively, the processor is configured to execute the steps of any of the above-mentioned image stitching methods by calling the program or instruction stored in the memory to stitch the surround view images.

[0095] The present disclosure also provides a vehicle, which includes any one of the above-mentioned surround-view cameras.

[0096] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0097] The camera calibration method provided by the embodiment of the present disclosure includes: obtaining a first image captured by a camera to be calibrated; the first image has an original image resolution; performing distortion correction on the first image, and determining target area parameters and a target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution; determining target intrinsic parameters of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution; obtaining a second image captured by the camera to be calibrated, and converting reference position information in the second image into target position information corresponding to the target image resolution; and determining extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and the target position information. Among them, the size of the retained effective area after image distortion correction is positively correlated with the size of the target image resolution, that is, the larger the target image resolution, the larger the retained effective area; and the larger the retained effective area, the easier it is for the stitched images to be connected, that is, the easier it is to avoid blind spots in the stitched image when stitching images; based on this, by adjusting the camera intrinsic parameters, that is, determining the target intrinsic parameters based on the target area parameters and the target image resolution, and setting the target image resolution to be greater than the original resolution, that is, increasing the target image resolution of the camera to be calibrated, it is possible to calibrate the external parameters of the camera to be calibrated based on the target intrinsic parameters and target position information corresponding to the larger retained effective area, which is beneficial to reducing image stitching defects and improving the problem of stitching blind spots. Specifically: by setting the resolution of the target image after distortion correction to be greater than the resolution of the original image, a distortion-corrected image with a larger resolution can be obtained, thereby expanding the retained effective area of the distortion-corrected image; based on this, the target intrinsic parameters of the camera to be calibrated are further determined, and combined with the target position information at the target image resolution, the extrinsic parameters of the camera to be calibrated can be calibrated, so that the external parameters with a smaller blind spot (or even no blind spot) around the surround-view stitched vehicle body can be obtained; in this way, the image area corresponding to the image used for stitching can be expanded, which is conducive to reducing image loss, and thus improving or even avoiding the problem of stitching blind spots. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0099] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0100] Figure 1 A schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

[0101] Figure 2A schematic structural diagram of a camera calibration device provided in an embodiment of the present disclosure;

[0102] Figure 3 A flowchart of an image stitching method provided by an embodiment of the present disclosure;

[0103] Figure 4 A schematic structural diagram of an image stitching device provided in an embodiment of the present disclosure;

[0104] Figure 5 A schematic structural diagram of a surround-view camera provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0105] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0106] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0107] In the embodiments of the present disclosure, key terms are first explained.

[0108] The intrinsic parameters of a camera, also known as internal parameters or internal parameters, are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, and other parameters; they are generally calculated and represented in the form of a 3×3 intrinsic parameter matrix.

[0109] The camera's extrinsic parameters, also known as external parameters or external parameters, are parameters in the world coordinate system, such as the camera's position and rotation direction. They are generally calculated and represented in the form of a 4×4 extrinsic parameter matrix.

[0110] Distortion parameter: the coefficient of the conversion relationship between theoretical pixel points and actual pixel points.

[0111] Distortion correction: The process of restoring actual pixels to the positions of theoretical pixels.

[0112] 3D: 3D refers to three dimensions, three coordinates, meaning length, width, and height. 3D is a spatial concept, defined by the three axes X, Y, and Z, as opposed to a flat surface (2D), which only has length and width.

[0113] OpenGL: Open Graphics Library, refers to the open graphics library.

[0114] OpenCV: Open Source Computer Vision Library, a computer vision and machine learning software library.

[0115] The camera calibration method and image stitching method provided in the embodiments of the present disclosure can be applied to a vehicle-mounted panoramic view system, for example, in assisted driving; they can also be applied to other panoramic view scenarios, which are not limited here.

[0116] Taking an on-board panoramic surround view system as an example, during the camera calibration process, the camera extrinsic parameters need to be calibrated. For example, conversion functions (e.g., SolvePnP) in computer vision and machine learning software libraries (e.g., OpenCV) can be used to calibrate the camera extrinsics. Typically, the SolvePnP function in OpenCV cannot directly apply to the raw fisheye camera image to determine the extrinsic parameters; instead, it is necessary to use distortion parameters to remove image distortion before determining the extrinsic parameters. Furthermore, the images of an on-board panoramic surround view system are typically stitched together from images of 4-8 cameras to achieve a surround view effect. Currently, for some large vehicles, if the stitching is still performed using the extrinsic parameters calibrated from the distortion-corrected images as in the related art, the preserved effective area of the corresponding image is limited because the distortion-corrected images in the related art maintain the original resolution (i.e., the original image resolution). In this case, the stitched image obtained based on the distortion-corrected images and the calibrated extrinsic parameters may contain partial missing information, which may result in blind spots in the stitched image used to observe the scene around the vehicle. That is, when the camera is calibrated according to the calibration method in the related art, the obtained camera intrinsic parameters and extrinsic parameters are used for image stitching, but a relatively complete stitched surround view video image cannot be obtained.

[0117] In response to this, the disclosed embodiment provides a camera calibration method, which can achieve calibration of external parameters under a larger retained effective area by adjusting the internal parameters of the camera to be calibrated (hereinafter referred to as "camera") and setting the target image resolution to be greater than the original image resolution, thereby reducing image stitching loss and improving the problem of stitching blind spots. Specifically: by setting the target image resolution after distortion correction to be greater than the original image resolution, an image with a larger image resolution can be obtained. Since the larger the image resolution, the larger the retained effective area, it is equivalent to expanding the retained effective area of the image after distortion correction; based on this, the internal parameters of the camera are modified in combination with the target area parameters and the target image resolution, that is, the target internal parameters are determined, and then the external parameters of the camera are calibrated, so as to obtain the external parameters with a smaller blind area (or even no blind area) around the surround stitching vehicle body. That is, the area corresponding to the image used for stitching can be expanded, thereby reducing image loss and improving the blind area problem.

[0118] The following combination Figure 1-Figure 5, the camera calibration method, camera calibration device, image stitching method, image stitching device, computer-readable storage medium, surround view camera and vehicle provided in the embodiments of the present disclosure are exemplarily described.

[0119] Figure 1 A camera calibration method provided by an embodiment of the present disclosure is shown. Figure 1 As shown, the camera calibration method may include the following steps:

[0120] S101: Obtain a first image captured by a camera to be calibrated.

[0121] The first image is captured by the camera to be calibrated and transmitted to the camera calibration device; correspondingly, the camera calibration device obtains the first image, and the first image has the original image resolution.

[0122] For example, taking the camera to be calibrated as a car-mounted fisheye camera, the first image is distorted, and distortion correction is required in subsequent steps to remove the distortion.

[0123] S102: Perform distortion correction on the first image, and determine target area parameters and target image resolution for the distortion correction.

[0124] The target area parameter is used to define the target area. For example, the target area parameter is the width value of the two intersecting directions representing the target area, which can be expressed as a physical length value or a number of pixels. This will be explained as an example below and is not limited here. Correspondingly, the target area can be a rectangular area defined by the target area parameter.

[0125] The target image resolution is greater than the original image resolution to increase the retained effective area after distortion correction.

[0126] In this step, the intrinsic parameters and distortion parameters of the camera to be calibrated can be used to perform distortion correction on the first image acquired in the aforementioned S101, that is, to obtain an image in the camera coordinate system after the distortion is removed.

[0127] At the same time, the target image resolution is set to be larger than the original image resolution. Image resolution represents the size of an image, and a larger resolution translates to a larger image, meaning a larger effective area is retained in the image after distortion correction. Therefore, setting the target image resolution larger than the original image resolution helps increase the target image resolution and the effective area, thereby reducing image loss and addressing blind spots during image stitching.

[0128] In some embodiments, this step may also include:

[0129] Get the intrinsic parameters and distortion parameters of the camera to be calibrated.

[0130] For example, the intrinsic parameters and distortion parameters of the camera to be calibrated may be built into the camera calibration device and can be directly retrieved in this step.

[0131] Alternatively, the intrinsic parameters and distortion parameters of the camera to be calibrated can be obtained through calibration and called during the extrinsic parameter calibration process.

[0132] Exemplarily, the steps of calibrating the intrinsic parameters and distortion parameters of the camera to be calibrated may include:

[0133] Get the initial image;

[0134] Thereafter, based on the changing relationship between the image coordinate system and the camera coordinate system, any method known to those skilled in the art may be used to solve the intrinsic parameters and distortion parameters of the camera to be calibrated.

[0135] For example, taking the camera to be calibrated as a vehicle-mounted fisheye camera as an example, calibrating the vehicle-mounted fisheye camera includes solving its intrinsic parameters and distortion parameters.

[0136] For example, the intrinsic parameters of the camera to be calibrated may include fx, fy, u0, and v0. fx and fy are the focal lengths of the camera to be calibrated, which can be pixel focal lengths converted from physical length units based on the pixel unit size to maintain consistency with the units of other parameters. u0 and v0 are the pixel coordinates of the center point of the image. Distortion parameters can be calculated by solving a polynomial corresponding to the camera model of the camera to be calibrated. The coefficients of the calculated polynomial may include the corresponding coefficients of different degrees in the polynomial, such as k1, k2, k3, and k4, which are not limited here.

[0137] It is understandable that the initial image obtained in the process of solving the internal parameters and distortion parameters may be the same as or different from the first image in the aforementioned S101, which is not limited here.

[0138] S103 : Determine target intrinsic parameters of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution.

[0139] The target intrinsic parameters are the intrinsic parameters of the camera to be calibrated at the target image resolution, converted based on the inherent intrinsic parameters of the camera to be calibrated, the target region parameters, and the target image resolution. Specifically, in this step, the intrinsic parameters of the camera to be calibrated are adjusted based on the target image resolution and the distortion-corrected target region parameters. This converts the intrinsic parameters at the original image resolution into target intrinsic parameters at the target image resolution. This ensures that the adjusted intrinsic parameters correspond to the size of the retained effective region corresponding to the target image resolution. This allows the extrinsic parameters of the camera to be calibrated to be calculated in the subsequent steps for the larger retained effective region, facilitating more accurate calibration of the extrinsic parameters of the camera to be calibrated.

[0140] S104: Acquire a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to the target image resolution.

[0141] The second image is an image used for extrinsic calibration of the camera to be calibrated.

[0142] For example, when there are at least two cameras to be calibrated, and their images are to be stitched together, this step may include: acquiring images of the calibration object in the same world coordinate system captured by each camera to be calibrated, and converting the coordinates of the reference points in the calibration object to coordinates corresponding to the target image resolution. Acquiring images in the same world coordinate system facilitates registering the at least two cameras to be calibrated to the same world coordinate system, thereby enabling image stitching between the different cameras to be calibrated.

[0143] In this step, by converting the reference position information (for example, the coordinates of the reference point in the calibration object) into the target position information corresponding to the target image resolution (for example, it can be expressed in coordinates), the reference position information in the image coordinate system corresponding to the second image can be converted into position information in the image coordinate system after the distortion is removed, and amplified to obtain the target position information after distortion correction and amplification, so as to facilitate combination with the target intrinsic parameters of the camera to be calibrated determined in the aforementioned S103 to obtain the extrinsic parameters of the camera to be calibrated at the target image resolution, that is, the extrinsic parameters of the camera to be calibrated corresponding to the larger retained effective area, that is, the subsequent S105.

[0144] It can be understood that the second image in this step may be the same as or different from the first image in the aforementioned S101 and the initial image mentioned above, and this is not limited here.

[0145] S105 : Determine the extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and the target position information.

[0146] That is, based on the modified intrinsic parameters of the camera to be calibrated and the target position information corresponding to the target image resolution, the extrinsic parameters of the camera to be calibrated are solved.

[0147] For example, in combination with the above, in this step, the 3D coordinates of the reference point of the calibration object in the world coordinate system are known, the camera intrinsic parameters and distortion parameters after the change are determined in the aforementioned S103, and the coordinates of the corresponding distortion-corrected and enlarged reference point in the image coordinate system are determined in the aforementioned S104. By combining them, the coordinates in the camera coordinate system and the transformation relationship matrix between the world coordinate system and the camera coordinate system can be solved, that is, the camera extrinsic parameters are solved.

[0148] For example, the SolvePNP function in OpenCV can be used to solve the camera extrinsic parameters. For example, for four cameras on a vehicle, the camera extrinsic parameters can be calibrated relative to the same world coordinate system.

[0149] In the camera calibration method provided by the embodiments of the present disclosure, a first image captured by a camera to be calibrated is obtained; the first image has an original image resolution; distortion correction is performed on the first image, and target area parameters and a target image resolution for the distortion correction are determined; wherein the target image resolution is greater than the original image resolution; based on the target area parameters and the target image resolution, target intrinsic parameters of the camera to be calibrated corresponding to the target image resolution are determined; a second image captured by the camera to be calibrated is obtained, and reference position information in the second image is converted into target position information corresponding to the target image resolution; and based on the target intrinsic parameters and the target position information, extrinsic parameters of the camera to be calibrated are determined. Among them, the size of the retained effective area after image distortion correction is positively correlated with the size of the target image resolution, that is, the larger the target image resolution, the larger the retained effective area; and the larger the retained effective area, the easier it is for the stitched images to be connected, that is, the easier it is to avoid blind spots in the stitched image when stitching images; based on this, by adjusting the camera intrinsic parameters, that is, determining the target intrinsic parameters based on the target area parameters and the target image resolution, and setting the target image resolution to be greater than the original resolution, that is, increasing the target image resolution of the camera to be calibrated, it is possible to calibrate the external parameters of the camera to be calibrated based on the target intrinsic parameters and target position information corresponding to the larger retained effective area, which is beneficial to reducing image stitching defects and improving the problem of stitching blind spots. Specifically: by setting the resolution of the target image after distortion correction to be greater than the resolution of the original image, a distortion-corrected image with a larger resolution can be obtained, thereby expanding the retained effective area of the distortion-corrected image; based on this, the target intrinsic parameters of the camera to be calibrated are further determined, and combined with the target position information at the target image resolution, the extrinsic parameters of the camera to be calibrated can be calibrated, so that the external parameters with a smaller blind spot (or even no blind spot) around the surround-view stitched vehicle body can be obtained; in this way, the image area corresponding to the image used for stitching can be expanded, which is conducive to reducing image loss, and thus improving or even avoiding the problem of stitching blind spots.

[0150] The following is an exemplary description of how to implement the steps in the camera calibration method.

[0151] In some embodiments, combined Figure 1 S102 may specifically include:

[0152] Based on the intrinsic parameters and the distortion parameters, the pixels on the first image are converted into pixels after distortion correction;

[0153] Determining a target area for distortion correction based on a reference area corresponding to the pixel points after distortion correction; the target area is a rectangular area within the reference area defined by the target area parameters;

[0154] Performing distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution;

[0155] The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameters, and the first direction intersects the second direction. Optionally, the first direction and the second direction are perpendicular, and are horizontal (e.g., X-axis direction) and vertical (e.g., Y-axis direction), respectively, to define the rectangular target area parameters.

[0156] Furthermore, in the above embodiment, performing distortion correction on the first image may include implementing image distortion correction using OpenGL or other methods known to those skilled in the art. The following is an illustrative description using OpenGL as an example of image distortion correction.

[0157] In some embodiments, “performing distortion correction on the first image corresponding to the target area” may specifically include:

[0158] An open image library (ie, OpenGL) is used to perform distortion correction on the first image corresponding to the target area.

[0159] The vertex attributes of the computer graphics corresponding to the first image include vertex coordinates and texture coordinates; the vertex coordinates are (m / W-1.0, n / H-1.0), and the texture coordinates are (x / X, y / Y). The texture coordinates corresponding to the first image are input into a built-in function of the open image library, and the rendered image is used as output, which is the distortion-corrected image.

[0160] Wherein, (x, y) represents any pixel in the first image, (m, n) represents the pixel (x, y) after distortion correction using the intrinsic parameters and distortion parameters, X×Y represents the original image resolution, W and H represent half of the side length of the target area along the first direction and the second direction, respectively. That is, W and H represent half the length of a single side of the target area along the first direction and the second direction, respectively.

[0161] In the disclosed embodiment, the first image corresponding to the target area is subjected to distortion correction according to the following scheme.

[0162] Here, any pixel point (x, y) on the first image is transformed according to the intrinsic distortion parameters of the camera to be calibrated, resulting in the corresponding pixel point (m, n) after distortion correction. A rectangular region of the original image is selected with the following dimensions: horizontal (xmin, xmax) and vertical (ymin, ymax). The corresponding rectangular region of the distortion-corrected pixel point is then obtained as follows: horizontal (mmin, mmax) and vertical (nmin, nmax). The pixel coordinates of the midline point (u0, v0) on the first image after distortion correction are (U, V). In this case, (U, V) can be used as the center point of the distortion correction. Specifically as follows: the horizontal half-side length of the target area parameter is W, and satisfies W = min (U-mmin, mmax-U), that is, the smaller of the left half-side length and the right half-side length is taken as the horizontal half-side length; the vertical half-side length of the target area parameter is H, and satisfies H = min (V-nmin, nmax-V), that is, the smaller of the upper half-side length and the lower half-side length is taken as the vertical half-side length; based on the determined horizontal half-side length and vertical half-side length, the retained area for distortion correction is determined, and its horizontal pixel area is (UW, U+W), and its vertical pixel area is (VH, V+H). Thus, W and H define the target area, which is a rectangular area. The reference area can be a circular area or polygonal irregular area with the same center as the rectangular area and greater than or equal to the target area, or it can be understood as a circular or polygonal irregular area circumscribed to the target area, which is not limited here.

[0163] Exemplarily, OpenGL is used to perform distortion correction on the first image, where the vertex attributes are vertex coordinates and texture coordinates, the vertex coordinates are (m / W-1.0, n / H-1.0), and the corresponding texture coordinates are (x / X, y / Y), that is, the vertex coordinates are normalized, and the horizontal and vertical coordinates range from -1 to 1; the texture coordinates are normalized, with X and Y as the normalization basis; then, the texture coordinates corresponding to the first image, such as the fisheye video image, are input into the built-in function of the open image library, and the rendered image video is the result of the distortion correction, that is, the distortion-corrected image is obtained.

[0164] In some embodiments, the target image resolution is M×N.

[0165] In the embodiment of the present disclosure, the resolution of the image after distortion correction is set, that is, which parts of the image are retained is determined. For example, the original image resolution is 1280×720, which is recorded as: X×Y; the target image resolution is 2160×1440 or other resolutions larger than 1280×720, which depends on the image stitching effect and can be recorded as M×N.

[0166] It can be understood that the upper limit of the target image resolution is the maximum inscribed rectangle of the image boundary obtained by distortion correction; at the same time, considering factors such as image accuracy, clarity, and data processing speed, the maximum inscribed rectangle can be shrunk, that is, a small rectangle can be taken to ensure image accuracy and clarity while improving blind spots, thereby ensuring better image quality.

[0167] Based on this, the method further includes: adjusting the target image resolution and / or adjusting the target area parameter so that the ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter, and specifically may include:

[0168] At least one of M, N, W, and H is adjusted so that M:N=W:H.

[0169] The target image resolution is M×N. In the disclosed embodiment, M:N=W:H must be ensured to maintain a fixed aspect ratio for the image, thereby maintaining image distortion and ensuring a good image display effect. Specifically, while ensuring that M:N remains unchanged, the value of W or H can be changed; when more image content is retained, the value of W or H can be increased; or while ensuring that W:H remains unchanged, the value of M or N can be changed. That is, as long as the ratio of M to N set in the previous step is inconsistent with the ratio of W to H in this step, at least one of M, N, W, and H can be directly changed to ensure that M:N=W:H. This is not limited here.

[0170] In some embodiments, the target region parameter and the target image resolution satisfy the following relationship: M≤W, N≤H.

[0171] This configuration reduces the amount of data to be processed and increases the speed of data processing, thereby speeding up the calibration of external parameters for the camera being calibrated. When applied to an on-board surround view system, it helps improve the timeliness and real-time nature of capturing the surrounding scene.

[0172] In some embodiments, combined Figure 1 S103 may specifically include:

[0173] Based on the target image resolution, determining the target image center point of the camera to be calibrated corresponding to the target image resolution;

[0174] Based on the target area parameters and the target image resolution, combined with the focal length of the camera to be calibrated, the target focal length of the camera to be calibrated corresponding to the target image resolution is determined;

[0175] Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated;

[0176] Wherein, Fx=fx / (W / M), Fy=fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, and Fx and Fy represent the target focal length of the camera to be calibrated.

[0177] Furthermore, the camera calibration method may further include: changing the distortion parameter to 0.

[0178] In this way, the intrinsic parameters and distortion parameters of the camera to be calibrated are determined, ensuring that the center point of the image remains unchanged and the focal length is proportionally changed; thus, it is suitable for solving the extrinsic parameters of the camera to be calibrated for a larger retained effective area corresponding to the target image resolution.

[0179] In some embodiments, combined Figure 1 The “converting the reference position information in the second image into target position information corresponding to the target image resolution” in S104 may specifically include:

[0180] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the reference position information is corrected for distortion;

[0181] Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution;

[0182] Among them, the target location information is the target location coordinates:

[0183] (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H));

[0184] Wherein, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction.

[0185] Thus, the reference position information is converted into the target position information. For example, the coordinates of the reference point in the calibration object are transformed into coordinates that are distortion corrected and magnified according to the target image resolution, so that the camera's extrinsic parameters can be solved corresponding to a larger retained effective area.

[0186] In some embodiments, the second image includes a checkerboard, and the reference position information is the coordinates of the corner points of the checkerboard;

[0187] Converting the reference position information in the second image into target position information corresponding to the target image resolution includes:

[0188] identifying coordinates of corner points of the checkerboard in the second image;

[0189] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the coordinates of the corner points are corrected for distortion;

[0190] The coordinates of the corner points after distortion correction are scaled according to the ratio between the target area parameters and the target image resolution.

[0191] In the embodiment of the present disclosure, a checkerboard grid may be laid or drawn on the ground (e.g., a plane surrounding the vehicle). A fisheye image including the checkerboard grid is then acquired using a camera to be calibrated, such as a fisheye camera. The fisheye image including the checkerboard grid is processed as follows:

[0192] A. Extract corner coordinates (Xsrc, Ysrc);

[0193] B. Perform distortion correction on the corner point coordinates based on the intrinsic parameters fx, fy, u0, v0 and distortion parameters obtained in step S1011. The coordinates of the corner point after distortion correction are expressed as (Xdst, Ydst);

[0194] C. further transformed into: (M*[Xdst-(UW)] / (2W), N*[Ydst-(VH)] / (2H));

[0195] Thus, the coordinates after distortion correction and magnification are obtained; in the subsequent steps, the extrinsic parameters of the camera can be solved by combining the changed intrinsic parameters and distortion parameters to achieve extrinsic parameter calibration.

[0196] The camera calibration method provided in the embodiments of the present disclosure is applicable to calibrating the extrinsic parameters of a fisheye camera. By retaining a larger effective area during image distortion processing, proportionally transforming the coordinates of reference points in the image based on the target image resolution, and correspondingly changing the camera's intrinsic parameters and distortion parameters, the method can retain a larger effective area than in related techniques for removing fisheye image distortion. Furthermore, by modifying the corresponding formulas for intrinsic parameters, distortion parameters, and pixel point dedistortion, the camera's extrinsic parameters are calculated. This ensures accurate calculation of the camera's extrinsic parameters and minimizes image loss, thereby reducing blind spots in vehicle-mounted surround view image stitching.

[0197] It can be understood that when the camera to be calibrated is a vehicle-mounted camera, the size of the target image resolution corresponding to the retained effective area is related to factors such as the vehicle size, camera position, and camera viewing angle. Qualitatively speaking, the larger the vehicle size, the further outward the camera is relative to the center of the vehicle, and the smaller the camera viewing angle, the larger the effective area should be retained, and the corresponding image resolution after distortion correction should be larger, so as to retain as much of the picture as possible and reduce the stitching blind area.

[0198] The embodiments of the present disclosure further provide a camera calibration device, which can execute the steps of any of the above-mentioned camera calibration methods provided in the embodiments of the present disclosure to achieve corresponding beneficial effects. The similarities can be understood by referring to the explanation of the camera calibration method above, and will not be repeated here.

[0199] In the following, combined Figure 2The camera calibration device provided in the embodiment of the present disclosure is exemplarily described.

[0200] Figure 2 A schematic structural diagram of a camera calibration device provided by an embodiment of the present disclosure is shown.

[0201] like Figure 2 As shown, the camera calibration device 20 may include:

[0202] A first acquisition module 210 is configured to acquire a first image captured by the camera to be calibrated; the first image has an original image resolution;

[0203] The distortion correction module 220 is configured to perform distortion correction on the first image and determine target region parameters and target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution;

[0204] An intrinsic parameter determination module 230 is configured to determine a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution;

[0205] A position conversion module 240 is configured to obtain a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to a target image resolution;

[0206] The extrinsic parameter solving module 250 is used to determine the extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and target position information.

[0207] In the camera calibration device 20 provided in the embodiment of the present disclosure, through the collaboration between the above-mentioned functional modules, a first image captured by the camera to be calibrated can be obtained; the first image has the original image resolution; distortion correction is performed on the first image, and the target area parameters and target image resolution of the distortion correction are determined; wherein the target image resolution is greater than the original image resolution; based on the target area parameters and the target image resolution, the target intrinsic parameters of the camera to be calibrated corresponding to the target image resolution are determined; a second image captured by the camera to be calibrated is obtained, and the reference position information in the second image is converted into target position information corresponding to the target image resolution; based on the target intrinsic parameters and the target position information, the extrinsic parameters of the camera to be calibrated are determined. Among them, the size of the retained effective area after image distortion correction is positively correlated with the size of the target image resolution, that is, the larger the target image resolution, the larger the retained effective area; and the larger the retained effective area, the easier it is for the stitched images to be connected, that is, the easier it is to avoid blind spots in the stitched image when stitching images; based on this, by adjusting the camera intrinsic parameters, that is, determining the target intrinsic parameters based on the target area parameters and the target image resolution, and setting the target image resolution to be greater than the original resolution, that is, increasing the target image resolution of the camera to be calibrated, it is possible to calibrate the external parameters of the camera to be calibrated based on the target intrinsic parameters and target position information corresponding to the larger retained effective area, which is beneficial to reducing image stitching defects and improving the problem of stitching blind spots. Specifically: by setting the resolution of the target image after distortion correction to be greater than the resolution of the original image, a distortion-corrected image with a larger resolution can be obtained, thereby expanding the retained effective area of the distortion-corrected image; based on this, the target intrinsic parameters of the camera to be calibrated are further determined, and combined with the target position information at the target image resolution, the extrinsic parameters of the camera to be calibrated can be calibrated, so that the external parameters with a smaller blind spot (or even no blind spot) around the surround-view stitched vehicle body can be obtained; in this way, the image area corresponding to the image used for stitching can be expanded, which is conducive to reducing image loss, and thus improving or even avoiding the problem of stitching blind spots.

[0208] In some embodiments, the apparatus further comprises:

[0209] Auxiliary acquisition module, used to obtain the intrinsic parameters and distortion parameters of the camera to be calibrated;

[0210] The distortion correction module 220 includes:

[0211] A pixel conversion submodule, configured to convert pixels on the first image into distortion-corrected pixels based on an intrinsic parameter and a distortion parameter;

[0212] The region determination submodule is used to determine the target region for distortion correction based on the reference region corresponding to the pixel points after distortion correction; the target region is a rectangular region within the reference region defined by the target region parameters;

[0213] An image correction submodule, configured to perform distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution;

[0214] The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter; and the first direction intersects the second direction.

[0215] In some embodiments, the image correction submodule is used to perform distortion correction on the first image corresponding to the target area, specifically including:

[0216] Using an open image library to perform distortion correction on the first image corresponding to the target area;

[0217] The vertex attributes of the computer graphics corresponding to the first image include vertex coordinates and texture coordinates; the vertex coordinates are (m / W-1.0, n / H-1.0), and the texture coordinates are (x / X, y / Y). The texture coordinates corresponding to the first image are input into a built-in function of the open image library, and the rendered image is used as output, which is the distortion-corrected image.

[0218] Wherein, (x, y) represents any pixel in the first image, (m, n) represents the pixel (x, y) after distortion correction using intrinsic parameters and distortion parameters, X×Y represents the original image resolution, and W and H represent half of the side length of the target area along the first direction and the second direction, respectively.

[0219] In some embodiments, the target image resolution is M×N; the apparatus further comprises:

[0220] The parameter adjustment submodule is used to adjust the target image resolution and / or the target area parameters so that the ratio of the first direction pixel data to the second direction pixel data in the target image resolution is equal to the ratio of the first direction pixel data to the second direction pixel data in the target area parameters, and specifically includes:

[0221] At least one of M, N, W, and H is adjusted so that M:N=W:H.

[0222] In some embodiments, the internal parameter determination module 230 includes:

[0223] A center point determination submodule is used to determine the center point of the target image of the camera to be calibrated corresponding to the target image resolution based on the target image resolution;

[0224] A focal length determination submodule is used to determine the target focal length of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution, combined with the focal length of the camera to be calibrated;

[0225] Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated;

[0226] Where, Fx = fx / (W / M), Fy = fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, and Fx and Fy represent the target focal length of the camera to be calibrated;

[0227] The device also includes:

[0228] The distortion parameter determination submodule is used to change the distortion parameter to 0.

[0229] In some embodiments, the position conversion module 240 is configured to convert the reference position information in the second image into target position information corresponding to the target image resolution, specifically including:

[0230] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the reference position information is corrected for distortion;

[0231] Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution;

[0232] Among them, the target location information is the target location coordinates:

[0233] (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H));

[0234] Wherein, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction.

[0235] In some embodiments, the second image includes a checkerboard, and the reference position information is the coordinates of the corner points of the checkerboard;

[0236] The position conversion module 240 is used to convert the reference position information in the second image into target position information corresponding to the target image resolution, specifically including:

[0237] identifying coordinates of corner points of the checkerboard in the second image;

[0238] Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the coordinates of the corner points are corrected for distortion;

[0239] The coordinates of the corner points after distortion correction are scaled according to the ratio between the target area parameters and the target image resolution.

[0240] In some embodiments, the target region parameter and the target image resolution satisfy the following relationship: M≤W, N≤H.

[0241] It should be noted that Figure 2 The camera calibration device 20 shown can perform Figure 1 The various steps in the method embodiment shown are implemented Figure 1 The various processes and effects in the illustrated method embodiment are not described in detail here.

[0242] On the basis of the above-mentioned embodiments, the embodiments of the present disclosure further provide an image stitching method, which includes any one of the camera calibration methods provided in the above-mentioned embodiments, and can reduce stitching blind areas, thereby improving the image stitching effect.

[0243] In the following, combined Figure 3 The image stitching method provided in the embodiment of the present disclosure is exemplified.

[0244] Figure 3 A flow chart of an image stitching method provided by an embodiment of the present disclosure is shown.

[0245] like Figure 3 As shown, the image stitching method may include the following steps:

[0246] S301: Acquire images to be stitched using at least two cameras.

[0247] S302: Calibrate at least two cameras using the steps of a camera calibration method.

[0248] In the embodiment of the present disclosure, the camera calibration method may adopt any of the camera calibration methods in the above-mentioned implementation manners.

[0249] S303: stitching the images to be stitched based on the calibrated camera parameters.

[0250] In the image stitching method provided in the embodiments of the present disclosure, any one of the camera calibration methods in the above-mentioned embodiments is adopted. By adjusting the camera intrinsic parameters and increasing the target image resolution, it is possible to calibrate the external parameters under a larger retained effective area, thereby reducing image stitching defects and improving the problem of stitching blind spots.

[0251] On the basis of the above-mentioned embodiments, the embodiments of the present disclosure further provide an image stitching device, which can execute the steps of any image stitching method provided in the embodiments of the present disclosure to achieve corresponding beneficial effects. The similarities can be understood by referring to the explanation of the image stitching method above, and will not be repeated here.

[0252] In the following, combined Figure 4 The image stitching device provided in the embodiment of the present disclosure is exemplarily described.

[0253] Figure 4A structural schematic diagram of an image stitching device provided by an embodiment of the present disclosure is shown.

[0254] like Figure 4 As shown, the image stitching device 40 may include:

[0255] An image acquisition module 410 is configured to acquire images to be stitched using at least two cameras;

[0256] A camera calibration module 420 is configured to calibrate at least two cameras using any of the above methods;

[0257] The image stitching module 430 is configured to stitch the images to be stitched based on the calibrated camera parameters.

[0258] In the image stitching device 40 provided in the embodiment of the present disclosure, since its functional modules can implement any of the camera calibration methods in the above-mentioned embodiments, by adjusting the camera intrinsic parameters and increasing the target image resolution, it is possible to calibrate the external parameters under a larger retained effective area, thereby reducing image stitching defects and improving the problem of stitching blind spots.

[0259] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores programs or instructions. The programs or instructions enable a computer to execute the steps of any of the above methods to achieve corresponding beneficial effects. To avoid repeated description, they will not be repeated here.

[0260] The disclosed embodiments further provide a surround-view camera comprising a processor and a memory. The processor is configured to execute the steps of any of the aforementioned camera calibration methods by invoking a program or instruction stored in the memory to calibrate the camera. Alternatively, the processor is configured to execute the steps of any of the aforementioned image stitching methods by invoking a program or instruction stored in the memory to stitch surround-view images. By adjusting the camera's intrinsic parameters and increasing the target image resolution, calibration of extrinsic parameters within a larger retained effective area can be achieved, thereby reducing image stitching defects and addressing stitching blind spots.

[0261] Exemplarily, the surround-view camera may be a vehicle-mounted surround-view camera.

[0262] For example, Figure 5 A schematic structural diagram of a vehicle-mounted surround-view camera provided by an embodiment of the present disclosure is shown.

[0263] like Figure 5As shown, the surround view camera may include: at least one processor 501, at least one memory 502 and at least one communication interface 503. The various components in the surround view camera are coupled together through a bus system 504. The communication interface 503 is used to transmit information with external devices. It can be understood that the bus system 504 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0264] It can be understood that the memory 502 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0265] In some embodiments, the memory 502 stores the following elements, executable units or data structures, or a subset or an extended set thereof: an operating system and application programs.

[0266] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and handle hardware-based tasks. Application programs include various application programs, such as media players and browsers, which are used to implement various application services. Programs that implement the methods provided in the embodiments of the present disclosure may be included in the application programs.

[0267] In the embodiment of the present disclosure, the processor 501 calls the program or instructions stored in the memory 502, specifically, the program or instructions stored in the application, and the processor 501 is used to execute the steps of each embodiment of the method provided in the embodiment of the present disclosure.

[0268] The method provided in the embodiment of the present disclosure can be applied to the processor 501, or implemented by the processor 501. The processor 501 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instructions in the form of software. The above-mentioned processor 501 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0269] The steps of the method provided in the embodiments of the present disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 502, and processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the method.

[0270] The disclosed embodiments further provide a vehicle including any of the aforementioned surround-view cameras. The surround-view camera can be used in an on-board panoramic view system to improve blind spots in image stitching, resulting in better image stitching. This enables the on-board panoramic view system to more accurately identify the environment around the vehicle, providing more accurate image information for autonomous or assisted driving.

[0271] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0272] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A camera calibration method, characterized in that: include: Obtain a first image captured by the camera to be calibrated; The first image has an original image resolution; Performing distortion correction on the first image, and determining target area parameters and target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution; Determining a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution; Acquire a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to the resolution of the target image; Determining the extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and the target position information; The target image resolution is M×N; Determining a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution includes: Based on the target image resolution, determining a target image center point of the camera to be calibrated corresponding to the target image resolution; Determining a target focal length of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution and in combination with the focal length of the camera to be calibrated; Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated; Wherein, Fx=fx / (W / M), Fy=fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, W and H represent half of the side length of the target area along the first direction and the second direction, respectively; Fx and Fy represent the target focal length of the camera to be calibrated, respectively; Converting the reference position information in the second image into target position information corresponding to the resolution of the target image includes: Performing distortion correction on the reference position information using the intrinsic parameters and distortion parameters of the camera to be calibrated; Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution; Among them, the target location information is the target location coordinates: (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H)); Among them, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction; (U, V) represents the pixel coordinates corresponding to the center point pixel coordinates (u0, v0) on the first image after distortion correction.

2. The method according to claim 1, characterized in that Before performing distortion correction on the first image and determining target area parameters and target image resolution for the distortion correction, the method further includes: Obtaining the intrinsic parameters and distortion parameters of the camera to be calibrated; The step of performing distortion correction on the first image and determining target area parameters and target image resolution for the distortion correction includes: Based on the intrinsic parameter and the distortion parameter, converting the pixel points on the first image into pixel points after distortion correction; Determining a target area for distortion correction based on a reference area corresponding to the pixel point after distortion correction; the target area is a rectangular area within the reference area defined by the target area parameters; Performing distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution; The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter, and the first direction intersects with the second direction.

3. The method according to claim 2, characterized in that Performing distortion correction on the first image corresponding to the target area includes: Performing distortion correction on the first image corresponding to the target area using an open image library; The vertex attributes of the computer graphics corresponding to the first image include vertex coordinates and texture coordinates; the vertex coordinates are (m / W-1.0, n / H-1.0), and the texture coordinates are (x / X, y / Y). The texture coordinates corresponding to the first image are input into the built-in function of the open image library, and the rendered image is used as output, which is the distortion-corrected image. Wherein, (x, y) represents any pixel in the first image, (m, n) represents the pixel after the distortion correction of the pixel (x, y) using the intrinsic parameter and the distortion parameter, and X×Y represents the original image resolution.

4. The method according to claim 3, characterized in that The method further comprises: Adjusting the target image resolution and / or adjusting the target area parameter so that a ratio of pixel data in the first direction to pixel data in the second direction in the target image resolution is equal to a ratio of pixel data in the first direction to pixel data in the second direction in the target area parameter includes: At least one of M, N, W, and H is adjusted so that M:N=W:H.

5. The method according to claim 4, characterized in that The method further comprises: Change the distortion parameter to 0.

6. The method according to claim 4, characterized in that The second image includes a checkerboard, and the reference position information is the coordinates of the corner points of the checkerboard; Converting the reference position information in the second image into target position information corresponding to the resolution of the target image includes: identifying coordinates of corner points of the checkerboard in the second image; Performing distortion correction on the coordinates of the corner points using the intrinsic parameters and distortion parameters of the camera to be calibrated; The coordinates of the corner points after distortion correction are scaled according to a ratio between the target area parameters and the target image resolution.

7. The method according to any one of claims 4 to 6, characterized in that The target area parameter and the target image resolution satisfy the following conditions: M≤W, N≤H.

8. A camera calibration device, characterized in that: include: A first acquisition module, configured to acquire a first image captured by the camera to be calibrated; The first image has an original image resolution; a distortion correction module, configured to perform distortion correction on the first image and determine target area parameters and target image resolution for the distortion correction; wherein the target image resolution is greater than the original image resolution; an intrinsic parameter determination module, configured to determine a target intrinsic parameter of the camera to be calibrated corresponding to the target image resolution based on the target area parameter and the target image resolution; a position conversion module, configured to obtain a second image captured by the camera to be calibrated, and convert reference position information in the second image into target position information corresponding to the resolution of the target image; an extrinsic parameter solving module, configured to determine the extrinsic parameters of the camera to be calibrated based on the target intrinsic parameters and the target position information; The target image resolution is M×N; The internal parameter determination module includes: A center point determination submodule is used to determine the center point of the target image of the camera to be calibrated corresponding to the target image resolution based on the target image resolution; A focal length determination submodule is used to determine the target focal length of the camera to be calibrated corresponding to the target image resolution based on the target area parameters and the target image resolution, combined with the focal length of the camera to be calibrated; Where U0=M / 2, V0=N / 2; (U0, V0) represents the pixel coordinates of the center point of the target image of the camera to be calibrated; Wherein, Fx=fx / (W / M), Fy=fy / (H / N); fx and fy represent the focal length of the camera to be calibrated, W and H represent half of the side length of the target area along the first direction and the second direction, respectively; Fx and Fy represent the target focal length of the camera to be calibrated, respectively; The position conversion module is used to convert the reference position information in the second image into target position information corresponding to the target image resolution, specifically including: Using the intrinsic parameters and distortion parameters of the camera to be calibrated, the reference position information is corrected for distortion; Converting the distortion-corrected reference position information into target position information corresponding to the target image resolution; Among them, the target location information is the target location coordinates: (M×[Xdst-(UW)] / (2W), N×[Ydst-(VH)] / (2H)); Among them, (Xdst, Ydst) represents the coordinates corresponding to the reference position information after distortion correction; (U, V) represents the pixel coordinates corresponding to the center point pixel coordinates (u0, v0) on the first image after distortion correction.

9. The device according to claim 8, characterized in that Also includes: An auxiliary acquisition module, used to obtain the intrinsic parameters and distortion parameters of the camera to be calibrated; The distortion correction module includes: a pixel conversion submodule, configured to convert pixels on the first image into distortion-corrected pixels based on the intrinsic parameter and the distortion parameter; a region determination submodule, configured to determine a target region for distortion correction based on a reference region corresponding to the pixel points after distortion correction; the target region is a rectangular region within the reference region defined by the target region parameters; an image correction submodule, configured to perform distortion correction on the first image corresponding to the target area to obtain a distortion-corrected image and a target image resolution; The ratio of the pixel data in the first direction to the pixel data in the second direction in the target image resolution is equal to the ratio of the pixel data in the first direction to the pixel data in the second direction in the target area parameter; and the first direction intersects the second direction.

10. An image stitching method, characterized in that: include: Acquiring images to be stitched using at least two cameras; Calibrate the at least two cameras using the steps of the method according to any one of claims 1 to 7; The images to be stitched are stitched based on the calibrated camera parameters.

11. An image stitching device, characterized in that: include: An image acquisition module, configured to acquire images to be stitched using at least two cameras; a camera calibration module, configured to calibrate the at least two cameras using the steps of the method according to any one of claims 1 to 7; The image stitching module is used to stitch the images to be stitched based on the calibrated camera parameters.

12. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores a program or instruction, which enables a computer to execute the steps of the method according to any one of claims 1 to 7.

13. A surround view camera, characterized in that: including processor and memory; The processor is configured to execute the steps of the method according to any one of claims 1 to 7 by calling the program or instruction stored in the memory to calibrate the camera; Alternatively, the processor is configured to execute the steps of the method according to claim 10 by calling the program or instruction stored in the memory to stitch the surround view images.

14. A vehicle, characterized in that: Including the surround-view camera as described in claim 13.

Images