Camera external parameter calibration method and system, computer equipment and medium

Abstract

The invention provides a camera external parameter calibration method and system, computer equipment and a medium, and the method comprises the steps: obtaining a target image, obtaining an image containing a preset region from the target image, and taking the image as a region-of-interest image; performing corner detection on the region-of-interest image to obtain pixel coordinates of corners in the preset region in the region-of-interest image; based on the pixel coordinates of the angular points in the preset region in the region-of-interest image and the pixel coordinates of the region-of-interest image in the target image, determining the pixel coordinates of the angular points in the preset region in the target image; and performing external parameter calibration on the target camera according to the pixel coordinates of the angular points in the preset area in the target image. When the target camera after external parameter calibration is used for generating the panoramic image picture, the problem that the panoramic image picture cannot be generated or the effect of the generated panoramic image picture is poor due to the fact that the brightness difference exists when the target camera carries out image shooting caused by the external environment can be relieved or solved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a camera extrinsic parameter calibration method and system, computer equipment and medium. Background Technology

[0002] In some vehicle-assisted driving scenarios, AVM (Around View Monitor) is used to generate a panoramic image of the vehicle's surroundings, allowing drivers and passengers to understand the surrounding environment and eliminate blind spots. However, during AVM use, the camera's extrinsic parameters can change due to external environmental factors (such as lighting intensity and road surface smoothness), resulting in either the inability to stitch together a panoramic image or a poor-quality image, thus affecting the user's driving experience. Summary of the Invention

[0003] This application provides a camera extrinsic parameter calibration method and system, computer equipment and medium to solve or alleviate the problems described above.

[0004] This application provides a camera extrinsic parameter calibration method, comprising the following steps: acquiring a target image, and obtaining an image containing a preset region from the target image as a region of interest (ROI) image; wherein the target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on a vehicle; performing corner detection on the ROI image to obtain the pixel coordinates of the corner points in the preset region within the ROI image; determining the pixel coordinates of the corner points in the preset region within the ROI image based on the pixel coordinates of the corner points in the preset region within the ROI image and the pixel coordinates of the ROI image in the target image; and calibrating the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the preset region within the target image.

[0005] In one embodiment of this application, if the preset area includes a checkerboard pattern area, the process of calibrating the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the preset area in the target image includes: using the checkerboard pattern in the checkerboard pattern area as a target image, and performing orientation and distance calibration on the target image to obtain a calibration target image; calculating the physical coordinates and pixel coordinates of all target points based on the physical coordinates of the origin in the world coordinate system and the pixel coordinates of the origin in the target image to obtain the calculated physical coordinates and calculated pixel coordinates of each target point; wherein, the target point is based on the checkerboard pattern. The corner points in the pattern area are determined; based on the calculated physical coordinates and calculated pixel coordinates of all target points, the target camera is calibrated for the first time, which serves as the initial camera extrinsic parameters of the target camera; a bird's-eye view is formed by projecting the initial camera extrinsic parameters and camera intrinsic parameters of the target camera; the corner points in the bird's-eye view are identified, and the identified corner points are back-projected to obtain the pixel coordinates of the corner points in the target image; based on the calculated physical coordinates of all target points and the pixel coordinates of the corner points in the bird's-eye view in the target image, the target camera is calibrated for the second time, which serves as the final camera extrinsic parameters of the target camera.

[0006] In one embodiment of this application, the method further includes: determining whether the vehicle is on a smooth road surface based on the vehicle's inertial navigation data; generating a panoramic image based on the extrinsic parameter calibration result of the target camera and the target image when the vehicle is on a smooth road surface; transforming the extrinsic parameter calibration result of the target camera on a smooth road surface based on the inertial navigation data to obtain the camera extrinsic parameters of the current road surface based on the camera extrinsic parameters of the current road surface and the target image to generate a panoramic image corresponding to the current road surface.

[0007] In one embodiment of this application, the method further includes: detecting the ambient brightness value of the target image, and comparing the average brightness of any two pixels selected from the target image; if the ambient brightness value of the target image is within a preset brightness range, and the average brightness of the two selected pixels is less than a preset brightness threshold, then calibrating the target camera based on the pixel coordinates of the corner points in the preset region in the target image; if the ambient brightness value of the target image is not within the preset brightness range, and / or the average brightness of the two selected pixels is greater than or equal to the preset brightness threshold, then acquiring a new target image.

[0008] In one embodiment of this application, the process of performing corner detection on the region of interest image to obtain the pixel coordinates of the corner points in the preset region within the region of interest image includes: performing grayscale processing on the region of interest image, and using a corner detection model to perform corner detection and recognition on the grayscale processed region of interest image to obtain the pixel coordinates of the corner points in the preset region within the region of interest image.

[0009] In one embodiment of this application, the training process of the corner detection model includes: acquiring a training sample set, which is obtained based on a back-projected target image with completed camera intrinsic and extrinsic parameter calibration and a target image with target coordinates corresponding to the back-projected target image; inputting the training set into a deep learning model architecture for training to obtain an initial model; using the initial model to filter the validation sample set, and adding the filtered predicted corners and samples with a distance greater than a preset distance to the training sample set, updating the training sample set to obtain an updated training sample set; iteratively training the initial model using the updated training sample set, adjusting the preset distance during the iterative training process, and using the model trained based on the last preset distance adjustment as the corner detection model.

[0010] In one embodiment of this application, the method further includes: acquiring a camera intrinsic parameter calibration image, wherein the camera intrinsic parameter calibration image is obtained by the target camera capturing the target image at different positions and angles; performing checkerboard corner point position detection on the camera intrinsic parameter calibration image using a computer vision library; and calculating the camera intrinsic parameters of the target camera based on the checkerboard corner point position detection results.

[0011] This application also provides a camera extrinsic calibration system, the system comprising: a data acquisition module for acquiring a target image and obtaining an image containing a preset region from the target image as a region of interest (ROI) image; wherein the target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on a vehicle; a corner detection module for performing corner detection on the ROI image to obtain the pixel coordinates of the corners in the preset region within the ROI image; a coordinate fusion module for determining the pixel coordinates of the corners in the preset region within the ROI image based on the pixel coordinates of the corners in the preset region within the ROI image and the pixel coordinates of the ROI image in the target image; and a camera extrinsic calibration module for performing extrinsic calibration on the target camera based on the pixel coordinates of the corners in the preset region within the target image.

[0012] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the camera extrinsic parameter calibration method described in any one of the above.

[0013] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the camera extrinsic parameter calibration method described in any one of the above.

[0014] The beneficial effects of this application are as follows: This application proposes a camera extrinsic parameter calibration method and system, computer equipment and medium. It involves acquiring a target image and obtaining an image containing a preset region from the target image as a region of interest (ROI) image. The target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on a vehicle. Corner detection is then performed on the ROI image to obtain the pixel coordinates of the corner points in the preset region within the ROI image. Based on the pixel coordinates of the corner points in the preset region within the ROI image and the pixel coordinates of the ROI image in the target image, the pixel coordinates of the corner points in the preset region in the target image are determined. Extrinsic parameter calibration of the target camera is then performed based on the pixel coordinates of the corner points in the preset region in the target image. Therefore, this application can obtain the pixel coordinates of the corner points in the preset area in the target image by fusing the pixel coordinates of the corner points in the preset area in the region of interest image and the pixel coordinates of the region of interest image in the target image. Then, based on the pixel coordinates of the corner points in the preset area in the target image, the target camera is calibrated using extrinsic parameters. When generating a panoramic image using the target camera after extrinsic parameter calibration, the problem of brightness differences in the target camera during image capture caused by external environment (such as light intensity, road surface smoothness, etc.) can be alleviated or solved, which makes it impossible to generate a panoramic image or the generated panoramic image has poor effect, thereby further improving the user's driving experience. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] In the attached diagram:

[0017] Figure 1 A schematic diagram of a scene where a target image is included in front of a vehicle, as captured by a fisheye camera on a vehicle. Figure 2A schematic diagram of a scene where a target image is included behind a vehicle, as captured by a fisheye camera on the vehicle. Figure 3 A schematic diagram of a scene where the target image is included on the right side of a vehicle, as captured by a fisheye camera on the vehicle. Figure 4 A schematic diagram of a scene where the left side of a vehicle contains a target image, as captured by a fisheye camera on a vehicle. Figure 5 This is a schematic flowchart of a camera extrinsic parameter calibration method provided in one embodiment of this application; Figure 6 A target image provided in one embodiment of this application; Figure 7 This is a calibration target diagram provided in one embodiment of this application; Figure 8 This is a schematic diagram of the vehicle coordinate system during vehicle calibration provided in one embodiment of this application; Figure 9 A schematic flowchart illustrating a camera extrinsic parameter calibration method provided in another embodiment of this application; Figure 10 This is a schematic diagram of the training of a deep learning model for target detection in a target image provided in one embodiment of this application; Figure 11 This is a schematic diagram of a target image after distortion correction provided in an embodiment of this application; Figure 12 A schematic diagram showing the pixel coordinates of a region of interest image in a target image according to an embodiment of this application; Figure 13 This is a schematic diagram of an image of a region of interest after grayscale processing, provided in an embodiment of this application. Figure 14 This is a schematic diagram of the corner pixel coordinates within a region of interest image provided in one embodiment of this application; Figure 15 A schematic diagram showing the pixel coordinates of a corner point in a preset area in a target image, as provided in an embodiment of this application. Figure 16 This is a flowchart illustrating the process of obtaining the pixel coordinates of a corner point in a preset area in a target image, as provided in one embodiment of this application. Figure 17 A schematic diagram of corner pixel coordinates calculated by orientation provided in an embodiment of this application; Figure 18 A bird's-eye view provided for one embodiment of this application; Figure 19 This is a schematic diagram of the training of a deep learning model for target detection in a target image provided in an embodiment of this application. Figure 20 A schematic diagram of corner point recognition results from a bird's-eye view provided in an embodiment of this application; Figure 21 A schematic diagram of the pixel coordinates of a corner point in a target image in a bird's-eye view provided in an embodiment of this application; Figure 22 A schematic flowchart illustrating a camera extrinsic parameter calibration method provided in yet another embodiment of this application; Figure 23 This is a schematic diagram of the hardware structure of a camera extrinsic parameter calibration system provided in one embodiment of this application; Figure 24 This is a schematic diagram of the hardware structure of a computer device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] It is understood that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0021] Using a black and white checkerboard pattern as the target image, image data containing the black and white checkerboard pattern is collected by four fisheye cameras on the vehicle, such as... Figures 1 to 4 As shown. From Figures 1 to 4As can be seen, the calibration cloth with its black and white checkerboard pattern is clearly visible in the camera image. However, the presence of other dark ambient points around the vehicle makes it impossible to correctly locate the corner points between the black and white checkerboards. Furthermore, when using the traditional Zhang Zhengyou calibration algorithm for automatic AVM camera extrinsic parameter calibration, calibration will fail when the ambient brightness of the image data is not between 300 Lux and 1000 Lux, and the average brightness of any two pixels in the image data is greater than or equal to 100 Lux.

[0022] Based on the above description Figure 5 A flowchart illustrating a camera extrinsic parameter calibration method is shown. Specifically, in an exemplary embodiment, as follows... Figure 5 As shown, this embodiment provides a camera extrinsic parameter calibration method, including the following steps: S510, acquire the target image, and obtain an image containing a preset region from the target image as the Region of Interest (ROI) image; wherein, the target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on the vehicle. In some examples, an image region containing the preset region can be cropped from the target image as the ROI image. In some examples, the preset region includes, but is not limited to, a checkerboard pattern region. In some examples, the target camera includes, but is not limited to, a fisheye camera or a fisheye lens camera. In some examples, the target image captured by a fisheye camera can be as follows: Figure 6 As shown. In some examples, if the preset region includes a checkerboard pattern region, the checkerboard pattern within that region can be used as the target image. In this case, the target image can be input into a deep learning model for target image detection to detect the target image. The training process of the deep learning model for target image detection can be as follows: Figure 10 As shown. In some examples, the RGB image generated by the i-th (e.g., i=1, 2, 3, 4) fisheye camera in the vehicle AVM can be used as the target image. For example, if the target camera is the first fisheye camera in a vehicle AVM, and a frame of RGB image captured by this fisheye camera contains a portion of a 9×6 checkerboard pattern, then this portion of the checkerboard pattern can be used as a preset region, and the corresponding image region can be used as the region of interest image.

[0023] S520: Perform corner detection on the region of interest image to obtain the pixel coordinates of corner points within the preset region of interest image. In some examples, a corner point can be an extreme point, the intersection of two lines, or a point on different objects in two adjacent directions. For example, a corner point can be the boundary between black and white squares in a checkerboard pattern.

[0024] S530, Based on the pixel coordinates of the corner points in the preset region within the region of interest image and the pixel coordinates of the region of interest image in the target image, determine the pixel coordinates of the corner points in the preset region within the target image; S540 performs extrinsic parameter calibration on the target camera based on the pixel coordinates of corner points in the preset area within the target image.

[0025] In an exemplary embodiment, if the preset area in step S540 includes a checkerboard pattern area, the process of calibrating the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the preset area in the target image may include: using the checkerboard pattern in the checkerboard pattern area as a target image, and performing orientation and distance calibration on the target image to obtain a calibration target image; calculating the physical coordinates and pixel coordinates of all target points based on the physical coordinates of the origin in the world coordinate system and the pixel coordinates of the origin in the target image to obtain the calculated physical coordinates and calculated pixel coordinates of each target point; and then, based on the... The target camera undergoes its first extrinsic parameter calibration using the calculated physical coordinates of the target points and the pixel coordinates of corner points in a preset area within the target image. This serves as the initial camera extrinsic parameters. Next, a bird's-eye view is projected from the initial extrinsic and intrinsic camera parameters. Corner points in the bird's-eye view are identified, and their pixel coordinates are back-projected to obtain their pixel coordinates within the target image. Finally, the target camera undergoes a second extrinsic parameter calibration using the calculated physical coordinates of all target points and the pixel coordinates of the corner points in the bird's-eye view. This serves as the final camera extrinsic parameters. The target points can be determined based on corner points within a checkerboard pattern area; for example, some or all corner points of the checkerboard pattern can be selected as target points. In some examples, this method can be applied to both smooth and uneven road surface scenarios.

[0026] In some examples, a 9×6 checkerboard pattern (each square has a side length of 200mm) is laid on a flat indoor calibration workshop floor as a target image. This target image includes (9-1)×(6-1)=40 corner points. Simultaneously, the target image covers the front and right sides of the vehicle and is located within the field of view of the right front fisheye camera. Specifically, the direction and distance calibration process for this target image may include: using a laser rangefinder and a level, defining the first interior corner point of the top left corner of the checkerboard pattern as the origin of the world coordinate system, where the physical coordinates of the origin can be denoted as Ow=(0, 0, 0). If the positive X-axis of the world coordinate system points to the left side of the vehicle, the positive Y-axis points to the vehicle's forward direction, and the positive Z-axis points vertically upwards, then the target image can be calibrated using the X, Y, and Z axes of the world coordinate system. Simultaneously, distance calibration can be performed based on the physical coordinates of the origin of the world coordinate system and the side length of each square, thus obtaining the physical coordinates of the 40 checkerboard corner points in the world coordinate system. Then, based on the direction and distance calibration results, the final calibrated target image can be obtained, denoted as the calibrated target image. In some examples, the calibrated target image can be as follows: Figure 7 As shown, Figure 7 The side length of each square is in mm.

[0027] In some examples, if the target camera is a fisheye camera in a vehicle, and a frame of RGB image captured by the fisheye camera contains a portion of a 9×6 checkerboard pattern, then this portion of the checkerboard pattern can be used as a preset region, and the corresponding image region can be used as the region of interest (ROI). Furthermore, if a pre-trained corner detection model detects 25 checkerboard corners in this portion of the checkerboard pattern from the ROI image, the initial extrinsic parameter calibration process for the target camera can include: treating each of these 25 corners as a target point; then, based on the physical coordinates of the origin of the 9×6 checkerboard pattern in the world coordinate system, calculating the physical coordinates of these 25 corners to obtain their calculated physical coordinates; and finally, based on the pixel coordinates of the origin of the 9×6 checkerboard pattern in the RGB image frame, calibrating the physical coordinates of these 25 corners. The pixel coordinates are calculated to obtain the calculated pixel coordinates of the 25 chessboard corner points. Then, the calculated physical coordinates and calculated pixel coordinates of each chessboard corner point are associated to obtain 25 coordinate pairs. Combined with the camera intrinsic parameters of the fisheye camera (such as focal length, principal point, distortion coefficient, etc.), the PnP (Perspective-n-Point) algorithm is used to solve the camera pose of the fisheye camera, that is, the initial position and orientation relative to the world coordinate system. It can be represented by rotation matrix R1 and translation vector T1. Finally, rotation matrix R1 and translation vector T1 are used as the initial camera extrinsic parameters of the fisheye camera.

[0028] In some examples, if the target camera is a fisheye camera in a vehicle, the process of projecting a bird's-eye view based on the initial camera extrinsic and intrinsic parameters of the target camera can include: using the initial camera extrinsic parameters (R1, T1) and the camera intrinsic parameters of the fisheye camera, projecting the original RGB image captured by the fisheye camera onto the ground (i.e., Z=0 in the world coordinate system) to generate a bird's-eye view. In this bird's-eye view, the originally curved checkerboard lines become approximately straight lines.

[0029] In some examples, if the target camera is a fisheye camera in a vehicle, the process of identifying corner points in the bird's-eye view and back-projecting the identified corner points to obtain the pixel coordinates of the corner points in the target image may include: detecting checkerboard corner points in the bird's-eye view using a pre-trained corner detection model, and then back-projecting (i.e., inverse perspective transformation) the detected checkerboard corner points to obtain more accurate pixel coordinates of the corner points in the original image.

[0030] In some examples, if the target camera is a fisheye camera in a vehicle, the process of performing a second extrinsic parameter calibration on the target camera, based on the estimated physical coordinates of all target points and the pixel coordinates of corner points in the bird's-eye view in the target image, includes: pairing the estimated physical coordinates of all target points and the pixel coordinates of corner points in the original image from the bird's-eye view; and based on the newly paired coordinate pairs, performing PnP or nonlinear optimization (e.g., Levenberg-Marquardt) again to solve for the new camera extrinsic parameters (R2, T2) of the fisheye camera. Then, this new camera extrinsic parameter (R2, T2) is used as the final camera extrinsic parameter of the fisheye camera. The error of the final camera extrinsic parameter compared to the initial camera extrinsic parameter can be reduced from the centimeter level to the millimeter level.

[0031] In an exemplary embodiment, after obtaining the extrinsic parameter calibration result of the target camera, the process may further include: determining whether the vehicle is on a smooth road surface based on the vehicle's inertial navigation data; when the vehicle is on a smooth road surface, generating a panoramic image based on the extrinsic parameter calibration result of the target camera and the target image; when the vehicle is not on a smooth road surface, transforming the extrinsic parameter calibration result of the target camera on a smooth road surface based on the inertial navigation data to obtain the camera extrinsic parameters of the current road surface; and generating a panoramic image corresponding to the current road surface based on the camera extrinsic parameters of the current road surface and the target image. In some examples, on uneven road surfaces, since the vehicle's own inertial navigation data can record the angle of the vehicle going uphill and downhill in real time, this angle can be used to transform the camera extrinsic parameter calibration result on a smooth road surface to obtain the camera extrinsic parameter calibration result on a non-smooth road surface, and then generating a panoramic image on a non-smooth road surface based on the camera extrinsic parameter calibration result on a non-smooth road surface. Specifically, when a vehicle is calibrated, its vehicle coordinate system is as follows: Figure 8As shown, when the vehicle is going up or down a slope, the actual angle of the Z-axis changes vertically. Figure 8 In this context, worldHeight represents the vehicle's height in the world coordinate system, and worldWidth represents the vehicle's width in the world coordinate system. Let the Z-axis rotation angle for the vehicle's inertial navigation data transmission be θ. (Unit: radians), the rotation matrix is: The extrinsic parameter matrix A of the calibration camera on a perfectly flat road surface is then based on the Z-axis rotation angle obtained from the vehicle's inertial navigation data transmission. The updated calibration camera extrinsic parameter matrix B is obtained as follows: Therefore, this method can generate panoramic images corresponding to uneven road surfaces based on the vehicle's inertial navigation data, thus solving or alleviating the problem of stitching together the vehicle's surround view images on uneven road surfaces such as uphill and downhill sections.

[0032] In an exemplary embodiment, the camera extrinsic calibration method may further include: detecting the ambient brightness value of the target image, and comparing the average brightness of two randomly selected pixels in the target image; if the ambient brightness value of the target image is within a preset brightness range, and the average brightness of the two randomly selected pixels is less than a preset brightness threshold, then calibrating the target camera's extrinsic parameters based on the pixel coordinates of corner points in the preset region in the target image; if the ambient brightness value of the target image is not within the preset brightness range, and / or the average brightness of the two randomly selected pixels is greater than or equal to the preset brightness threshold, then acquiring a new target image. The preset brightness range and preset brightness threshold can be selected or set according to actual conditions, and no specific numerical limit is specified here. For example, the preset brightness range can be 300 Lux-1000 Lux, and the preset brightness threshold can be 100 Lux. Specifically, in some examples, for instance, ambient brightness value detection is performed on target image P1 to determine whether the ambient brightness value of target image P1 is within 300 Lux-1000 Lux; and two pixels are randomly selected from target image P1, denoted as pixel A and pixel B, and their average brightness values ​​are compared; if the ambient brightness value of target image P1 is within 300 Lux-1000 Lux, and the average brightness values ​​of pixel A and pixel B are both less than 100 Lux, then the target camera's extrinsic parameters are calibrated. Conversely, if the ambient brightness value of target image P1 is not within 300 Lux-1000 Lux, and / or the average brightness values ​​of pixel A and pixel B are greater than or equal to 100 Lux, then a new target image is acquired, which can be denoted as P2. Then, the processing flow for target image P2 is the same as that for target image P1, which will not be elaborated further here. Therefore, in some extrinsic parameter calibration scenarios, we can select images from the target camera's captured images where the ambient brightness value is within a preset brightness range, and the average brightness of any two selected pixels is less than a preset brightness threshold, to perform extrinsic parameter calibration on the target camera, thereby improving the efficiency and accuracy of extrinsic parameter calibration. For example, when the ambient brightness value of the target image is within 300 Lux-1000 Lux, and the average brightness of any two selected pixels is less than 100 Lux, the target camera's extrinsic parameters can be calibrated based on the coordinates of corner points in the preset area within the target image. In some examples, Zhang Zhengyou's calibration algorithm can also be used for camera extrinsic parameter calibration.

[0033] In an exemplary embodiment, the process of performing corner detection on a region of interest (ROI) image to obtain the pixel coordinates of corner points within a preset region in the ROI image includes: converting the ROI image to grayscale, and using a pre-trained or real-time trained corner detection model to perform corner detection and recognition on the grayscale-converted ROI image to obtain the pixel coordinates of corner points within the ROI image. The grayscale conversion process may include merging the red (R), green (G), and blue (B) channel values ​​of each pixel in the ROI image into a single grayscale value, thereby simplifying the image from three-channel color data to single-channel grayscale data, reducing computational complexity and storage overhead. Furthermore, by performing corner detection and recognition on the grayscale-converted ROI image, it is convenient to analyze the spatial changes in pixel grayscale. In some examples, the training process of a corner detection model may include: acquiring a training sample set, which is obtained based on a back-projected target image and a target image with target point coordinates corresponding to the back-projected target image after camera intrinsic and extrinsic parameter calibration; inputting the training set into a pre-determined or real-time determined deep learning model architecture for training to obtain an initial model; using the initial model to filter the validation sample set, and adding the filtered samples with predicted corners and sample label distances greater than a preset distance to the training sample set, updating the training sample set to obtain an updated training sample set; iteratively training the initial model using the updated training sample set, adjusting the preset distance during the iterative training process, and using the model trained based on the last preset distance adjustment as the corner detection model. The validation sample set can be a target image containing target points, such as the back-projected target image and the target image with target point coordinates corresponding to the back-projected target image in the training sample set.

[0034] In some examples, the aforementioned deep learning model architectures include, but are not limited to, the MobileNet-v4 model architecture. In some examples, since grayscale conversion of images may result in grayscale images and potentially lose image information, color images can be directly trained to generate corner detection models for corner detection. In some examples, the training sample set can be collected from fisheye camera calibration extrinsic images calibrated by the camera manufacturer. Since the fisheye camera intrinsics are calibrated by default by the manufacturer, the calibration accuracy is high. Calibrating the camera extrinsic parameters based on these intrinsics allows for high-precision calibration. Based on the high-precision extrinsic and intrinsic parameters, each target point can be back-projected onto the corresponding image, facilitating training.

[0035] As an example, the training process of a corner detection model may include: selecting multiple (e.g., 5000) back-projected target images and corresponding target pixel coordinate images based on high-precision camera intrinsic and extrinsic calibration data as an initial training set, and training an initial model using the initial training set. Then, using the initial model, multiple (e.g., 1 million) datasets are selected: multiple (e.g., 3000) samples whose predicted point-to-label distance is greater than a set value (e.g., 10) are added to the initial training set, and training is iterated repeatedly. Samples are then added to the initial training set using pre-set or real-time thresholds (e.g., 5, 4, 3, 2, 1). Finally, a stable version of the training set is trained, and the trained model is used as the corner detection model.

[0036] In an exemplary embodiment, the camera extrinsic calibration method may further include: acquiring a camera intrinsic calibration image, wherein the camera intrinsic calibration image is obtained by capturing a target image from different positions and angles using a target camera; detecting the corner positions of the camera intrinsic calibration image using a computer vision library; and calculating the camera intrinsic parameters of the target camera based on the corner position detection results.

[0037] Specifically, a fisheye camera can be used to photograph a black and white checkerboard pattern, obtaining a set of camera intrinsic calibration images. These images should include checkerboard images from different positions and angles. Then, the `findChessboardCorners` function from the OpenCV computer vision library is used to detect the position of each checkerboard corner (the boundary between black and white squares). The `calibrateCamera` function from OpenCV is used to calculate the fisheye camera's intrinsic parameters and distortion coefficients. In some examples, the transformation relationship between the camera coordinate system and the image coordinate system can also be referred to as the camera intrinsic parameter matrix.

[0038] In an exemplary embodiment of this application, as Figure 9 As shown, a method for calibrating camera extrinsic parameters is provided, including the following steps: Step 1: Initial Camera Extrinsic Parameter Calibration. The RGB image generated by the i-th (e.g., i=1, 2, 3, 4) fisheye camera in the vehicle AVM is used as the target image and input into the target image object detection deep learning model to detect the target image. The training process of the target image object detection deep learning model can be as follows: Figure 10 As shown. Figure 10The dataset can collect extrinsic images of fisheye cameras calibrated by the camera manufacturer. In vehicle intelligent driving calibration, the intrinsic parameters of the fisheye camera are calibrated by default by the manufacturer, resulting in high calibration accuracy. Calibrating the camera extrinsic parameters based on these intrinsic parameters allows for high-precision calibration of the extrinsic parameters. Based on these high-precision extrinsic and intrinsic parameters, along with the target image, each target point can be back-projected onto its corresponding image. As an example, an RGB image generated by a fisheye camera can be displayed as follows: Figure 6 As shown, based on the fisheye camera intrinsic parameter model, Figure 6 The RGB image shown is distorted to obtain the distorted RGB image, as shown below. Figure 11 As shown. The distortion-free RGB image is input into the target image detection deep learning model to detect the target image, and the target image detection box is obtained. Based on the detection box, the target image ROI image is obtained, and the pixel coordinates of the ROI image in the original RGB image are recorded and output, denoted as coordinate 1, as shown. Figure 12 As shown. The target ROI image can also be denoted as the region of interest image, and coordinate 1 can be denoted as the pixel coordinate of the region of interest image within the target image. The target ROI image is then converted to grayscale, as shown below. Figure 13 As shown. The grayscale image of the target ROI can also be denoted as the grayscale image of the region of interest. The OpenCV interface is used to perform computer vision corner detection and recognition on the grayscale image of the target ROI, obtaining the pixel coordinates of the corner points in the target image within the target ROI image, denoted as coordinate 2, as shown. Figure 14 As shown. Coordinate 2 can also be represented as the pixel coordinates of a corner point within the preset region in the region of interest (ROI) image. Merging coordinates 1 and 2 yields the pixel coordinates of the corner point in the target ROI image within the original RGB image, as shown below. Figure 15As shown. The pixel coordinates of corner points in the target ROI image within the original RGB image can also be represented as the pixel coordinates of corner points in the preset area within the target image. Then, using the given keypoint (e.g., origin) pixel coordinates and the black-and-white checkerboard pattern in the target ROI image, the pixel coordinates of the calibration points of the single-channel fisheye camera in the vehicle coordinate system are calculated. Finally, the initial camera extrinsic parameters of the fisheye camera are calculated based on the pixel coordinates of the calibration points in the single-channel fisheye camera. The calibration points are obtained based on corner points or target points. For example, the initial camera extrinsic parameters of the fisheye camera can be calculated based on the calculated physical coordinates of all target points and the pixel coordinates of corner points in the target image within the original RGB image. In some examples, the calibration point coordinate pairs can consist of calibration points in the camera coordinate system and calibration points in the vehicle coordinate system. In another example, using a black and white checkerboard calibration map as the target image, and denoting the algorithm for detecting the target image based on a deep learning model for target image detection as the target detection algorithm, the process of obtaining the pixel coordinates of the corner points in the ROI of the black and white checkerboard calibration map in the original RGB image by combining the target detection algorithm with the Zhang Zhengyou calibration algorithm is as follows: Figure 16 As shown. Since the checkerboard pattern on the target image is known before calibration, as... Figure 17 As shown, the top left corner of the target image can be set as the origin of the coordinate system. Then, based on the direction and distance calibration results, the target point coordinates between each black block can be calculated. After obtaining the initial camera extrinsic parameters in the two examples above, the initial camera extrinsic parameters and camera intrinsic parameters are then used to project and generate a bird's-eye view, as shown... Figure 18 As shown in the diagram. When using the OpenCV interface to perform computer vision corner detection and recognition on a grayscale target image (ROI), the grayscale image may lose information due to the resulting grayscale value. Therefore, a deep learning model for target detection on a color image can also be directly trained for target detection. The training process of the deep learning model for target detection is as follows: Figure 19 As shown, the process includes: selecting multiple (e.g., 5000) back-projected target images and corresponding target pixel coordinates based on high-precision camera intrinsic and extrinsic calibration data as the initial training set, and training an initial model using the initial training set. Then, using the initial model, multiple (e.g., 1 million) datasets are selected: multiple (e.g., 3000) samples whose predicted point-to-label distance is greater than a set value (e.g., 10) are added to the initial training set, and training is iteratively repeated. Samples are added to the initial training set using pre-set or real-time thresholds (e.g., 5, 4, 3, 2, 1). Finally, the final version of the training set is trained using the MobileNet-v4 model to obtain the target detection deep learning model. This target detection deep learning model is also referred to as the corner detection model.

[0039] Step 2: Secondary camera extrinsic parameter calibration. The calibration points in the projected bird's-eye view are identified, where the calibration points are obtained based on corner points or target points. For example... Figure 20 As shown, identify the large squares on the left and right sides of the image. Figure 20 The red-marked locations in the image can be recorded as calibration points. Then, back-projection is performed on the identified calibration points to obtain their pixel coordinates in the original RGB image within the bird's-eye view. Next, a sub-pixel-level corner detection function is used to improve the accuracy of these pixel coordinates, obtaining their pixel coordinates in the camera coordinate system. Figure 21 As shown. Subpixel level can be half, one-third, one-quarter, one-fiftieth, one-hundredth, etc., of a normal pixel. Finally, the final camera extrinsic parameters are obtained based on the pixel coordinates of the calibration point coordinate pairs of the single-channel fisheye camera. For example, the final camera extrinsic parameters can be obtained based on the calculated physical coordinates of all target points and the pixel coordinates of corner points in the original RGB image from the bird's-eye view. In some examples, the calibration point coordinate pairs can consist of calibration points in the camera coordinate system and calibration points in the vehicle coordinate system.

[0040] exist Figure 9 In the image, the front image represents the original image taken by the fisheye camera directly in front of the vehicle, the rear image represents the original image taken by the fisheye camera directly behind the vehicle, the left image represents the original image taken by the fisheye camera on the left side of the vehicle, and the right image represents the original image taken by the fisheye camera on the right side of the vehicle.

[0041] The ROI image front image represents the ROI image corresponding to the fisheye camera directly in front of the vehicle, the ROI image rear image represents the ROI image corresponding to the fisheye camera directly behind the vehicle, the ROI image left image represents the ROI image corresponding to the fisheye camera on the left side of the vehicle, and the ROI image right image represents the ROI image corresponding to the fisheye camera on the right side of the vehicle.

[0042] The front points of the ROI map represent the corner pixel coordinates within the ROI corresponding to the fisheye camera directly in front of the vehicle; the rear points of the ROI map represent the corner pixel coordinates within the ROI corresponding to the fisheye camera directly behind the vehicle; the left points of the ROI map represent the corner pixel coordinates within the ROI corresponding to the fisheye camera on the left side of the vehicle; and the right points of the ROI map represent the corner pixel coordinates within the ROI corresponding to the fisheye camera on the right side of the vehicle.

[0043] The fisheye camera front points represent the pixel coordinates of the corner points within the ROI corresponding to the fisheye camera directly in front of the vehicle in the original image; the fisheye camera rear points represent the pixel coordinates of the corner points within the ROI corresponding to the fisheye camera directly behind the vehicle in the original image; the fisheye camera left points represent the pixel coordinates of the corner points within the ROI corresponding to the fisheye camera on the left side of the vehicle in the original image; and the fisheye camera right points represent the pixel coordinates of the corner points within the ROI corresponding to the fisheye camera on the right side of the vehicle in the original image.

[0044] The calibration target image front points represent the calibration point pixel coordinates or target point pixel coordinates corresponding to the fisheye camera directly in front of the vehicle; the calibration target image rear points represent the calibration point pixel coordinates or target point pixel coordinates corresponding to the fisheye camera directly behind the vehicle; the calibration target image left points represent the calibration point pixel coordinates or target point pixel coordinates corresponding to the fisheye camera on the left side of the vehicle; and the calibration target image right points represent the calibration point pixel coordinates or target point pixel coordinates corresponding to the fisheye camera on the right side of the vehicle.

[0045] The front points of the AVM image represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera directly in front of the vehicle; the rear points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera directly behind the vehicle; the left points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera on the left side of the vehicle; and the right points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera on the right side of the vehicle.

[0046] In the bird's-eye view, front points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera directly in front of the vehicle; rear points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera directly behind the vehicle; left points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera on the left side of the vehicle; and right points represent the pixel coordinates of the calibration point or target point corresponding to the fisheye camera on the right side of the vehicle.

[0047] Regarding the issue of AVM (Aspect-to-Vehicle Monitoring) display deviation when vehicles are going up or down slopes (mountain roads or entering underground parking garages), this problem arises because the camera's extrinsic parameters are based on those obtained on a flat road surface during production line calibration. When the vehicle is on a slope, the camera's extrinsic parameters become invalid, resulting in abnormal AVM display. Therefore, to address this issue, in an exemplary embodiment of this application, as follows... Figure 22As shown, a method for generating graphic images is provided, including the following steps: based on the intrinsic parameter data of the fisheye camera, the image captured by the fisheye camera is distorted to obtain a distorted image; the extrinsic parameters of the target camera under the smooth road surface are denoted as extrinsic parameter data 1, and then the extrinsic parameter data 1 is transformed based on inertial navigation data to obtain the extrinsic parameter data of the current road surface, denoted as extrinsic parameter data 2; a bird's-eye view is generated based on the distorted image and extrinsic parameter data 2, and then the bird's-eye view images of all fisheye cameras in the vehicle are stitched together to form an AVM surround view stitched image.

[0048] exist Figure 22 In the image, the fisheye camera front image represents the original image captured by the fisheye camera directly in front of the vehicle, the fisheye camera rear image represents the original image captured by the fisheye camera directly behind the vehicle, the fisheye camera left image represents the original image captured by the fisheye camera on the left side of the vehicle, and the fisheye camera right image represents the original image captured by the fisheye camera on the right side of the vehicle.

[0049] The front image represents the distorted image corresponding to the fisheye camera directly in front of the vehicle, the rear image represents the distorted image corresponding to the fisheye camera directly behind the vehicle, the left image represents the distorted image corresponding to the fisheye camera on the left side of the vehicle, and the right image represents the distorted image corresponding to the fisheye camera on the right side of the vehicle.

[0050] The front image represents the bird's-eye view image corresponding to the fisheye camera directly in front of the vehicle; the rear image represents the bird's-eye view image corresponding to the fisheye camera directly behind the vehicle; the left image represents the bird's-eye view image corresponding to the fisheye camera on the left side of the vehicle; and the right image represents the bird's-eye view image corresponding to the fisheye camera on the right side of the vehicle.

[0051] In summary, this application proposes a camera extrinsic calibration method. This method involves acquiring a target image and obtaining an image containing a preset region from the target image, which serves as the region of interest (ROI). The target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on the vehicle. Corner detection is then performed on the ROI image to obtain the pixel coordinates of the corners within the preset region. Based on the pixel coordinates of the corners within the ROI image and the pixel coordinates of the ROI image in the target image, the pixel coordinates of the corners in the preset region are determined in the target image. Finally, the extrinsic parameters of the target camera are calibrated according to the pixel coordinates of the corners in the preset region in the target image. Therefore, this method can obtain the pixel coordinates of the corner points in the preset area in the target image by fusing the pixel coordinates of the corner points in the preset area in the region of interest image and the pixel coordinates of the region of interest image in the target image. Then, based on the pixel coordinates of the corner points in the preset area in the target image, the target camera is calibrated using extrinsic parameters. When generating a panoramic image using the target camera after extrinsic parameter calibration, it can alleviate or solve the problem that external environmental factors (such as light intensity, road surface smoothness, etc.) cause brightness differences in the target camera when taking pictures, resulting in the inability to generate a panoramic image or the generation of a poor panoramic image, thereby further improving the user's driving experience.

[0052] In another exemplary embodiment of this application, such as Figure 23 As shown, a camera extrinsic parameter calibration system is also provided, including: The data acquisition module 2310 is used to acquire a target image and obtain an image containing a preset region from the target image as a region of interest image; wherein, the target image is obtained by taking a picture of the preset region outside the vehicle by a target camera set on the vehicle; The corner detection module 2320 is used to perform corner detection on the region of interest image and obtain the pixel coordinates of the corners in the preset region within the region of interest image. The coordinate fusion module 2330 is used to determine the pixel coordinates of the corner points in the preset area in the target image based on the pixel coordinates of the corner points in the preset area in the region of interest and the pixel coordinates of the region of interest image in the target image. The camera extrinsic calibration module 2340 is used to calibrate the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the preset area in the target image.

[0053] It is understood that the camera extrinsic calibration system and the camera extrinsic calibration method provided in the above embodiments belong to the same concept. The specific execution method of the camera extrinsic calibration method has been described in detail in the above method embodiments and will not be repeated here. In practical applications, the camera extrinsic calibration system provided in the above embodiments can be assigned to different functional modules as needed. That is, the internal structure of the camera extrinsic calibration system can be divided into different functional modules, and then all or part of the functions of the corresponding functional modules can be implemented by the camera extrinsic calibration method described in the above embodiments. For example, all or part of the functions of the data acquisition module 2310 can be implemented by the relevant execution process of step S510, all or part of the functions of the corner detection module 2320 can be implemented by the relevant execution process of step S520, all or part of the functions of the coordinate fusion module 2330 can be implemented by the relevant execution process of step S530, and all or part of the functions of the camera extrinsic calibration module 2340 can be implemented by the relevant execution process of step S540. No specific limitations are imposed here.

[0054] In summary, this application proposes a camera extrinsic calibration system. This system acquires a target image and extracts an image containing a preset region from the target image, which is then used as a region of interest (ROI) image. The target image is obtained by capturing the preset region outside the vehicle using a target camera mounted on the vehicle. Corner detection is then performed on the ROI image to obtain the pixel coordinates of the corners within the preset region. Based on the pixel coordinates of the corners within the ROI image and the pixel coordinates of the ROI image in the target image, the pixel coordinates of the corners in the preset region are determined in the target image. Finally, the extrinsic parameters of the target camera are calibrated according to the pixel coordinates of the corners in the preset region in the target image. Therefore, this system can obtain the pixel coordinates of the corner points in the preset area in the target image by fusing the pixel coordinates of the corner points in the preset area within the region of interest image and the pixel coordinates of the region of interest image in the target image. Then, based on the pixel coordinates of the corner points in the preset area in the target image, the target camera is calibrated using extrinsic parameters. When generating panoramic images using the target camera after extrinsic parameter calibration, this system can alleviate or solve the problem that external environmental factors (such as light intensity and road surface smoothness) cause brightness differences in the target camera during image capture, resulting in the inability to generate panoramic images or generating poor-quality panoramic images, thereby further improving the user's driving experience.

[0055] In an exemplary embodiment of the present invention, a computer device is also provided. The computer device may include a memory, a processor, and a computer program stored in the memory. The processor can execute the computer program to cause the computer device to perform actions such as... Figure 5 or Figure 9 or Figure 22 The steps of the camera extrinsic parameter calibration method are shown. Figure 24 A schematic diagram of the structure of a computer device 1000 is shown. (See attached diagram.) Figure 24 As shown, the computer device 1000 includes: a processor 1010, a memory 1020, a power supply 1030, a display unit 1040, and an input unit 1060.

[0056] The processor 1010 is the control center of the computer device 1000. It connects various components via interfaces and lines, and performs various functions of the computer device 1000 by running or executing computer programs / instructions stored in the memory 1020, thereby providing overall monitoring of the computer device 1000. In some embodiments, when the processor 1010 calls a computer program stored in the memory 1020, it can execute, for example... Figure 5 or Figure 9 or Figure 22 The steps of the camera extrinsic parameter calibration method are shown. Optionally, the processor 1010 may include one or more processing units; preferably, the processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. In some embodiments, the processor 1010 and the memory 1020 may be implemented on a single chip; in other embodiments, they may be implemented on separate chips.

[0057] The memory 1020 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, various applications, etc.; the data storage area can store instruction data created according to the use of the computer device 1000. In addition, the memory 1020 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0058] The computer device 1000 also includes a power supply 1030 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 1010 through a power management system, thereby enabling the management of functions such as charging, discharging, and power consumption through the power management system.

[0059] The display unit 1040 can be used to display information input by the user or information provided to the user, and can also be used to display various menus of the computer device 1000, etc. In this embodiment of the invention, it is mainly used to display the display interfaces of various applications in the computer device 1000, as well as text, images, and other objects displayed in the display interfaces. The display unit 1040 may include a display panel 1050. The display panel 1050 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0060] The input unit 1060 can be used to receive information such as numbers or characters input by the user. The input unit 1060 may include a touch panel 1070 and other input devices 1080. The touch panel 1070 can also be referred to as a touch screen, and the touch panel 1070 can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1070).

[0061] Specifically, the touch panel 1070 can detect user touch operations and the signals generated by these operations, convert these signals into touch point coordinates and send them to the processor 1010, and receive and execute commands transmitted by the processor 1010. Furthermore, the touch panel 1070 can employ various input methods such as resistive, capacitive, infrared, and surface acoustic waves to achieve interaction. Other input devices 1080 include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, and joystick.

[0062] Of course, the touch panel 1070 can also cover the display panel 1050. When the touch panel 1070 detects a touch operation on or near it, it can transmit the information to the processor 1010 to determine the type of touch event. Subsequently, the processor 1010 provides corresponding visual output on the display panel 1050 based on the type of touch event. Although in Figure 24 In this embodiment, the touch panel 1070 and the display panel 1050 are two separate components to realize the input and output functions of the computer device 1000. However, in some embodiments, the touch panel 1070 and the display panel 1050 can be integrated to realize the input and output functions of the computer device 1000.

[0063] The computer device 1000 may also include one or more sensors, such as pressure sensors, gravity acceleration sensors, proximity sensors, etc. Of course, depending on the specific application scenario, the computer device 1000 may also include other components such as cameras.

[0064] In an exemplary embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program / instructions. When executed by a processor, the computer program / instructions enable the computer device to perform the functions described in the present invention. Figure 5 or Figure 9 or Figure 22 The steps of the camera extrinsic parameter calibration method are shown.

[0065] It will be understood by those skilled in the art that Figure 24 The examples of computer devices are merely illustrative and do not constitute a limitation on the device. The device may include more or fewer components than illustrated, or a combination of certain components, or different components. For ease of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, in implementing this invention, the functions of each module (or unit) can be implemented in one or more software or hardware components. For example, as some examples, the aforementioned computer device may be a vehicle, an in-vehicle system, etc.

[0066] Those skilled in the art will understand that the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described in accordance with flowcharts and / or block diagrams of camera extrinsic parameter calibration methods, camera extrinsic parameter calibration systems, and computer program products according to some embodiments. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be applied to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] It is understood that the above embodiments, in collecting, storing, using, processing, transmitting, providing, disclosing, deleting, etc., related data (such as target images), are completed with or with the user's consent. For example, the target image is obtained with the user's knowledge and consent; or it is provided voluntarily by the user after reading the relevant instructions; or it is actively authorized / provided / uploaded by the user when using some or all of the functions described in the above embodiments; or it is obtained through other means or channels with the user's consent.

[0068] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application are still covered by the claims of this application.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, The method includes the following steps: A target image is acquired, and an image containing a preset region is obtained from the target image as a region of interest image; wherein, the target image is obtained by taking a picture of the preset region outside the vehicle using a target camera installed on the vehicle; Corner detection is performed on the region of interest image to obtain the pixel coordinates of the corner points in the preset region within the region of interest image; Based on the pixel coordinates of the corner points in the preset region within the region of interest image and the pixel coordinates of the region of interest image in the target image, the pixel coordinates of the corner points in the preset region in the target image are determined. The target camera is calibrated based on the pixel coordinates of the corner points in the preset area in the target image.

2. The camera extrinsic parameter calibration method according to claim 1, characterized in that, If the preset area includes a checkerboard pattern area, then the process of calibrating the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the target image within the preset area includes: The checkerboard pattern in the checkerboard pattern area is used as the target image, and the direction and distance of the target image are calibrated to obtain the calibration target image; Based on the physical coordinates of the origin in the world coordinate system and the pixel coordinates of the origin in the target image, the physical coordinates and pixel coordinates of all target points are calculated to obtain the calculated physical coordinates and calculated pixel coordinates of each target point; wherein, the target points are determined based on the corner points in the checkerboard pattern area. Based on the calculated physical coordinates and calculated pixel coordinates of all target points, the target camera is calibrated for the first time, which serves as the initial camera extrinsic parameters for the target camera. A bird's-eye view is generated based on the initial projection of the target camera's extrinsic and intrinsic parameters. The corner points in the bird's-eye view are identified, and the identified corner points are back-projected to obtain the pixel coordinates of the corner points in the target image. Based on the calculated physical coordinates of all target points and the pixel coordinates of the corner points in the bird's-eye view in the target image, the target camera is calibrated a second time to obtain the final camera extrinsic parameters.

3. The camera extrinsic parameter calibration method according to claim 2, characterized in that, The method further includes: Determine whether the vehicle is on a flat road surface based on the vehicle's inertial navigation data; Under the condition that the vehicle is on a flat road surface, a panoramic image is generated based on the extrinsic parameter calibration results of the target camera and the target image; When the vehicle is not on a smooth road surface, the extrinsic parameter calibration results of the target camera on a smooth road surface are transformed based on the inertial navigation data to obtain the camera extrinsic parameters of the current road surface; based on the camera extrinsic parameters of the current road surface and the target image, a panoramic image corresponding to the current road surface is generated.

4. The camera extrinsic parameter calibration method according to claim 1, characterized in that, The method further includes: The target image is subjected to ambient brightness value detection, and two pixels are randomly selected from the target image for brightness mean comparison. If the ambient brightness value of the target image is within a preset brightness range, and the average brightness value of any two selected pixels is less than a preset brightness threshold, then the target camera is calibrated based on the pixel coordinates of the corner points in the preset region in the target image. If the ambient brightness value of the target image is not within the preset brightness range, and / or the average brightness of any two selected pixels is greater than or equal to the preset brightness threshold, then a new target image is acquired.

5. The camera extrinsic parameter calibration method according to claim 1, characterized in that, The process of performing corner detection on the region of interest image to obtain the pixel coordinates of the corner points in the preset region within the region of interest image includes: performing grayscale processing on the region of interest image, and using a corner detection model to perform corner detection and recognition on the grayscale processed region of interest image to obtain the pixel coordinates of the corner points in the preset region within the region of interest image.

6. The camera extrinsic parameter calibration method according to claim 5, characterized in that, The training process of the corner detection model includes: A training sample set is obtained, which is based on the back-projected target image after the camera intrinsic and extrinsic parameters have been calibrated, and the target image with target point coordinates corresponding to the back-projected target image. The training set is input into a deep learning model architecture for training to obtain an initial model; The initial model is used to filter the validation sample set, and the samples whose predicted corner points and sample label distances are greater than a preset distance are added to the training sample set. The training sample set is then updated to obtain the updated training sample set. The initial model is iteratively trained using the updated training sample set, and the preset distance during the iterative training process is adjusted. The model trained based on the last preset distance adjustment is used as the corner detection model.

7. The camera extrinsic parameter calibration method according to claim 2, characterized in that, The method further includes: Acquire camera intrinsic parameter calibration images, which are obtained by capturing the target image from different positions and angles using the target camera; The camera intrinsic parameters of the target camera are calculated based on the checkerboard corner position detection results of the camera intrinsic parameter calibration image using a computer vision library.

8. A camera extrinsic parameter calibration system, characterized in that, The system includes: The data acquisition module is used to acquire a target image and obtain an image containing a preset region from the target image as a region of interest image; wherein, the target image is obtained by taking a picture of the preset region outside the vehicle by a target camera installed on the vehicle; The corner detection module is used to perform corner detection on the region of interest image to obtain the pixel coordinates of the corners in the preset region within the region of interest image. The coordinate fusion module is used to determine the pixel coordinates of the corner points in the preset region in the target image based on the pixel coordinates of the corner points in the preset region in the region of interest image and the pixel coordinates of the region of interest image in the target image; The camera extrinsic calibration module is used to calibrate the extrinsic parameters of the target camera based on the pixel coordinates of the corner points in the preset area in the target image.

9. A computer device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the camera extrinsic calibration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the camera extrinsic parameter calibration method as described in any one of claims 1 to 7.

Images