Vehicle-mounted camera calibration optimization method and device, equipment and storage medium

Abstract

The invention discloses a vehicle-mounted camera calibration optimization method and device, equipment and a storage medium, and relates to the technical field of camera calibration, and the method comprises the steps: obtaining camera images, camera internal parameters and initial camera external parameters collected by a plurality of vehicle-mounted fisheye cameras; obtaining a relative coordinate position according to the camera image and the camera internal reference; obtaining a world coordinate error value according to the relative coordinate position and the initial camera external parameter; and obtaining a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to realize camera calibration optimization, thereby solving the problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration under the condition of not depending on deep learning, reducing the dependence of the vehicle-mounted camera calibration on a calibration site and the computing power demand, and improving the calibration efficiency of the vehicle-mounted camera. And the calibration flexibility and the environmental adaptability are improved.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a vehicle-mounted camera calibration optimization method, device, equipment and storage medium. Background Art

[0002] In the widespread application of AVM (Around View Monitor), the calibration of vehicle-mounted cameras is a key link to ensure the quality of surround view image stitching and user experience. However, existing vehicle-mounted camera calibration methods face multiple challenges. First, the dynamic changes in the external environment, such as bumps during vehicle driving, slight deformation of the vehicle body, or after-sales replacement of parts, often cause the external parameters (i.e., extrinsic parameters) of the vehicle-mounted camera to shift, thereby affecting the stitching effect of the surround view image. Secondly, traditional calibration methods mostly rely on specific calibration sites and static calibration processes, which are not only time-consuming, but also have strict requirements on site conditions, such as the need for straight lane lines or fixed positions, which limits the flexibility and convenience of calibration. Furthermore, some methods rely on deep learning algorithms, which require a lot of computing resources and sample set support, increasing the complexity and cost of the system.

[0003] Therefore, how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a vehicle-mounted camera calibration optimization method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning.

[0005] To achieve the above objectives, the present application proposes a vehicle-mounted camera calibration optimization method, the method comprising:

[0006] Obtain camera images, camera intrinsic parameters, and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras;

[0007] Obtaining a relative coordinate position according to the camera image and the camera intrinsic parameters;

[0008] Obtaining a world coordinate error value according to the relative coordinate position and the initial camera extrinsic parameter;

[0009] The target camera extrinsic parameter matrix is ​​obtained according to the preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0010] In one embodiment, the step of obtaining a world coordinate error value according to the relative coordinate position and the initial camera extrinsic parameter comprises:

[0011] Get the world coordinate system position;

[0012] Obtaining an initial rotation matrix and an initial translation vector according to the initial camera extrinsic parameters;

[0013] Projecting the relative coordinate position according to the initial rotation matrix, the initial translation vector, and the world coordinate system position to obtain a back-projected world coordinate;

[0014] A world coordinate error value is obtained according to the back-projected world coordinate.

[0015] In one embodiment, the step of obtaining a world coordinate error value according to the back-projected world coordinate comprises:

[0016] Obtaining a first back-projected world coordinate and a second back-projected world coordinate according to the back-projected world coordinate;

[0017] A back-projection error value is calculated according to the first back-projection coordinates and the second back-projection coordinates, where the first back-projection coordinates and the second back-projection coordinates are back-projection coordinates of two adjacent vehicle-mounted fisheye cameras;

[0018] A world coordinate error value is obtained according to the back-projection error value.

[0019] In one embodiment, the step of obtaining the relative coordinate position according to the camera image and the camera intrinsic parameter comprises:

[0020] Extracting features from the camera image to obtain feature point coordinates;

[0021] Obtaining the camera focal length and the image center point according to the camera intrinsic parameters;

[0022] The relative coordinate position is determined according to the feature point coordinates, the camera focal length and the image center point.

[0023] In one embodiment, the step of extracting features from the camera image to obtain feature point coordinates includes:

[0024] determining lighting conditions based on the camera image;

[0025] Feature extraction is performed on the camera image according to the lighting conditions to obtain feature point coordinates.

[0026] In one embodiment, the step of extracting features from the camera image according to the lighting conditions to obtain feature point coordinates includes:

[0027] Determining a corresponding preprocessing strategy according to the lighting condition, and processing the camera image according to the preprocessing strategy to obtain a target image;

[0028] The target image is subjected to feature extraction according to a preset extraction strategy to obtain feature point coordinates.

[0029] In one embodiment, the step of extracting features from the camera image to obtain feature point coordinates includes:

[0030] Obtaining a color value and a brightness value according to the camera image;

[0031] determining a shadow area according to the color value and the brightness value;

[0032] A local contrast enhancement strategy is applied to the shadow area to increase the brightness of the shadow area.

[0033] In addition, to achieve the above purpose, the present application also proposes a vehicle-mounted camera calibration optimization device, the device comprising:

[0034] A data acquisition module, used to acquire camera images, camera intrinsic parameters and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras;

[0035] A coordinate mapping module, used to obtain a relative coordinate position according to the camera image and the camera intrinsic parameters;

[0036] A data processing module, used for obtaining a world coordinate error value according to the relative coordinate position and the initial camera external parameter;

[0037] The calibration optimization module is used to obtain the target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0038] In addition, to achieve the above-mentioned objectives, the present application also proposes a vehicle-mounted camera calibration optimization device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the vehicle-mounted camera calibration optimization method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the vehicle-mounted camera calibration optimization method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the vehicle-mounted camera calibration optimization method as described above.

[0041] The present application provides a method for optimizing the calibration of a vehicle-mounted camera. The present application obtains camera images, camera intrinsic parameters, and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; obtains relative coordinate positions according to the camera images and the camera intrinsic parameters; obtains world coordinate error values ​​according to the relative coordinate positions and the initial camera extrinsic parameters; and obtains a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization. In summary, the present application adjusts the extrinsic parameters of the vehicle-mounted camera in real time until the calculated error converges to cope with parameter offsets caused by changes in the external environment. At the same time, a static calibration site is used to solve the problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning, reduce the dependence of vehicle-mounted camera calibration on the calibration site and computing power requirements, and improve the flexibility and environmental adaptability of calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the vehicle camera calibration optimization method provided in this application;

[0045] Figure 2 A flowchart diagram of the second embodiment of the vehicle camera calibration optimization method of the present application;

[0046] Figure 3 A flowchart diagram of the third embodiment of the vehicle camera calibration optimization method of the present application;

[0047] Figure 4 This is a schematic diagram of the module structure of the vehicle camera calibration optimization device according to an embodiment of the present application;

[0048] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the vehicle camera calibration optimization method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: obtain camera images, camera intrinsic parameters and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; obtain relative coordinate positions according to the camera images and the camera intrinsic parameters; obtain world coordinate error values ​​according to the relative coordinate positions and the initial camera extrinsic parameters; obtain a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0053] In the wide application of AVM, the calibration of vehicle-mounted cameras is a key link to ensure the quality of surround view image stitching and user experience. However, existing vehicle-mounted camera calibration methods face multiple challenges. First, the dynamic changes in the external environment, such as bumps during vehicle driving, slight deformation of the vehicle body, or after-sales replacement of parts, often cause the external parameters (i.e., extrinsic parameters) of the vehicle-mounted camera to shift, thereby affecting the stitching effect of the surround view image. Secondly, traditional calibration methods mostly rely on specific calibration sites and static calibration processes, which are not only time-consuming, but also have strict requirements on site conditions, such as the need for straight lane lines or fixed positions, which limits the flexibility and convenience of calibration. Furthermore, some methods rely on deep learning algorithms, which require a lot of computing resources and sample set support, increasing the complexity and cost of the system. Therefore, how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning is a problem that needs to be solved urgently.

[0054] This application adjusts the external parameters of the vehicle-mounted camera in real time until the calculated error converges to cope with the parameter offset caused by changes in the external environment. At the same time, a static calibration site is used to solve the problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning, reducing the dependence of vehicle-mounted camera calibration on the calibration site and computing power requirements, and improving the flexibility and environmental adaptability of calibration.

[0055] It should be noted that the execution subject of this embodiment can be a vehicle-mounted camera calibration optimization system, or a computing service device with data processing, network communication and program running functions, or an electronic device capable of realizing the above-mentioned vehicle-mounted camera calibration optimization function, etc., and this embodiment does not specifically limit this. The following takes the vehicle-mounted camera calibration optimization system as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a method for optimizing the calibration of a vehicle-mounted camera. Figure 1 , Figure 1 Schematic diagram of the flow chart of the first embodiment of the vehicle camera calibration optimization method of the present application.

[0057] In this embodiment, the vehicle-mounted camera calibration optimization method includes steps S10 to S40:

[0058] Step S10: Obtain camera images, camera intrinsic parameters, and initial camera extrinsic parameters captured by multiple vehicle-mounted fisheye cameras.

[0059] It should be noted that the fisheye camera can capture a wider field of view due to its large field of view. Camera intrinsics refer to the inherent properties inside the camera, such as focal length and image center point; while camera extrinsics refer to the position and posture of the camera in the world coordinate system. Specifically, in this step, the system will synchronously collect environmental image data around the vehicle through the on-board fisheye camera (including front-view, rear-view, left-view and right-view cameras). The system will transmit these image data, camera intrinsics and the initial extrinsic parameters of each camera to the image processing unit in real time for processing. For example, when the vehicle is stationary, the four fisheye cameras will capture the road images in front of the vehicle, behind the vehicle, on the left and on the right of the vehicle, respectively, and send these images and the corresponding camera intrinsics and initial extrinsic parameter data to the on-board computing unit for further processing.

[0060] In addition, it should be noted that the camera intrinsic parameters are set in advance according to the camera type, model, etc., while the initial extrinsic parameters are set when the vehicle leaves the factory or recorded during the previous calibration process.

[0061] Step S20: Obtaining a relative coordinate position according to the camera image and the camera intrinsic parameters.

[0062] It should be noted that in this step, the system will extract feature points from each camera image in order to identify the feature points of the target from the image. Subsequently, based on these feature points, a feature point matching algorithm is used to establish feature point matching pairs between adjacent camera images. Using the camera's intrinsic parameters and the pixel coordinates of the matching feature points, the matching points are converted from the image coordinate system to the relative coordinate positions in their respective camera coordinate systems through projective geometry calculations.

[0063] In addition, it should be noted that the relative coordinate position refers to the position of the feature point in the respective camera coordinate system.

[0064] Step S30: obtaining a world coordinate error value according to the relative coordinate position and the initial camera extrinsic parameters.

[0065] It should be noted that in this step, the system will back-project the relative coordinate position in each camera coordinate system into a unified world coordinate system based on the known initial camera extrinsics. For each matching point, its expected position in the world coordinate system can be calculated based on the initial camera extrinsics. However, since the camera extrinsics may change due to various reasons (such as external disturbances, body deformation, etc.) in actual situations, there will be errors in the world coordinates obtained by back-projection. By comparing and calculating the world coordinates obtained by back-projection of all matching points, the world coordinate error values ​​of these points can be obtained.

[0066] Step S40: obtaining a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0067] It should be noted that the preset optimization strategy refers to the Lie algebra nonlinear optimization algorithm. Specifically, in this step, the system will perform iterative optimization using the Lie algebra nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm, the Gauss-Newton method, etc.) according to the world coordinate error value calculated in step S30. The goal of the optimization is to find a new set of camera extrinsics so that the world coordinate error value after back projection is minimized. The optimization process starts with the initial extrinsic parameters, and by continuously adjusting the extrinsic parameters, feature point matching, world coordinate back projection and error calculation are repeated until the error value converges or reaches a preset number of iterations. The optimal extrinsic parameter matrix of each camera finally obtained will be used to update the calibration parameters of the vehicle-mounted camera, thereby optimizing the camera calibration.

[0068] The present embodiment provides a method for optimizing the calibration of a vehicle-mounted camera. The present embodiment obtains camera images, camera intrinsic parameters, and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; obtains relative coordinate positions according to the camera images and the camera intrinsic parameters; obtains world coordinate error values ​​according to the relative coordinate positions and the initial camera extrinsic parameters; and obtains a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization. In summary, the present embodiment adjusts the extrinsic parameters of the vehicle-mounted camera in real time until the calculated error converges to cope with parameter offsets caused by changes in the external environment. At the same time, a static calibration site is used to solve the problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning, reduce the dependence of vehicle-mounted camera calibration on the calibration site and computing power requirements, and improve the flexibility and environmental adaptability of calibration.

[0069] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , Figure 2This is a flow chart of the second embodiment of the vehicle camera calibration optimization method of the present application, wherein step S30 specifically includes:

[0070] Step S301: Obtain the world coordinate system position.

[0071] It should be noted that the world coordinate system position refers to mapping the feature points in the image captured by each vehicle-mounted camera to the coordinates in a unified, global three-dimensional space. This process first requires the definition of a unified world coordinate system. Generally, the origin of the world coordinate system is usually selected at a fixed point of the vehicle (such as the center of the vehicle, any point on the chassis, etc.) so as to serve as a reference during the calibration process. The specific process of obtaining the world coordinate system position is as follows: First, determine the origin, coordinate axis direction and unit length of the world coordinate system. For example, the center of the rear axle of the vehicle can be selected as the origin, the X axis points forward along the longitudinal direction of the vehicle, the Y axis points to the left of the driver, and the Z axis points upward perpendicular to the ground. Then, based on the known camera intrinsic parameters (such as focal length, image center point, etc.) and the preliminarily estimated camera extrinsics (rotation matrix R and translation vector t), the feature points in each camera image are converted from the pixel coordinate system to the camera coordinate system, and then further mapped to the world coordinate system. For example, after the vehicle stops, each vehicle-mounted camera (such as the front view, rear view, left view, and right view camera) simultaneously captures the same landmark point on the ground (such as the intersection of road markings). The feature points corresponding to the landmark points in each image are extracted through the image feature point detection algorithm, and the feature points are converted from the image coordinate system to the world coordinate system using the preliminarily estimated camera extrinsic parameter and camera intrinsic parameter matrix to obtain a unified world coordinate system position.

[0072] Step S302: obtaining an initial rotation matrix and an initial translation vector according to the initial camera extrinsic parameters.

[0073] It should be noted that the initial camera extrinsic parameters are obtained through preliminary calibration or rough estimation and are used to initialize the calibration process. The initial camera extrinsic parameters include the rotation matrix R and the translation vector t, which respectively describe the rotation relationship between the camera coordinate system and the world coordinate system, and determine how the image captured by the camera is converted to the world coordinate system. Specifically, in this step, the system decomposes the initial camera extrinsic parameters into the rotation matrix R and the translation vector t. The rotation matrix R is a 3x3 matrix that describes the direction of the camera coordinate system in the world coordinate system; the translation vector t is a 3x1 vector that describes the position of the origin of the camera coordinate system in the world coordinate system.

[0074] Step S303: Projecting the relative coordinate position according to the initial rotation matrix, the initial translation vector, and the world coordinate system position to obtain a back-projected world coordinate.

[0075] It should be noted that the back-projected world coordinates refer to converting these relative coordinate positions to coordinates in the world coordinate system through the initial rotation matrix and the initial translation vector. Specifically, in this step, for each feature point in the camera image, the system will convert it from the pixel coordinate system to the relative coordinate position in the camera coordinate system according to the camera intrinsic parameters. Then, using the initial rotation matrix R and the initial translation vector t, according to the formula of spatial coordinate transformation, the relative coordinate position in the camera coordinate system is back-projected into the world coordinate system to obtain the back-projected world coordinates, as shown in Formula 1 and Formula 2.

[0076]

[0077]

[0078] Among them, u, v are the pixel coordinates of the feature point, Xc, Yc, Zc are the positions of the camera coordinate system, fx, fy are the horizontal and vertical focal lengths of the camera, and cx, cy are the center points of the image. For example, for a feature point in the front-view camera, first convert it from the pixel coordinate system (u, v) to the camera coordinate system (Xc, Yc, Zc) through the camera intrinsic parameter matrix. Then, use the initial rotation matrix R and the initial translation vector t to convert the coordinates (Xc, Yc, Zc) in the camera coordinate system to the world coordinate system (Xw, Yw, Zw) to obtain the back-projected world coordinates of the feature point.

[0079] Step S304: obtaining a world coordinate error value according to the back-projected world coordinate.

[0080] It should be noted that after obtaining the back-projected world coordinates of all matching feature points, it is necessary to calculate the error between these coordinates and the ideal value to evaluate the accuracy of the current camera extrinsic parameters. The error calculation is based on the relative position relationship of the feature points in the world coordinate system. Specifically, the error value e consists of four parts: the back-projection error of the matching points between the front camera and the left camera, the back-projection error of the matching points between the front camera and the right camera, the back-projection error of the matching points between the rear camera and the left camera, and the back-projection error of the matching points between the rear camera and the right camera.

[0081] It can be understood that the world coordinate error value reflects the degree of difference in the position of the feature point in the world coordinate system under different camera perspectives. By minimizing the sum of these error values, the camera's external parameter matrix can be iteratively optimized to make the final calibration result more accurate.

[0082] In a feasible implementation manner, the step S304 specifically includes:

[0083] Step A10: Obtaining a first back-projected world coordinate and a second back-projected world coordinate according to the back-projected world coordinate.

[0084] It should be noted that the first back-projected world coordinates and the second back-projected world coordinates refer to the back-projected world coordinates of two adjacent cameras. Adjacent cameras refer to two cameras that are adjacent in physical position, such as the front view and the left view and the right view, and the rear view and the left view and the right view. Specifically, during the online calibration process of the on-board camera, the system will back-project the feature points in the image into the world coordinate system through the camera intrinsic parameters and the initial extrinsic parameters. Since the calibration process involves multiple on-board cameras (such as front view, rear view, left view, and right view), the back-projected coordinates of each feature point under different cameras may be different.

[0085] Step A20: Calculate a back-projection error value according to the first back-projection coordinates and the second back-projection coordinates, where the first back-projection coordinates and the second back-projection coordinates are back-projection coordinates of two adjacent vehicle-mounted fisheye cameras.

[0086] It should be noted that after obtaining the back-projected world coordinates of two adjacent on-board fisheye cameras, it is necessary to calculate the difference between the two coordinates, that is, the back-projection error value. This error value reflects the difference in observation of the same physical point by different cameras under the current external parameters. Specifically, for each pair of matched feature points, calculate the difference in their back-projected world coordinates under the two cameras. Then traverse all the matched feature point pairs, accumulate the back-projection error values ​​of each pair of adjacent points, and obtain the total back-projection error value.

[0087] It is understandable that the back-projection error value is an important indicator to measure the accuracy of the camera's extrinsic parameters. A smaller back-projection error value means that different cameras have more consistent observations of the same physical point, and the extrinsic calibration is more accurate.

[0088] Step A30: Obtaining a world coordinate error value according to the back-projection error value.

[0089] It should be noted that after obtaining the back-projection error value, the system will convert it into a world coordinate error value for subsequent optimization of external parameters. The world coordinate error value is a comprehensive indicator that takes into account the back-projection errors of all matching feature points. Specifically, as shown in Formula 3.

[0090]

[0091] Where e is the world coordinate error value, Xw1 and Xw2, Yw1 and Yw2, Zw1 and Zw2 represent the x, y, and z coordinates of the matching points of two adjacent cameras in the world coordinate system respectively.

[0092] In this embodiment, a mathematical mapping from the camera coordinate system to the world coordinate system is established by utilizing known initial camera extrinsics, and errors of multiple adjacent cameras are calculated to obtain a total world coordinate error value. This reduces dependence on an external fixed calibration template, making the calibration process more flexible and capable of being performed in a variety of environments and conditions. At the same time, system errors are reduced, thereby improving the stitching effect and user experience of the vehicle-mounted camera surround view system.

[0093] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 3 , Figure 3 This is a flow chart of the third embodiment of the vehicle camera calibration optimization method of the present application, wherein step S20 specifically includes:

[0094] Step S201: extracting features from the camera image to obtain feature point coordinates.

[0095] It should be noted that in this step, the system will pre-process the original image captured by the camera to eliminate noise, enhance contrast, etc., to improve image quality and reduce errors. Then the preset feature point detection algorithm is used to identify the feature points of the target. For each feature point detected, its pixel coordinates (u, v) in the image are calculated. In addition, it should be noted that the pixel coordinates refer to the position of the feature point on the two-dimensional plane of the image, usually expressed as (u, v), where u is the horizontal coordinate and v is the vertical coordinate.

[0096] In a feasible implementation manner, the step S201 specifically includes:

[0097] Step B10: Determine the lighting condition according to the camera image.

[0098] It should be noted that lighting conditions refer to the intensity and distribution of light in the camera shooting environment, which directly affects the quality of the image and the effect of subsequent feature extraction. Strong light conditions may cause overexposure of the image, backlight conditions may cause loss of dark details in the image, and low light conditions will reduce the image brightness and increase noise.

[0099] In addition, it should be noted that, first, the system receives real-time image data from the on-board camera. Subsequently, these images are preliminarily analyzed to identify the current lighting conditions. The analysis process includes calculating the average brightness of the image, the ratio of bright and dark areas, histogram distribution and other features. Based on the above image analysis results, the system determines the current lighting conditions through preset thresholds and algorithm logic. For example, if the average brightness of the image exceeds a certain set threshold and the ratio of bright and dark areas is unbalanced, it is judged as a strong light condition; if there is an obvious dark area in the image and the bright details are lost, it may be judged as a backlight condition; if the overall brightness is low and there is a lot of noise, it is judged as a low light condition. For example: During the driving process of the vehicle, the on-board camera captures an image. The system first calculates the average brightness of the image as 200 (assuming that the brightness value range is 0-255). The ratio of bright and dark areas shows that the bright part accounts for a large proportion, and the histogram distribution is biased towards the high brightness end. At this time, the system determines that the current condition is a strong light condition.

[0100] Step B20: extracting features from the camera image according to the lighting conditions to obtain feature point coordinates.

[0101] It should be noted that in this step, the system will use different preprocessing measures and feature detection methods (such as SIFT, SURF, ORB, etc.) to extract feature points according to different lighting conditions (such as strong light conditions, backlight conditions, and low light conditions). For example: under strong light conditions, the system first performs local histogram equalization on the image to reduce the impact of overexposure. Subsequently, the SIFT algorithm is used to extract feature points on the preprocessed image to obtain a series of stable feature point coordinates.

[0102] In a feasible implementation manner, the step B20 specifically includes:

[0103] Step B201: determining a corresponding preprocessing strategy according to the illumination condition, and processing the camera image according to the preprocessing strategy to obtain a target image.

[0104] It should be noted that in this step, the system will adopt different preprocessing strategies for different lighting conditions. For example: under strong light conditions, the system uses algorithms such as local histogram equalization to reduce the overexposed part of the image to reduce the loss of details caused by excessive brightness. In the face of backlit scenes, the system uses backlight compensation and HDR (high dynamic range) synthesis technology to enhance the dark details of the image. HDR technology can capture multiple images with different exposures and merge them into one image, thereby retaining enough details in both bright and dark areas, so that the feature points of shadows and overexposed areas can be more easily detected. In low-light environments, the system will use methods such as noise reduction and gain adjustment to improve image quality. Noise reduction can reduce noise in the image and improve image clarity; while gain adjustment can increase the brightness of the image without increasing noise, so that feature points can be clearly identified under low-light conditions.

[0105] In addition, it should be noted that the target image refers to the camera image obtained after preprocessing. Local histogram equalization is an image enhancement technology that improves the contrast of the image by performing histogram equalization on a small area of ​​the image, thereby enhancing the image details in a local range. HDR synthesis is a technology that can capture and reproduce high dynamic range images, which can show a wider range of colors and brightness from very bright to very dim, making image recognition clearer.

[0106] Step B202: extracting features from the target image according to a preset extraction strategy to obtain feature point coordinates.

[0107] It should be noted that after obtaining the processed target image, the system will perform feature point detection on the image according to the preset feature point extraction strategy. The preset extraction strategy includes feature point detection algorithms, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), etc. These algorithms can detect stable feature points under different scales, rotations and illumination changes.

[0108] In a feasible implementation manner, before step S201, the method further includes:

[0109] Step C10: obtaining a color value and a brightness value according to the camera image.

[0110] It should be noted that in this step, the system will extract the color value and brightness value of the image before preprocessing. By traversing each pixel of the image, its RGB (red, green, blue) color value and brightness value are calculated (such as by calculating the weighted sum of the RGB values). It can be understood that the acquisition of color value and brightness value provides basic data for subsequent lighting condition detection and shadow area judgment. In addition, it should be noted that color value and brightness value are basic concepts in image processing, where the color value reflects the color information of each pixel in the image, and the brightness value reflects the brightness of the pixel.

[0111] Step C20: determining a shadow area according to the color value and the brightness value.

[0112] It should be noted that in this step, the system uses color and brightness analysis technology to determine the shadow area in the image based on the obtained color value and brightness value. The specific method includes comparing the color and brightness differences of adjacent pixels or areas, and identifying areas where the brightness suddenly decreases and the color changes meet the shadow characteristics. This step can be achieved through pre-set thresholds and judgment criteria to ensure accurate identification of shadow areas. In addition, it should be noted that the shadow area refers to the dark area in the image formed due to the obstruction of light, and its color and brightness values ​​are significantly different from the surrounding areas.

[0113] Step C30: applying a local contrast enhancement strategy to the shadow area to increase the brightness of the shadow area.

[0114] It should be noted that in this step, the system will process the identified shadow area using local contrast enhancement algorithms such as adaptive histogram equalization. These algorithms can adjust the contrast of the shadow area alone without affecting the brightness and color of other areas in the image. By adjusting the parameters of the algorithm (such as the degree of contrast enhancement and the size of the neighborhood), the brightness of the shadow area can be effectively controlled.

[0115] Step S202: Obtain the camera focal length and the image center point according to the camera intrinsic parameters.

[0116] It should be noted that the camera intrinsic parameters are a set of parameters that describe the internal geometric and optical characteristics of the camera, including focal length (fx, fy) and image center point (cx, cy). Specifically, in this step, the system will obtain the camera's intrinsic parameter matrix through the camera calibration process, which contains key information such as focal length and image center point.

[0117] Step S203: determining a relative coordinate position according to the feature point coordinates, the camera focal length and the image center point.

[0118] It should be noted that the relative coordinate position refers to the position (Xc, Yc, Zc) in the camera coordinate system corresponding to the matching point. By combining the feature point coordinates, camera focal length and image center point, the principle of projection geometry can be used to convert these two-dimensional image coordinates into coordinates in the three-dimensional camera coordinate system. Specifically, the system uses the camera intrinsic parameters and feature point coordinates to calculate the three-dimensional coordinates (Xc, Yc, Zc) of the feature point in the camera coordinate system through a perspective projection algorithm. For feature points between different cameras, the system will establish matching pairs through feature point matching algorithms (such as FLANN, BFMatcher, etc.) to ensure that the coordinates of the same physical point captured by different cameras in the camera coordinate system can correspond to each other.

[0119] In this embodiment, feature extraction is first performed based on the camera image and camera internal parameters to obtain feature point coordinates, and then the relative coordinate position is determined by combining the camera focal length and the image center point. At the same time, the image preprocessing strategy is adaptively adjusted according to the lighting conditions to improve the accuracy of feature point extraction. The problem of the influence of lighting changes on feature point extraction and calibration accuracy is solved, and the accuracy and robustness of online calibration of vehicle-mounted cameras are improved.

[0120] This application also provides a vehicle-mounted camera calibration optimization device, please refer to Figure 4 , the vehicle-mounted camera calibration optimization device comprises:

[0121] The data acquisition module 10 is used to acquire camera images, camera intrinsic parameters and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras;

[0122] A coordinate mapping module 20, used to obtain a relative coordinate position according to the camera image and the camera internal parameters;

[0123] A data processing module 30, configured to obtain a world coordinate error value according to the relative coordinate position and the initial camera external parameters;

[0124] The calibration optimization module 40 is used to obtain a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0125] The vehicle-mounted camera calibration optimization device provided by the present application adopts the vehicle-mounted camera calibration optimization method in the above embodiment, which can solve the technical problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning. Compared with the prior art, the beneficial effects of the vehicle-mounted camera calibration optimization device provided by the present application are the same as the beneficial effects of the vehicle-mounted camera calibration optimization method provided by the above embodiment, and the other technical features in the vehicle-mounted camera calibration optimization device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0126] In one embodiment, the coordinate mapping module is also used to extract features from the camera image to obtain feature point coordinates; obtain the camera focal length and the image center point based on the camera intrinsic parameters; and determine the relative coordinate position based on the feature point coordinates, the camera focal length and the image center point.

[0127] In one embodiment, the coordinate mapping module is further used to determine the lighting conditions according to the camera image; and to extract features from the camera image according to the lighting conditions to obtain feature point coordinates.

[0128] In one embodiment, the coordinate mapping module is further used to determine a corresponding preprocessing strategy according to the lighting conditions, and process the camera image according to the preprocessing strategy to obtain a target image; and extract features from the target image according to a preset extraction strategy to obtain feature point coordinates.

[0129] In one embodiment, the data processing module is also used to obtain the world coordinate system position; obtain an initial rotation matrix and an initial translation vector based on the initial camera extrinsic parameters; project the relative coordinate position based on the initial rotation matrix, the initial translation vector, and the world coordinate system position to obtain back-projected world coordinates; and obtain a world coordinate error value based on the back-projected world coordinates.

[0130] In one embodiment, the data processing module is also used to obtain a first back-projection world coordinate and a second back-projection world coordinate based on the back-projection world coordinate; calculate a back-projection error value based on the first back-projection coordinate and the second back-projection coordinate, wherein the first back-projection coordinate and the second back-projection coordinate are the back-projection coordinates of two adjacent vehicle-mounted fisheye cameras; obtain a world coordinate error value based on the back-projection error value; and obtain a target camera extrinsic parameter matrix based on a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0131] The present application provides a vehicle-mounted camera calibration optimization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle-mounted camera calibration optimization method in the above-mentioned embodiment one.

[0132] Reference below Figure 5, which shows a schematic diagram of the structure of a vehicle-mounted camera calibration optimization device suitable for implementing the embodiment of the present application. The vehicle-mounted camera calibration optimization device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The vehicle-mounted camera calibration optimization device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 5 As shown, the vehicle-mounted camera calibration optimization device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the vehicle-mounted camera calibration optimization device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the on-board camera calibration optimization device to communicate wirelessly or wired with other devices to exchange data. Although the on-board camera calibration optimization device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0134] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0135] The vehicle-mounted camera calibration optimization device provided by the present application adopts the vehicle-mounted camera calibration optimization method in the above embodiment, which can solve the technical problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning. Compared with the prior art, the beneficial effects of the vehicle-mounted camera calibration optimization device provided by the present application are the same as the beneficial effects of the vehicle-mounted camera calibration optimization method provided by the above embodiment, and the other technical features in the vehicle-mounted camera calibration optimization device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0136] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0138] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the vehicle-mounted camera calibration optimization method in the above-mentioned embodiment.

[0139] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0140] The computer-readable storage medium may be included in the vehicle-mounted camera calibration optimization device; or may exist independently without being assembled into the vehicle-mounted camera calibration optimization device.

[0141] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the vehicle-mounted camera calibration optimization device, the vehicle-mounted camera calibration optimization device: obtains camera images, camera intrinsic parameters and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; obtains relative coordinate positions according to the camera images and the camera intrinsic parameters; obtains world coordinate error values ​​according to the relative coordinate positions and the initial camera extrinsic parameters; obtains a target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

[0142] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0145] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle-mounted camera calibration optimization method, and can solve the technical problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the vehicle-mounted camera calibration optimization method provided in the above-mentioned embodiment, and will not be repeated here.

[0146] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned vehicle-mounted camera calibration optimization method when executed by a processor.

[0147] The computer program product provided in this application can solve the technical problem of how to improve the flexibility and convenience of vehicle-mounted camera calibration without relying on deep learning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the vehicle-mounted camera calibration optimization method provided in the above embodiment, and will not be repeated here.

[0148] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle-mounted camera calibration optimization method, characterized in that: The method comprises: Obtain camera images, camera intrinsic parameters, and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; Obtaining a relative coordinate position according to the camera image and the camera intrinsic parameters; Obtaining a world coordinate error value according to the relative coordinate position and the initial camera extrinsic parameter; The target camera extrinsic parameter matrix is ​​obtained according to the preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

2. The method according to claim 1, characterized in that The step of obtaining the world coordinate error value according to the relative coordinate position and the initial camera external parameter comprises: Get the world coordinate system position; Obtaining an initial rotation matrix and an initial translation vector according to the initial camera extrinsic parameters; Projecting the relative coordinate position according to the initial rotation matrix, the initial translation vector, and the world coordinate system position to obtain a back-projected world coordinate; A world coordinate error value is obtained according to the back-projected world coordinate.

3. The method according to claim 2, characterized in that The step of obtaining the world coordinate error value according to the back-projected world coordinate comprises: Obtaining a first back-projected world coordinate and a second back-projected world coordinate according to the back-projected world coordinate; A back-projection error value is calculated according to the first back-projection coordinate and the second back-projection coordinate, where the first back-projection coordinate and the second back-projection coordinate are back-projection coordinates of two adjacent vehicle-mounted fisheye cameras; A world coordinate error value is obtained according to the back-projection error value.

4. The method according to claim 1, characterized in that The step of obtaining the relative coordinate position according to the camera image and the camera intrinsic parameter comprises: Extracting features from the camera image to obtain feature point coordinates; Obtaining the camera focal length and the image center point according to the camera intrinsic parameters; The relative coordinate position is determined according to the feature point coordinates, the camera focal length and the image center point.

5. The method according to claim 4, characterized in that The step of extracting features from the camera image to obtain feature point coordinates comprises: determining lighting conditions based on the camera image; Feature extraction is performed on the camera image according to the lighting conditions to obtain feature point coordinates.

6. The method according to claim 5, characterized in that The step of extracting features from the camera image according to the lighting conditions to obtain feature point coordinates comprises: Determining a corresponding preprocessing strategy according to the lighting condition, and processing the camera image according to the preprocessing strategy to obtain a target image; The target image is subjected to feature extraction according to a preset extraction strategy to obtain feature point coordinates.

7. The method according to claim 4, characterized in that Before the step of extracting features from the camera image to obtain feature point coordinates, the method further includes: Obtaining a color value and a brightness value according to the camera image; determining a shadow area according to the color value and the brightness value; A local contrast enhancement strategy is applied to the shadow area to increase the brightness of the shadow area.

8. A vehicle-mounted camera calibration optimization device, characterized in that: The device comprises: A data acquisition module, used to acquire camera images, camera intrinsic parameters and initial camera extrinsic parameters collected by multiple vehicle-mounted fisheye cameras; A coordinate mapping module, used to obtain a relative coordinate position according to the camera image and the camera intrinsic parameters; A data processing module, used for obtaining a world coordinate error value according to the relative coordinate position and the initial camera external parameter; The calibration optimization module is used to obtain the target camera extrinsic parameter matrix according to a preset optimization strategy and the world coordinate error value to achieve camera calibration optimization.

9. A vehicle-mounted camera calibration optimization device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle-mounted camera calibration optimization method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle-mounted camera calibration optimization method according to any one of claims 1 to 7 are implemented.

Images