Image processing and camera calibration method for automotive vision system and related apparatus

Abstract

This application discloses an image processing and camera calibration method and related apparatus for an automotive imaging system. The method includes: generating current calibration data corresponding to the current height of the camera using pre-stored calibration data and height change characterization information of the camera in the automotive imaging system. The pre-stored calibration data is calibration data collected during calibration of the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height. The method also includes calibrating the camera using the current calibration data to obtain current calibration parameters; and processing the image captured by the camera at the current height based on the current calibration parameters. Through this method, this application can recalibrate the camera of the automotive imaging system to obtain a better display effect.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to an image processing and camera calibration method and related apparatus for an automotive imaging system. Background Technology

[0002] As automobiles continue to develop, they possess increasingly more functions, such as the installation of automotive imaging systems. These systems use cameras to capture images of the vehicle's surroundings, stitching them together and sending the results back to the driver for navigation purposes. The stitched images from these systems need to be as smooth as possible, without misalignment or broken lines, which places higher demands on camera calibration. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide an image processing and camera calibration method and related apparatus for an automotive imaging system, which can recalibrate the camera of the automotive imaging system to obtain a better display effect of the automotive imaging system.

[0004] To address the aforementioned technical problems, the first aspect of this application provides an image processing method for an automotive imaging system. This method includes: generating current calibration data corresponding to the current height of the camera using pre-stored calibration data and height change characterization information from the camera of the automotive imaging system; wherein the pre-stored calibration data is calibration data collected during calibration of the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height; calibrating the camera using the current calibration data to obtain current calibration parameters; and processing the image collected by the camera at the current height based on the current calibration parameters.

[0005] To address the aforementioned technical problems, a second aspect of this application provides a camera calibration method for an automotive imaging system. The method includes: generating current calibration data corresponding to the current height of the camera using pre-stored calibration data and height change characterization information of the camera in the automotive imaging system; wherein the pre-stored calibration data is calibration data collected during calibration of the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height; and calibrating the camera using the current calibration data to obtain current calibration parameters.

[0006] To address the aforementioned technical problems, a third aspect of this application provides an image processing apparatus for an automotive imaging system. This apparatus includes a calibration data generation module, a calibration parameter generation module, and an image processing module. The calibration data generation module generates current calibration data corresponding to the current height of the camera using pre-stored calibration data from the camera of the automotive imaging system and height change characterization information. The pre-stored calibration data is the calibration data collected during calibration of the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height. The calibration parameter generation module calibrates the camera using the current calibration data to obtain current calibration parameters. The image processing module processes the image captured by the camera at the current height based on the current calibration parameters.

[0007] To address the aforementioned technical problems, a fourth aspect of this application provides an electronic device comprising a memory and a processor coupled to each other, wherein the memory stores program instructions and the processor executes the program instructions to implement the methods provided in the first or second aspect.

[0008] To address the aforementioned technical problems, the fifth aspect of this application also provides a computer-readable storage medium for storing program instructions that can be executed by a processor to implement the methods provided in the first or second aspect.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, this application utilizes pre-stored calibration data and height change characterization information of the camera in the automotive imaging system to generate current calibration data corresponding to the current height of the camera. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height. By using the height change characterization information to generate current calibration data, the influence of camera height changes on camera parameters can be resolved, thereby improving the accuracy of the current calibration parameters. The current calibration data is then used to calibrate the camera to obtain current calibration parameters, thus recalibrating the camera in the automotive imaging system. Finally, based on the current calibration parameters, the image captured by the camera at the current height is processed to obtain a better display effect for the automotive imaging system. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating one embodiment of the image processing method for the automotive imaging system provided in this application;

[0011] Figure 2 This is a flowchart illustrating another embodiment of the image processing method for the automotive imaging system provided in this application;

[0012] Figure 3 This is a flowchart illustrating another embodiment of the image processing method for the automotive imaging system provided in this application;

[0013] Figure 4 This is a flowchart illustrating one embodiment of the camera calibration method for the automotive imaging system provided in this application;

[0014] Figure 5 This is a schematic diagram of the framework of an embodiment of the image processing device for the automotive imaging system provided in this application;

[0015] Figure 6 This is a schematic diagram of the framework of one embodiment of the camera calibration device for the automotive imaging system provided in this application;

[0016] Figure 7 This is a schematic diagram of the framework of one embodiment of the electronic device provided in this application;

[0017] Figure 8 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image processing method for an automotive imaging system provided in this application. The method includes:

[0022] S11: Using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information, generate current calibration data corresponding to the current height of the camera.

[0023] In one embodiment, the vehicle imaging system is a surround view monitor (AVM). Specifically, the vehicle imaging system includes multiple cameras, for example, four cameras, respectively positioned at the four corners of the vehicle, for capturing the surrounding environment. The pre-stored calibration data of the cameras is the calibration data collected when calibrating cameras located at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height.

[0024] In one embodiment, the pre-stored calibration data may include the original calibration image; the height change characterization information includes transformation parameters between images captured by the camera at the current height and a preset height, where the current height is the height of the camera under the current load of the vehicle, and the preset height can be the height of the camera during previous calibration. The current height and the preset height can be the same or different, and both the current height and the preset height are related to the load of the vehicle. The original calibration image can then be transformed using the transformation parameters to obtain the current calibration image, which is then used as the current calibration data. Alternatively, the current calibration data can be obtained based on the current calibration image. For example, the current corner point can be detected in the current calibration image, and the original object point and the current corner point in the original calibration image can be used as the current calibration data. Here, a corner point can be understood as the intersection of two edges, meaning that the local neighborhood of a corner point should have boundaries in different directions of two different regions; an object point can be an object in the image considered as a point, meaning that an object point represents an object in the image.

[0025] In another embodiment, the pre-stored calibration data may include calibration data collected by multiple sets of cameras at different preset heights. Specifically, the calibration data may be the original calibration images used by the cameras at different preset heights, or the corner points and object points used. Height change characterization information includes the height difference between the current height and the preset height. Then, the pre-stored calibration data matching the current height is selected from the multiple sets of pre-stored calibration data as reference calibration data. The current calibration data is generated using the height difference between the preset height and the current height corresponding to the reference calibration data, as well as the reference calibration data itself.

[0026] S12: Use the current calibration data to calibrate the camera and obtain the current calibration parameters.

[0027] In one embodiment, the current calibration data is the current calibration image. The camera can be calibrated directly using the current calibration image. Alternatively, the current corner point in the current calibration image can be detected, and the original object point in the original calibration image can be obtained. The Zhang's calibration method can be used to calibrate the camera using the current corner point and the original object point.

[0028] In another embodiment, if the current calibration data consists of the original object point and the current corner point, then Zhang's calibration method is directly used to calibrate the camera using the current corner point and the original object point. It is understood that other calibration methods can also be used in other embodiments, and this is not limited here.

[0029] The current calibration parameters may include the camera intrinsic matrix, distortion coefficients, and extrinsic matrix. The camera intrinsic matrix includes parameters such as the camera's focal length, pixel width, and height, which can be used to convert image coordinates to camera coordinates; the distortion coefficients are used to eliminate radial and tangential distortion of the image; the extrinsic matrix includes information such as the camera's position and orientation in the world coordinate system, which can be used to convert camera coordinates to world coordinates.

[0030] S13: Process the image captured by the camera at the current altitude based on the current calibration parameters.

[0031] In one embodiment, the images captured by each camera at the current height can be stitched together based on the current calibration parameters of each camera in the vehicle, and the stitched image can be displayed.

[0032] In another embodiment, for each camera in the vehicle imaging system, steps S11 and S12 are executed to obtain the current calibration parameters of each camera. The images captured by each camera at the current height are processed using the current calibration parameters of each camera, and the processed images of each camera are stitched together. It is determined whether there are any misalignments, broken lines, or other phenomena in the stitched image, or whether the misalignments or broken lines in the stitched image are within a preset error range. In response to whether there are any misalignments, broken lines, or other phenomena in the stitched image, or whether the misalignments or broken lines in the stitched image are within a preset error range, the camera calibration is completed; otherwise, the camera is recalibrated.

[0033] The above method utilizes pre-stored calibration data and height change characterization information from the camera in the automotive imaging system to generate current calibration data corresponding to the camera's current height. The pre-stored calibration data is the data collected during calibration of the camera at a preset height, and the height change characterization information represents the difference between the current height and the preset height. By using the height change characterization information to generate current calibration data, the impact of camera height changes on camera parameters can be addressed, thereby improving the accuracy of the camera's current calibration parameters. The camera is then recalibrated using the current calibration data to obtain the current calibration parameters, thus calibrating the camera in the automotive imaging system. Finally, based on the current calibration parameters, the image captured by the camera at the current height is processed to obtain a better display effect for the automotive imaging system.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the image processing method for the automotive imaging system provided in this application. The method includes:

[0035] S21: Perform image transformation on the original calibration image using transformation parameters to obtain the current calibration image.

[0036] In one embodiment, the transformation parameter is a transformation parameter between images captured by the camera at the current height and at a preset height. Specifically, a first image of the target scene captured by the camera at the preset height is acquired. The preset height can be the height of the camera when the vehicle is unloaded, the height of the camera when only the driver is sitting in the vehicle, or the height of the camera when the vehicle is full of passengers. A second image of the target scene captured by the camera at the current height is acquired. The current height can be the height of the camera when the vehicle is full of passengers, the height of the camera when the vehicle is unloaded, or the height of the camera when only the driver is sitting in the vehicle.

[0037] The current height and the preset height can be the same. If the current field of view of the camera is the same as the field of view at the preset height, it indicates that the current height and the preset height are the same. In this case, only the first image captured by the camera at the preset height needs to be acquired to determine whether the display effect of the first image meets the preset display effect. If it does, no calibration of the camera is required; if the display effect of the first image does not meet the preset display effect, the camera needs to be recalibrated. The current height and the preset height can also be different. If the current field of view of the camera changes relative to the field of view at the preset height, it indicates that the current height and the preset height are different. In this case, the first image and the second image need to be acquired to recalibrate the camera. In one specific embodiment, the preset height is the height of the camera when only the driver is sitting in the car, and the current height is the height of the camera when the car is full of passengers, that is, the preset height is higher than the current height.

[0038] Transformation parameters are determined based on matching feature points in the first and second images captured by the camera. In one specific implementation, the SIFT (Scale-invariant feature transform), ORB (Oriented FAST and Rotated BRIEF), or FAST (Features from accelerated segment test) algorithm can be used to detect key points and calculate image features in the first and second images, respectively, to obtain key points in the first and second images. Further, the key points in the first and second images are matched to obtain matching feature points, which can also be called corresponding points. Transformation parameters are calculated based on the matching feature points, and the transformation parameters can be a transformation matrix. The matching of key points in the first and second images can be performed using the RANSAC (Random Sample Consensus) algorithm. It is understood that other methods can also be used for key point matching in other implementations, and no specific limitation is made here.

[0039] Furthermore, the original calibration image is transformed using transformation parameters to obtain the current calibration image. Specifically, the position and pixel value (i.e., color value) of each pixel in the original calibration image are obtained. The pixel position is multiplied by the transformation parameters to obtain the new pixel position. The new pixel position is the position of that pixel in the current calibration image. The new pixel position and pixel value correspond one-to-one. The current calibration image is generated using the new pixel position and pixel value.

[0040] S22: Use the current calibration image as the current calibration data, or obtain the current calibration data based on the current calibration image.

[0041] S23: Use the current calibration data to calibrate the camera and obtain the current calibration parameters.

[0042] In one embodiment, the current calibration image can be used as the current calibration data, and the camera can be calibrated using the current calibration image to obtain the current calibration parameters.

[0043] In another embodiment, the current calibration image can be detected to obtain the current corner point. The original object point and the current corner point in the original calibration image can be used as the current calibration constants, and the camera can be calibrated using the original object point and the current corner point.

[0044] S24: Process the image captured by the camera at the current altitude based on the current calibration parameters.

[0045] For the specific implementation of steps S23 and S24, please refer to S12 and S13 of the first embodiment of the image processing method for the automotive imaging system provided in this application, which will not be repeated here.

[0046] In other embodiments, before performing steps S21 or S24, this application further includes: determining whether the effect of displaying the current image using the original calibration parameters achieves a preset display effect, wherein the current image is an image captured by the camera at the current height; and in response to the effect of displaying the current image using the original calibration parameters achieving the preset display effect, using the original calibration parameters as the current calibration parameters.

[0047] In one specific implementation, when a change in camera height is detected, it can be first determined whether the display effect of the current image captured by each camera meets the preset display effect when the calibration parameters of each camera are the original calibration parameters; or, if the calibration parameters of each camera in the vehicle imaging system are the original calibration parameters, the current images captured by each camera are stitched together, and the display effect of the stitched image meets the preset display effect. If the display effect meets the preset display effect, the original calibration parameters are used as the current calibration parameters; if the display effect does not meet the preset display effect, steps S21 to S24 are executed. The preset display effect can be that there are no misalignments, unclear images, broken lines, unevenness, or other phenomena in the image.

[0048] In this embodiment, after the camera height changes, it can first determine whether the effect of displaying the current image based on the original calibration parameters meets the preset display effect, and then determine whether the camera needs to be recalibrated. If the effect of displaying the current image based on the original calibration parameters does not meet the preset display effect, then the camera needs to be recalibrated; otherwise, recalibration is not required.

[0049] Please see Figure 3 , Figure 3This is a flowchart illustrating another embodiment of the image processing method for the automotive imaging system provided in this application. The method includes:

[0050] S31: Select the pre-stored calibration data that matches the current height from multiple sets of pre-stored calibration data as the reference calibration data.

[0051] In one embodiment, the pre-stored calibration data is the calibration data collected when the camera is calibrated at different preset heights. Therefore, there are multiple sets of calibration data, each set of calibration data corresponds to a preset height. The calibration data can be the original calibration image or corner points. The preset height of the camera can be related to the load of the vehicle, that is, the camera can be at different preset heights when the vehicle is under different load conditions.

[0052] Before executing step S31, multiple sets of calibration data used for calibrating the camera at different preset heights can be acquired, i.e., multiple sets of pre-stored calibration data. Each set of pre-stored calibration data includes corner point data used for calibrating each camera in the automotive imaging system. The calibration load detection value of the vehicle and the preset height corresponding to each set of pre-stored calibration data are also acquired when collecting each set of pre-stored calibration data. The calibration load detection value of the vehicle can be the pressure value detected by the pressure sensor on the vehicle seat. The preset height corresponding to each set of pre-stored calibration data can be obtained by manually measuring the camera height, by detecting it with a height sensor, or by determining it based on the calibration parameters corresponding to the pre-stored calibration data. The calibration parameters corresponding to the pre-stored calibration data can include the camera's extrinsic parameters, including rotation and translation matrices. The camera's position coordinates are obtained based on the rotation and translation matrices, and the preset height can be obtained based on the camera's position coordinates, for example, by using t... c =-R T *t can calculate the camera's position coordinates, using the y-direction value as the preset height value, where R is the rotation matrix and t is the translation matrix. c These are the coordinates of the camera's location.

[0053] Furthermore, by utilizing the calibration load detection values ​​and corresponding preset heights corresponding to each set of pre-stored calibration data, the relationship between the vehicle load detection value and the camera height is obtained. Taking a car imaging system containing four cameras as an example, each set of pre-stored calibration data contains calibration data used to calibrate the four cameras respectively. The calibration load detection value corresponding to each set of pre-stored calibration data may include a first pressure value, and the preset height corresponding to each set of pre-stored calibration data may include the height values ​​of the four cameras. The average height value of the four cameras is calculated to obtain the average height of the cameras corresponding to that set of pre-stored calibration data. Therefore, for each set of pre-stored calibration data, it corresponds to a first pressure value and an average height. By linearly fitting the functional relationship between the pressure value and the average height corresponding to all pre-stored calibration data, the relationship between the vehicle load detection value and the camera height is obtained. The relationship between the vehicle load detection value and the camera height can be represented by a relational function.

[0054] Obtain the vehicle's current load detection value, which can be the second pressure value detected by the pressure sensor on the vehicle seat under the current load condition. Utilize the relationship between the vehicle load detection value and the camera's height to obtain the current height corresponding to the current load detection value. Substitute the second pressure value into the relational function to obtain the current height corresponding to the current load detection value. Select the pre-stored calibration data that matches the current height from the corresponding preset heights as reference calibration data. From different preset heights, select the preset height that is closest to the current height corresponding to the current load detection value as the target height, and use the pre-stored calibration data corresponding to the target height as the reference calibration data.

[0055] S32: Generate current calibration data using the height difference between the preset height and the current height corresponding to the reference calibration data, as well as the reference calibration data.

[0056] In one embodiment, the height coordinates of the object point in the reference calibration data can be added to the height difference to obtain a new object point; the new object point and the corner points in the reference calibration data are used as the current calibration data.

[0057] S33: Use the current calibration data to calibrate the camera and obtain the current calibration parameters.

[0058] In one embodiment, Zhang's calibration method can be used to calibrate each camera based on new object points and corner points in reference calibration data to obtain the current calibration parameters.

[0059] S34: Process the image captured by the camera at the current altitude based on the current calibration parameters.

[0060] For details on the specific implementation of steps S33 and S34, please refer to S12 and S13 of the first embodiment of the image processing method for the automotive imaging system provided in this application, which will not be repeated here.

[0061] In one embodiment, the images captured by each camera at the current height can be stitched together based on the current calibration parameters of each camera in the vehicle, and the stitched image can be displayed. Furthermore, it is determined whether the stitched image exhibits misalignment, broken lines, or unevenness. If none of these issues exist, or if misalignment, broken lines, or unevenness exist but are within a certain error range, the stitched image is considered to meet the requirements.

[0062] The above method involves pre-calibrating cameras at different preset heights, obtaining calibration data used during calibration at these heights as pre-stored calibration data, acquiring the pressure values ​​from the pressure sensors on the car seat during calibration at these heights, and obtaining calibration parameters obtained after calibration at these heights. Using these calibration parameters, the camera height is determined; a functional relationship between different preset heights and pressure values ​​is established. Based on the pressure sensor detection values ​​and the functional relationship under the current load, the current height of the camera under the current load is determined. The pre-stored calibration data corresponding to the preset height closest to the current height is selected as reference calibration data. The height coordinates of the object point in the reference calibration data are added to the height difference to obtain a new object point. The new object point and the corner points in the reference calibration data are used as the current calibration data. The camera is calibrated using the current calibration data to obtain the current calibration parameters, and the image acquired by the camera at the current height is processed based on these parameters.

[0063] This implementation method can calibrate the camera, thereby optimizing the stitching effect of the automotive imaging system. Compared to the existing method of manually reducing tire pressure to calibrate the camera in the automotive imaging system, it can reduce costs and improve image processing efficiency.

[0064] Please see Figure 4 , Figure 4 This is a flowchart illustrating one embodiment of the camera calibration method for an automotive imaging system provided in this application. The method includes:

[0065] S41: Using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information, generate current calibration data corresponding to the current height of the camera.

[0066] Among them, the pre-stored calibration data is the calibration data collected when calibrating the camera located at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height.

[0067] S42: Use the current calibration data to calibrate the camera and obtain the current calibration parameters.

[0068] Please refer to the relevant description in the image processing method of the automotive imaging system provided in this application for this embodiment, which will not be repeated here.

[0069] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the image processing device for the automotive imaging system provided in this application.

[0070] The image processing device 50 includes a calibration data generation module 51, a calibration parameter generation module 52, and an image processing module 53. The calibration data generation module 51 is used to generate current calibration data corresponding to the current height of the camera by using the pre-stored calibration data of the camera in the vehicle imaging system and height change characterization information. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height. The calibration parameter generation module 52 is used to calibrate the camera using the current calibration data to obtain the current calibration parameters. The image processing module 53 is used to process the image collected by the camera at the current height based on the current calibration parameters.

[0071] In one embodiment, the pre-stored calibration data includes the original calibration image, and the height change characterization information includes the transformation parameters between the images captured by the camera at the current height and the preset height respectively; the calibration data generation module 51 can also be used to perform image transformation on the original calibration image using the transformation parameters to obtain the current calibration image; use the current calibration image as the current calibration data, or obtain the current calibration data based on the current calibration image.

[0072] In one embodiment, the image processing device 50 further includes a first acquisition module and a transformation parameter determination module (not shown) capable of communicating with each other. The first acquisition module is used to acquire a first image of the target scene captured by the camera at a preset height, and a second image of the target scene captured by the camera at the current height. The transformation parameter determination module is used to determine transformation parameters based on matching feature points in the first and second images captured by the camera. The transformation parameter determination module can send the transformation parameters to the calibration data generation module 51, so that the calibration data generation module 51 can use the transformation parameters to perform image transformation on the original calibration image to obtain the current calibration image.

[0073] In one embodiment, the transformation parameter is a transformation matrix; the first acquisition module can also acquire a second image of the target scene captured by the camera at the current height in response to a change in the current field of view of the camera relative to the field of view at a preset height.

[0074] In one embodiment, the calibration data generation module 51 can also be used to detect the current corner point from the current calibration image; and use the original object point and the current corner point in the pre-stored calibration data as the current calibration data.

[0075] In one embodiment, the image processing device 50 further includes a judgment module (not shown in the figure). Before the calibration parameter generation module 52 calibrates the camera using the current calibration data to obtain the current calibration parameters, the judgment module can determine whether the effect of displaying the current image using the original calibration parameters achieves the preset display effect. The current image is an image captured by the camera at the current height. In response to the effect of displaying the current image using the original calibration parameters achieving the preset display effect, the original calibration parameters are used as the current calibration parameters. The calibration parameter generation module 52 can also be used to calibrate the camera using the current calibration data to obtain the current calibration parameters in response to the effect of displaying the current image using the original calibration parameters not achieving the preset display effect.

[0076] In one embodiment, there are multiple sets of pre-stored calibration data. These sets of pre-stored calibration data are calibration data collected by the camera at different preset heights. The height change characterization information includes the height difference between the current height and the preset height. The calibration data generation module 51 can also be used to select the pre-stored calibration data that matches the current height from the multiple sets of pre-stored calibration data as reference calibration data. The current calibration data is generated using the height difference between the preset height and the current height corresponding to the reference calibration data and the reference calibration data.

[0077] In one embodiment, the image processing device 50 may further include a second acquisition module (not shown in the figure). The different preset heights of the camera when acquiring multiple sets of pre-stored calibration data are caused by the vehicle under different load conditions. Before the calibration data generation module 51 selects the pre-stored calibration data that matches the current height from the multiple sets of pre-stored calibration data as reference calibration data, the device further includes: acquiring the calibration load detection value of the vehicle when acquiring each set of pre-stored calibration data, and acquiring the preset height corresponding to each set of pre-stored calibration data using the calibration load detection value corresponding to each set of pre-stored calibration data and the corresponding preset height; obtaining the relationship between the vehicle load detection value and the height of the camera using the calibration load detection value corresponding to each set of pre-stored calibration data and the corresponding preset height; the calibration data generation module 51 may also be used to acquire the current load detection value of the vehicle; obtain the current height corresponding to the current load detection value using the relationship between the vehicle load detection value and the height of the camera; and select the pre-stored calibration data whose corresponding preset height matches the current height as reference calibration data.

[0078] In one embodiment, both the calibration load detection value and the current load detection value are determined using pressure values ​​detected by pressure sensors on the seats of the vehicle; and / or, for each set of pre-stored calibration data, the second acquisition module can also use the calibration parameters corresponding to the pre-stored calibration data to determine the preset height corresponding to the pre-stored calibration data; and / or, the second acquisition module can also be used to linearly fit the functional relationship between the calibration load detection value and the corresponding preset height corresponding to each set of pre-stored calibration data, as the relationship between the vehicle load detection value and the height of the camera.

[0079] In one embodiment, the calibration data generation module 51 can also be used to add the height difference to the height coordinates of the object point in the reference calibration data to obtain a new object point; and use the new object point and the corner point in the reference calibration data as the current calibration data.

[0080] In one embodiment, the image processing module 53 is further configured to perform a stitching operation on the images captured by each camera of the vehicle at the current height based on the current calibration parameters of each camera, and display the stitched image.

[0081] Understandably, the image processing apparatus provided in this embodiment is used to execute the image processing method in the method embodiment. For detailed implementation of the image processing method, please refer to the above method embodiment, which will not be repeated here.

[0082] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of one embodiment of the camera calibration device for the automotive imaging system provided in this application.

[0083] The camera calibration device 60 includes a calibration data generation module 51 and a calibration parameter generation module 52. The calibration data generation module 51 is used to generate current calibration data corresponding to the current height of the camera by using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height, and the height change characterization information characterizes the difference between the current height and the preset height. The calibration parameter generation module 52 is used to calibrate the camera using the current calibration data to obtain the current calibration parameters.

[0084] Please see Figure 7 , Figure 7 This is a schematic diagram of one embodiment of the electronic device provided in this application.

[0085] The electronic device 70 includes a memory 71 and a processor 72 coupled to each other. The memory 71 stores program instructions, and the processor 72 executes the program instructions to implement the steps in any of the above method embodiments. Specifically, the electronic device 70 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablets, etc., and is not limited thereto.

[0086] Specifically, processor 72 controls itself and memory 71 to implement the steps in any of the above method embodiments. Processor 72 may also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 72 may be implemented using integrated circuit chips.

[0087] Please see Figure 8 , Figure 8 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application.

[0088] The computer-readable storage medium 80 stores program instructions 81, which, when executed by a processor, are used to implement the steps in any of the above method embodiments.

[0089] The computer-readable storage medium 80 can specifically be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store computer programs. Alternatively, it can be a server that stores the computer program, which can send the stored computer program to other devices for execution or can also run the stored computer program itself.

[0090] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0091] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method for an automotive imaging system, characterized in that, include: Using the pre-stored calibration data and height change characterization information of the camera in the vehicle imaging system, current calibration data corresponding to the current height of the camera is generated. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height. There are multiple sets of pre-stored calibration data. The height change characterization information characterizes the difference between the current height and the preset height. The camera is calibrated using the current calibration data to obtain the current calibration parameters; Based on the current calibration parameters, the image captured by the camera at the current height is processed; Wherein, when the height change characterization information includes the height difference between the current height and the preset height, the step of generating current calibration data corresponding to the current height of the camera using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information includes: Select the pre-stored calibration data that matches the current height from the multiple sets of pre-stored calibration data, and use it as the reference calibration data; A new object point is obtained by adding the height coordinates of the object point in the reference calibration data to the height difference between the preset height corresponding to the reference calibration data and the current height; The new object point and the corner points in the reference calibration data are used as the current calibration data.

2. The method according to claim 1, characterized in that, The pre-stored calibration data includes the original calibration image, and the height change characterization information includes the transformation parameters between the images captured by the camera at the current height and the preset height, respectively. The step of generating current calibration data corresponding to the current height of the camera using pre-stored calibration data and height change characterization information of the vehicle imaging system's camera includes: The original calibration image is transformed using the transformation parameters to obtain the current calibration image; The current calibration image is used as the current calibration data, or the current calibration data is obtained based on the current calibration image.

3. The method according to claim 2, characterized in that, Before performing image transformation on the original calibration image using the transformation parameters to obtain the current calibration image, the method further includes: Acquire a first image of the target scene captured by the camera at the preset height, and acquire a second image of the target scene captured by the camera at the current height; The transformation parameters are determined based on the matching feature points in the first image and the second image captured by the camera.

4. The method according to claim 3, characterized in that, The transformation parameter is a transformation matrix; And / or, acquiring the second image of the target scene captured by the camera at the current height includes: In response to a change in the current field of view of the camera relative to the field of view at the preset height, a second image of the target scene captured by the camera at the current height is acquired.

5. The method according to claim 3, characterized in that, The process of obtaining the current calibration data based on the current calibration image includes: The current corner point is detected from the current calibration image; The original object point and the current corner point in the pre-stored calibration data are used as the current calibration data.

6. The method according to claim 2, characterized in that, Before calibrating the camera using the current calibration data to obtain the current calibration parameters, the method further includes: Determine whether the effect of displaying the current image using the original calibration parameters achieves the preset display effect, wherein the current image is the image captured by the camera at the current height; In response to the effect of displaying the current image using the original calibration parameters to achieve the preset display effect, the original calibration parameters are used as the current calibration parameters; The step of calibrating the camera using the current calibration data to obtain the current calibration parameters includes: In response to the fact that the effect of displaying the current image using the original calibration parameters does not achieve the preset display effect, the camera is calibrated using the current calibration data to obtain the current calibration parameters.

7. The method according to claim 1, characterized in that, The different preset heights at which the camera is positioned when collecting multiple sets of the pre-stored calibration data are caused by the vehicle under different load conditions. Before selecting the pre-stored calibration data that matches the current height from multiple sets of pre-stored calibration data as reference calibration data, the method further includes: Obtain the calibration load detection value of the vehicle when collecting each set of the pre-stored calibration data, and obtain the preset height corresponding to each set of the pre-stored calibration data; By using the calibration load detection values ​​corresponding to each set of pre-stored calibration data and the corresponding preset height, the relationship between the vehicle load detection value and the height of the camera is obtained; The step of selecting the pre-stored calibration data that matches the current height from multiple sets of pre-stored calibration data as reference calibration data includes: Obtain the current load detection value of the vehicle; By utilizing the relationship between the vehicle load detection value and the camera height, the current height corresponding to the current load detection value is obtained; Select the pre-stored calibration data that matches the preset height and the current height as the reference calibration data.

8. The method according to claim 7, characterized in that, Both the calibrated load detection value and the current load detection value are determined using the pressure values ​​detected by the pressure sensors on the seats of the vehicle. And / or, obtaining the preset height corresponding to each group of the pre-stored calibration data includes: For each set of pre-stored calibration data, the preset height corresponding to the pre-stored calibration data is determined using the calibration parameters corresponding to the pre-stored calibration data; And / or, the step of obtaining the relationship between the vehicle load detection value and the height of the camera by utilizing the calibration load detection values ​​corresponding to each set of pre-stored calibration data and the corresponding preset height includes: The linear fitting of the functional relationship between the calibration load detection value and the corresponding preset height corresponding to each set of pre-stored calibration data is used as the relationship between the vehicle load detection value and the height of the camera.

9. The method according to claim 1, characterized in that, The step of processing the image captured by the camera at the current height based on the current calibration parameters includes: Based on the current calibration parameters of each of the vehicle's cameras, the images captured by each of the vehicle's cameras at the current height are stitched together, and the stitched image is displayed.

10. A camera calibration method for an automotive imaging system, characterized in that, include: Using the pre-stored calibration data and height change characterization information of the camera in the vehicle imaging system, current calibration data corresponding to the current height of the camera is generated. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height. There are multiple sets of pre-stored calibration data. The height change characterization information characterizes the difference between the current height and the preset height. The camera is calibrated using the current calibration data to obtain the current calibration parameters; Wherein, when the height change characterization information includes the height difference between the current height and the preset height, the step of generating current calibration data corresponding to the current height of the camera using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information includes: Select the pre-stored calibration data that matches the current height from the multiple sets of pre-stored calibration data, and use it as the reference calibration data; A new object point is obtained by adding the height coordinates of the object point in the reference calibration data to the height difference between the preset height corresponding to the reference calibration data and the current height; The new object point and the corner points in the reference calibration data are used as the current calibration data.

11. An image processing device for an automotive imaging system, characterized in that, include: The calibration data generation module is used to generate current calibration data corresponding to the current height of the camera by using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information. The pre-stored calibration data is the calibration data collected when calibrating the camera at a preset height. There are multiple sets of pre-stored calibration data. The height change characterization information characterizes the difference between the current height and the preset height. The calibration parameter generation module is used to calibrate the camera using the current calibration data to obtain the current calibration parameters; An image processing module is used to process the image captured by the camera at the current height based on the current calibration parameters. Wherein, when the height change characterization information includes the height difference between the current height and the preset height, the step of generating current calibration data corresponding to the current height of the camera using the pre-stored calibration data of the camera in the vehicle imaging system and the height change characterization information includes: Select the pre-stored calibration data that matches the current height from the multiple sets of pre-stored calibration data, and use it as the reference calibration data; A new object point is obtained by adding the height coordinates of the object point in the reference calibration data to the height difference between the preset height corresponding to the reference calibration data and the current height; The new object point and the corner points in the reference calibration data are used as the current calibration data.

12. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions to implement the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the method of any one of claims 1-10.

Images