Vehicle-mounted panoramic image calibration method and device, vehicle-mounted panoramic image generation method and device, medium and vehicle

Abstract

The invention provides a vehicle-mounted panoramic image calibration method and device, a vehicle-mounted panoramic image generation method and device, a medium and a vehicle, and relates to the technical field of vehicles. According to the method, firstly, four view images collected by a to-be-calibrated vehicle in a preset calibration site are obtained, and in response to calibration operation of a user, a target calibration point in each view image is determined, so that the accuracy of the calibration points is ensured in a manner of selecting the calibration points in a user-defined manner. Then, based on the original coordinate of the target calibration point in the view image and the calibration coordinate of the target calibration point in the top view image corresponding to the view image, determining a coordinate conversion parameter corresponding to the view image, and calibrating the overlapping area of the four top view images; and the splicing weight parameter of the overlapping area between any two adjacent top view images is determined, so that the vehicle-mounted panoramic image is obtained after the view images are correspondingly processed, and the generation quality of the vehicle-mounted panoramic image is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle-mounted panoramic image calibration method, generation method, device, medium and vehicle. Background Art

[0002] With the continuous development of living standards and science and technology, cars have gradually become an indispensable part of people's daily lives. At present, panoramic image assistance systems are generally installed in car computers. The panoramic image assistance system is mainly a set of 360-degree panoramic fusion images around the car body through the on-board navigation display, which can effectively help car owners understand the blind spots of the vehicle and make it safer and more convenient for drivers to drive the car. The 360-degree panoramic fusion image is generally calculated through the calibration function (based on the feature points of the image of a specific site) to calculate the position and posture of the camera in the real world, and the four camera images are blended according to the data obtained by the calibration function, and then spliced ​​into a complete 360-degree panoramic image of the car body.

[0003] Currently, the calibration of vehicle-mounted panoramic images is performed using the chessboard calibration method. This method is to arrange several chessboards around the car and calibrate the camera's calculation parameters by identifying the coordinates of the intersections of black and white grids. However, the recognition of the chessboard is greatly affected by the light and shadow environment. The problem of the intersection of the chessboard cannot be recognized due to light or ground wear often occurs, which affects the calibration efficiency and accuracy. Summary of the invention

[0004] The present application provides a vehicle-mounted panoramic image calibration method, generation method, device, medium and vehicle, which can improve the efficiency and accuracy of vehicle-mounted panoramic image calibration, thereby improving the quality of vehicle-mounted panoramic images.

[0005] A first aspect of an embodiment of the present application provides a vehicle-mounted panoramic image calibration method, the method comprising:

[0006] Acquire four view images collected by the vehicle to be calibrated in a preset calibration site, and determine a target calibration point in each of the view images in response to a user's calibration operation;

[0007] Determine, based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image, coordinate conversion parameters corresponding to the view image; the coordinate conversion parameters are used to process the view image to obtain the overhead view image corresponding to the view image;

[0008] The overlapping area of ​​the four overhead images is calibrated to determine the stitching weight parameters of the overlapping area between any two adjacent overhead images; the stitching weight parameters are used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

[0009] Optionally, in response to a calibration operation of a user, determining a target calibration point in each of the view images includes:

[0010] In response to a first selection instruction of a user for any of the view images, identifying a plurality of original calibration points in the view image;

[0011] In response to a second selection instruction of the user for any of the original calibration points, a preset area including the original calibration point is enlarged to obtain a local enlarged image;

[0012] In response to a user's movement instruction for the original calibration point, the original calibration point is moved in the partial magnified image to obtain the target calibration point corresponding to the original calibration point.

[0013] Optionally, before moving the original calibration point in the local enlarged image, the method further includes:

[0014] In response to a user's instruction to select a movement coefficient for the original calibration point, a unit movement distance corresponding to the original calibration point in the partial magnified image is determined.

[0015] Optionally, the coordinate transformation parameters include a distortion mapping matrix and a perspective transformation matrix;

[0016] Determining coordinate conversion parameters corresponding to the view image based on original coordinates of the target calibration point in the view image and calibration coordinates of the target calibration point in the overhead view image corresponding to the view image includes:

[0017] Determine a distortion mapping matrix corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image;

[0018] Based on the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image, a perspective transformation matrix corresponding to the view image is determined.

[0019] Optionally, calibrating the overlapping area of ​​the four overhead images includes:

[0020] In response to a stitching instruction from a user, respectively marking the to-be-stitched areas of the four overhead images;

[0021] Based on the to-be-joined areas corresponding to any two adjacent overhead images, an overlapping area between the two overhead images is determined.

[0022] Optionally, the overlapping area is a polygonal area containing acute angles;

[0023] Determine the stitching weight parameters of the overlapping area between any two adjacent overhead images, including:

[0024] For any pixel point in the overlapping area, determine a weight value corresponding to the pixel point based on a ratio of distances between the pixel point and two acute-angle edges of the overlapping area;

[0025] The splicing weight parameter is formed based on the weight values ​​corresponding to all the pixels in the overlapping area.

[0026] Based on the same inventive concept, a second aspect of an embodiment of the present application provides a vehicle-mounted panoramic image calibration device, the device comprising:

[0027] A calibration point selection module, used to obtain four view images collected by the vehicle to be calibrated in a preset calibration site, and determine a target calibration point in each of the view images in response to a user's calibration operation;

[0028] A first calibration module, configured to determine a coordinate conversion parameter corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image; the coordinate conversion parameter is used to process the view image to obtain the overhead view image corresponding to the view image;

[0029] The second calibration module is used to calibrate the overlapping area of ​​the four overhead images and determine the stitching weight parameters of the overlapping area between any two adjacent overhead images; the stitching weight parameters are used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

[0030] A third aspect of the present application provides a method for generating a vehicle-mounted panoramic image, the method comprising:

[0031] Acquire four target view images of the target vehicle;

[0032] For any target view image, the target view image is processed based on the coordinate conversion parameters corresponding to the target view image to obtain a target overhead view image corresponding to the target view image;

[0033] Based on the stitching weight parameter corresponding to the overlapping area between any two adjacent target overhead images, the four target overhead images are stitched and fused to obtain a target vehicle-mounted panoramic image;

[0034] The coordinate conversion parameters and the stitching weight parameters are obtained by the vehicle-mounted panoramic image calibration method in the first aspect.

[0035] A fourth aspect of the present application provides a vehicle-mounted panoramic image generation device, the device comprising:

[0036] A view acquisition module, used for acquiring four target view images of a target vehicle;

[0037] An image processing module, used for processing any target view image based on the coordinate conversion parameters corresponding to the target view image to obtain a target overhead view image corresponding to the target view image;

[0038] An image stitching module, configured to stitch and fuse the four target overhead images based on a stitching weight parameter corresponding to an overlapping area between any two adjacent target overhead images, to obtain a target vehicle-mounted panoramic image;

[0039] The coordinate conversion parameters and the stitching weight parameters are obtained by the vehicle-mounted panoramic image calibration method in the first aspect.

[0040] Based on the same inventive concept, the fifth aspect of an embodiment of the present application provides a storage medium, which stores machine executable instructions. When the machine executable instructions are executed by a processor, the vehicle-mounted panoramic image calibration method proposed in the first aspect of the present application is implemented.

[0041] Based on the same inventive concept, the sixth aspect of an embodiment of the present application provides a vehicle, comprising a processor and a memory; the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the vehicle-mounted panoramic image generation method proposed in the third aspect of the present application.

[0042] Compared with the prior art, this application has the following advantages:

[0043] The embodiment of the present application provides a vehicle-mounted panoramic image calibration method, firstly obtains four view images collected by the vehicle to be calibrated in a preset calibration site, and determines the target calibration point in each view image in response to the user's calibration operation. Then, based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead image corresponding to the view image, the coordinate conversion parameters corresponding to the view image are determined; the coordinate conversion parameters are used to process the view image to obtain the overhead image corresponding to the view image. After that, the overlapping area of ​​the four overhead images is calibrated to determine the splicing weight parameters of the overlapping area between any two adjacent overhead images; the splicing weight parameters are used to splice and fuse the four overhead images to obtain the vehicle-mounted panoramic image. Thus, by customizing the selection of calibration points, the accuracy of the calibration points is guaranteed, the problem of inaccurate calibration point recognition in the prior art is solved, and the flexibility and adaptability of panoramic image calibration are improved. At the same time, the coordinate conversion parameters and splicing weight parameters are obtained by calibration, which are used to obtain the vehicle-mounted panoramic image after the view image is processed accordingly, thereby improving the generation quality of the vehicle-mounted panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 is a flow chart of a vehicle-mounted panoramic image calibration method in one embodiment of the present application;

[0046] Figure 2 is a schematic diagram of selecting calibration points in one embodiment of the present application;

[0047] Figure 3 It is a custom schematic diagram of the outer contour of the left area of ​​the front bird's-eye view in one embodiment of the present application;

[0048] Figure 4 It is a self-defined schematic diagram of the outer contour of the front area of ​​the left bird's-eye view in one embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the overlapping area of ​​the front bird's-eye view and the left bird's-eye view in one embodiment of the present application;

[0050] Figure 6 is a vehicle-mounted panoramic image before weight optimization in one embodiment of the present application;

[0051] Figure 7 is a vehicle-mounted panoramic image after weight optimization in one embodiment of the present application;

[0052] Figure 8 It is a schematic diagram of the architecture of vehicle-mounted panoramic image calibration in one embodiment of the present application;

[0053] Fig. 9 is a flow chart of vehicle-mounted panoramic image calibration in one embodiment of the present application;

[0054] Fig.10 It is a functional module diagram of a vehicle-mounted panoramic image calibration device in one embodiment of the present application;

[0055] Fig.11 is a flow chart of a method for generating a vehicle-mounted panoramic image in one embodiment of the present application;

[0056] Fig.12 This is a functional module diagram of a vehicle-mounted panoramic image generating device in one embodiment of the present application;

[0057] Fig.13 It is a structural schematic diagram of a vehicle in one embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The on-board panoramic image assistance system includes four cameras installed around the car, an image acquisition component, a video processing component, a digital image processing component, and an on-board display. The cameras take images of the front, back, left, and right sides of the car respectively, and the images are converted into digital information by the image acquisition component and sent to the video processing component. The images processed by the video processing component are converted into analog signals and output by the digital image processing component, and panoramic image information of the car and its surrounding environment is generated on the on-board display installed inside the car. The video processing component requires a series of parameters during the calculation process, and this series of parameters is strongly related to the position and angle of the camera installed on the vehicle. Due to assembly errors, the installation positions and angles of the cameras on the car are different. Therefore, after the four cameras are assembled on the car production line, they must be calibrated to calculate these parameters.

[0060] Currently, the calibration of vehicle-mounted panoramic images is carried out using the chessboard calibration method. This method is to arrange several chessboards around the car and calibrate the calculation parameters of image processing by identifying the coordinates of the intersections of black and white grids. However, the recognition of chessboards is greatly affected by the light and shadow environment. The intersections of chessboards are often not recognized due to light or ground wear, which affects the calibration efficiency and accuracy.

[0061] In view of this, the present application proposes a method for calibrating a vehicle-mounted panoramic image. On the one hand, by customizing the selection of calibration points, the accuracy of the calibration points is ensured, the problem of inaccurate identification of calibration points in the prior art is solved, and the flexibility and adaptability of panoramic image calibration are improved. At the same time, calibration conversion is performed based on the selected calibration points to obtain coordinate conversion parameters and splicing weight parameters. In actual application, the four view images are processed using the coordinate conversion parameters to obtain the top view image corresponding to the view image, and the four top view images are spliced ​​and fused using the splicing weight parameters to obtain the vehicle-mounted panoramic image. Thereby ensuring the conversion accuracy and splicing fusion effect of the four view images, and improving the generation quality of the final vehicle-mounted panoramic image.

[0062] Please refer to Figure 1 , Figure 1 1 is a flow chart of a vehicle-mounted panoramic image calibration method proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0063] S101: Acquire four view images collected by the vehicle to be calibrated in a preset calibration site, and determine a target calibration point in each view image in response to a calibration operation of a user.

[0064] When a vehicle is being developed or sold, the on-board panoramic image needs to be calibrated. Before calibration, the vehicle usually needs to be placed in a specific preset calibration site, such as a site with a black and white chessboard, or a site with rectangular color blocks, etc., to provide easy-to-find reference points with accurate coordinates for the calibration of the on-board panoramic image, such as chessboard intersections, color block corners, etc.

[0065] In this embodiment, the vehicle to be calibrated has four cameras, namely, a front view camera, a rear view camera, a left view camera and a right view camera, and the view images captured by the four cameras are the front view, the rear view, the left view and the right view. When determining the target calibration point (i.e., the reference point) in each view image, by providing the user with a calibrable interactive interface, the user can select and determine the position of the target calibration point by himself, which not only improves the accuracy of the calibration point, reduces the coordinate error of the calibration point, effectively avoids secondary calibration, but also makes the calibration process of the panoramic image more flexible, can adapt to various calibration environments, and improves the calibration efficiency.

[0066] It should be noted that the number of target calibration points can be set according to actual conditions, for example, it can be set to 4 or 8. In a feasible implementation, the preset calibration site is a site with black rectangular blocks in the upper left corner, upper right corner, lower left corner and lower right corner of the vehicle, so the front view of the vehicle will include two black blocks in the upper left corner and upper right corner, the left view will include two black blocks in the upper left corner and lower left corner, the rear view will include two black blocks in the lower left corner and lower right corner, and the right view will include two black blocks in the upper right corner and lower right corner, and then 8 target calibration points will be determined in each view image, and the target calibration points are the corner points of the black blocks. Figure 2 As shown, the 8 circled black corner points are the 8 target calibration points.

[0067] S102: Determine coordinate conversion parameters corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image; the coordinate conversion parameters are used to process the view image to obtain the overhead view image corresponding to the view image.

[0068] In this embodiment, the four view images collected by the four cameras of the vehicle are fisheye images. The fisheye image has a very wide viewing angle and can capture wide scenes that are difficult to capture with ordinary lenses, but there is also obvious perspective distortion, that is, the straight lines in the picture may be bent, especially at the edge of the picture, and this distortion makes the fisheye image visually different from the real world in people's eyes. Therefore, in order to obtain a vehicle-mounted panoramic image, it is necessary to convert the four view images (i.e., fisheye images) into a top view (i.e., a bird's-eye view) through a corresponding algorithm, and then splice the top views corresponding to the four perspectives, and finally obtain a panoramic bird's-eye view that can present a 360-degree panoramic view around the vehicle without dead angles. In other words, the purpose of the vehicle-mounted panoramic image calibration in this application is to generate a mapping relationship (or calibration product) that meets the requirements, including: the coordinate mapping relationship (i.e., coordinate conversion parameters) from the fisheye image of each perspective to the bird's-eye view, and the splicing relationship (splicing weight parameters) of the four bird's-eye views.

[0069] Specifically, in the process of determining the coordinate conversion parameters of each fisheye image, first, according to the target calibration point selected in the above step S101, the pixel coordinates of the target calibration point in the fisheye image can be known. Then, through the position of the target calibration point calibrated in the bird's-eye view corresponding to the fisheye image, the pixel coordinates of the target calibration point in the bird's-eye view corresponding to the fisheye image can be known. Afterwards, according to the pixel coordinates of multiple target calibration points in the fisheye image and its bird's-eye view, the coordinate mapping relationship of multiple target calibration points in the process of converting from the fisheye image to the bird's-eye view can be calculated. At the same time, for the remaining pixel points in the fisheye image, the pixel coordinates of the pixel point in the bird's-eye view can be obtained according to the relative position relationship between the pixel point and a certain target calibration point. Finally, the pixel coordinates of each pixel point in the fisheye image after being converted to the bird's-eye view are counted to obtain the coordinate conversion parameters corresponding to the fisheye image.

[0070] It is easy to understand that the coordinate conversion parameters are equivalent to a map table, which records the correspondence between the first coordinate of each pixel in the fisheye image and the second coordinate of the pixel in the bird's-eye view corresponding to the fisheye image. For example, the pixel 1 with coordinates (80,80) in the fisheye image has coordinates (50,50) in the bird's-eye view. Therefore, when generating a panoramic image later, the view image is directly processed according to the coordinate conversion parameters (directly mapping the pixel 1 with coordinates (80,80) in the fisheye image to the position (50,50) in the bird's-eye view), and the top-view image corresponding to the view image is obtained.

[0071] S103: Calibrate the overlapping area of ​​the four overhead images to determine the stitching weight parameter of the overlapping area between any two adjacent overhead images; the stitching weight parameter is used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

[0072] In this embodiment, the overlapping area refers to the overlapping and spliced ​​area between two adjacent overhead images. In this embodiment, the overlapping area can be freely set. By adjusting the size of the overlapping area, a custom overlapping area of ​​appropriate size can be selected for calibration, so that four overhead images can be spliced ​​based on the calibrated overlapping area during actual use. The splicing weight parameter of the overlapping area is a weight matrix composed of the weight values ​​of all pixels in the overlapping area. Based on the weight matrix, the fusion effect of the overlapping area can be optimized to improve the generation quality of the vehicle-mounted panoramic image.

[0073] Specifically, due to the wide shooting angle of the fisheye camera, the image areas captured by the four adjacent cameras on the front, back, left, and right sides of the vehicle body overlap, which will cause ghosting and misalignment problems in the splicing and fusion process of the bird's-eye view, such as Figure 6As shown in the box. When stitching the four overhead images corresponding to the four perspectives, if the overlapping area between two adjacent overhead images is too large, ghosting, blurring and the like are likely to occur, and if the overlapping area is too small, the stitching and fusion effect is likely to be stiff. In this embodiment, the overlapping area of ​​the four overhead images is calibrated to determine a suitable overlapping area between two adjacent overhead images to ensure the image stitching effect and quality. At the same time, the pixel value of each pixel in the overlapping area is adjusted according to the weight value corresponding to the pixel to optimize the stitching and fusion effect of the overlapping area and improve the visual display effect of the on-board panoramic image (such as Figure 7 shown).

[0074] On the one hand, this embodiment provides a calibrable interactive interface for users, so that users can select and determine the position of the target calibration point by themselves, and then adjust the calibration effect by themselves, with a higher degree of freedom. On the other hand, calibration conversion is performed based on the selected calibration point to obtain coordinate conversion parameters and splicing weight parameters. In actual application, the four view images are processed using the coordinate conversion parameters to obtain the top view image corresponding to the view image, and the four top view images are spliced ​​and fused using the splicing weight parameters to obtain the vehicle-mounted panoramic image. This ensures the conversion accuracy and splicing fusion effect of the four view images, and improves the generation quality of the final vehicle-mounted panoramic image.

[0075] In another embodiment of the present application, the calibration process of the vehicle-mounted panoramic image specifically includes the following steps:

[0076] S201: In response to a first selection instruction of a user for any view image, identifying a plurality of original calibration points in the view image.

[0077] In this embodiment, by providing a calibrable interactive interface for users, users can select views and select calibration points by themselves. Specifically, when the user selects a certain view image, the view page corresponding to the view will be entered, and the reference point position in the view image will be automatically identified to obtain multiple original calibration points corresponding to the view image. Among them, the reference point (i.e., the original calibration point) is related to the setting of the calibration site. Exemplarily, the reference point can be the corner point of the rectangular color block set at the upper left corner, upper right corner, lower left corner and lower right corner of the vehicle. That is, after the user selects a certain view image, the 8 corner points of the two rectangular color blocks in the view image will be automatically identified, and the 8 corner points will be marked in the view image in the form of a cursor (such as Figure 2 ), in order to provide users with a basis for point selection and avoid users blindly selecting calibration points, resulting in poor panoramic image calibration results.

[0078] Furthermore, when performing corner point recognition, the Harris corner point detection algorithm, the FAST corner point detection algorithm, the SUSAN corner point detection algorithm, etc. may be used. For details, please refer to the prior art, which will not be described in detail here.

[0079] S202: In response to a second selection instruction of the user for any original calibration point, a preset area including the original calibration point is enlarged to obtain a local enlarged image.

[0080] In this embodiment, the preset area refers to a regular area containing the original calibration point, for example, a square area of ​​10×10 pixels centered on the original calibration point. When the user selects any original calibration point in a view, the area near the currently selected original calibration point will be magnified and displayed at the bottom of the view image. Figure 2 As shown in the figure, the ideal original calibration point is the corner point of the color block, but due to the error of image recognition, the original calibration point actually detected may be a point next to the corner point of the color block. Figure 2 After the original calibration point close to the upper left corner of the left color block is found, the original calibration point and the corresponding right-angle point of the color block will be magnified and displayed at the bottom.

[0081] S203: In response to a user's instruction to select a movement coefficient for an original calibration point, determining a unit movement distance corresponding to the original calibration point in the partial magnified image.

[0082] In this embodiment, if Figure 2 As shown in the figure, the original calibration point is displayed in the form of a cursor. The user can move the cursor corresponding to the original calibration point in the partially enlarged image to adjust the position of the original calibration point. During the movement, the user can select or customize the unit movement distance of the cursor. For example, if the movement coefficient is set to 0.2, the cursor moves 0.2 times the current pixel at one time, thereby realizing flexible adjustment of the original calibration point.

[0083] S204: In response to a user's movement instruction for the original calibration point, the original calibration point is moved in the partially enlarged image to obtain a target calibration point corresponding to the original calibration point.

[0084] In this embodiment, since there is a certain deviation between the original calibration point actually detected and the ideal color block corner point, this embodiment amplifies the original calibration point and the surrounding area so that the user can more intuitively see the difference between the original calibration point and the color block corner point. Then the user can adjust the position of the original calibration point by moving the cursor in the partially enlarged image (the cursor movement direction can be controlled by clicking the up, down, left, and right buttons in the operation area). When the cursor coincides with the right angle point of the color block, click OK to obtain the target calibration point corresponding to the original calibration point. Thus, by manual fine-tuning, it is ensured that the final target calibration point is more closely aligned with the color block corner point, so as to provide a more accurate and effective reference point for the subsequent calibration of the panoramic image.

[0085] S205: Determine a distortion mapping matrix corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image.

[0086] In this embodiment, the captured view image is a fisheye image. In order to obtain its corresponding top view image for panoramic image stitching, the fisheye image needs to be dedistorted and perspective transformed. Therefore, in the calibration process, the distortion mapping matrix and perspective transformation matrix corresponding to the view image need to be determined. The distortion mapping matrix is ​​used to dedistort the view image to obtain a dedistorted image corresponding to the view image.

[0087] In the process of determining the distortion mapping matrix corresponding to the view image, the fisheye image is dedistorted mainly based on the camera intrinsic parameters (including the camera intrinsic parameter matrix, distortion coefficient, etc.), the original coordinates of the target calibration points in the view image, and the calibration coordinates of the target calibration points in the dedistorted image corresponding to the view image (calibrated in advance by humans). The cv::fisheye::initUndistortRectifyMap() function of the OpenCV software is used to obtain the mapping relationship from the fisheye image to the dedistorted image, that is, the distortion mapping matrix.

[0088] Specifically, the mapping relationship is given in the form of a map, including mapX (mapping of x coordinates) and mapY (mapping of y coordinates). That is, each of the four view images corresponds to a distortion mapping matrix, and any distortion mapping matrix contains mapping matrices of x coordinates and y coordinates.

[0089] S206: Determine a perspective transformation matrix corresponding to the view image based on the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image.

[0090] Although the de-distortion process can correct the distortion in the image and make the image content more realistic visually, it often cannot change the overall shape of the image. In this case, the perspective transformation can be used to adjust the shape of the image from a circle or semicircle to a standard rectangle or square, making it more in line with common image display and processing requirements.

[0091] In this implementation, when determining the perspective transformation matrix corresponding to any view image, the dedistorted image is perspectively transformed using the cv::findHomography() function of the OpenCV software, mainly based on the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image and the calibration coordinates of the target calibration point in the overhead image corresponding to the view image, so as to obtain a mapping relationship from the dedistorted image to the overhead image, i.e., the perspective transformation matrix. That is, the perspective transformation matrix is ​​used to perform view transformation on the dedistorted image to obtain the overhead image corresponding to the view image.

[0092] S207: In response to the user's stitching instruction, the to-be-stitched areas of the four overhead images are marked respectively.

[0093] In this embodiment, after obtaining the top view images corresponding to the four view images, the four top view images can be spliced ​​to obtain the vehicle-mounted panoramic image. When splicing, it is first necessary to determine how to splice two adjacent top view images, that is, it is necessary to calibrate the size of the overlapping area when the four top view images are spliced.

[0094] Specifically, in order to determine an overlapping area of ​​appropriate size, firstly, the area to be stitched of the four overhead images is circled. For example, taking the images of the front camera and the left camera as an example, the overhead image obtained by the front camera after the above processing has the following stitching range in the left area: Figure 3 As shown, the top view image obtained by the left camera after the above processing has a stitching range in the front area as shown in Figure 4 shown.

[0095] S208: Determine an overlapping area between two bird's-eye view images based on the to-be-joined area corresponding to any two adjacent bird's-eye view images.

[0096] In this embodiment, for any two adjacent top-view images corresponding to the to-be-joined regions, the intersection of the two to-be-joined regions is determined as the overlapping region between the two top-view images. In subsequent applications, the two adjacent top-view images are directly joined according to the overlapping region. Figure 5 As shown, Figure 5 The quadrilateral in the middle frame is the overlapping area between the front top view image and the left top view image.

[0097] It should be noted that the overlapping area is a quadrilateral with an acute angle. Therefore, in this embodiment, the size of the overlapping area can be freely controlled by customizing the outer contour (customizing the area to be spliced) (for example, Figure 5 The included angle of the acute angle of the quadrilateral in the overlapping area becomes smaller, so that the entire overlapping area is reduced to an appropriate size), which solves the problem of ghosting caused by too large an overlapping area and a stiff splicing and fusion effect caused by too small an overlapping area.

[0098] S209: For any pixel point in the overlapping area, determine a weight value corresponding to the pixel point based on the ratio of the distance between the pixel point and the two acute-angle edges of the overlapping area.

[0099] In this embodiment, if Figure 5 As shown, after determining the overlapping area between any two adjacent overhead images, for any pixel point in the overlapping area, the weight value corresponding to the pixel point is determined based on the ratio of the distance between the pixel point and the two acute angles of the overlapping area. During splicing and fusion, for any pixel point in the overlapping area, the pixel value of the pixel point after fusion is obtained by multiplying the pixel value of the pixel point with the weight value corresponding to the pixel point, so as to improve the display effect after splicing and fusion.

[0100] S210: forming a splicing weight parameter based on weight values ​​corresponding to all pixels in the overlapping area.

[0101] In this embodiment, for the overlapping area of ​​any two adjacent overhead images, the stitching weight parameter corresponding to the overlapping area can be obtained by counting the weight values ​​corresponding to all pixels in the overlapping area, thereby obtaining a total of four stitching weight parameters.

[0102] Specifically, when stitching and fusing four overhead view images, the four overhead view images are first stitched according to the calibrated overlapping area, and then the pixel values ​​of all pixel points in the overlapping area are adjusted using the stitching weight parameters corresponding to the overlapping area to reduce the stitching ghosting, maintain a smooth transition of the stitching, and improve the final display effect of the on-board panoramic image.

[0103] The following is a specific example to illustrate the calibration process of this application. Figure 8 , Figure 8 FIG. 1 is a schematic diagram of the architecture of vehicle-mounted panoramic image calibration in one embodiment of the present application. Figure 8 As shown, this embodiment includes a three-level hierarchical architecture, namely, the App layer (application layer), the Native layer (service layer) and the algorithm library layer.

[0104] Among them, the App layer: an application software running on the car computer. The App has two pages, namely the view selection page and the calibration point selection (including calibration point selection and movement) page. The view selection page is responsible for switching the view images of the four cameras in the front, back, left and right, and sending the calibration instructions after the calibration point selection is completed. The App layer interacts with the Native layer and sends the user operations to the Native layer through buttons. Native layer: responsible for receiving the message instructions sent by the App layer, and forwarding the message instructions to the algorithm library layer, and responsible for displaying the camera screen and the algorithm library layer screen to the App layer, playing a connecting role. Algorithm library layer: responsible for receiving the message instructions passed by the Native layer, and performing different operations according to different instructions, including internal parameter reading, distortion correction, perspective transformation and stitching and fusion operations, to complete the panoramic image calibration process, and finally generate the product required for the panoramic image function.

[0105] like Fig. 9 As shown in the figure, in the process of panoramic image calibration using this architecture: when the App is started, it will send a message command to the Native layer, the Native layer will turn on the camera and pass the camera image to the App for display. The user can select one of the four views, front, back, left, and right, on the view selection page, and enter the calibration point selection page corresponding to the view to customize the position of the calibration point. Figure 2 As shown, the right area of ​​the App will display Figure 2 The camera screen and the cursors corresponding to the 8 calibration points are shown. At the same time, the area near the currently selected calibration point will be magnified at the bottom. When the user adjusts any calibration point (the cursor currently being selected is blue, and the others are red), the cursor needs to be moved by the up / down / left / right buttons to ensure that the cursor moves to the corner position of the corresponding black block in the camera image. In addition, the pixel distance moved by the cursor at one time can be adjusted by clicking different cursor movement coefficient buttons to ensure that the cursor moves accurately to the corner point. After all 4 views have selected points, click the point selection completion button on the view selection page. Then, the algorithm library will receive the calibration instructions passed down from the Native layer. At this time, the internal parameter reading module of the algorithm library will read the camera internal parameters, and the distortion correction module will perform dedistortion processing on the fisheye image based on the camera internal parameters, the coordinates of the selected points (coordinates in the fisheye image) and the calibrated coordinates of the point (coordinates in the dedistorted image) to obtain the distortion mapping matrix. The perspective transformation module will perform perspective transformation on the dedistorted image based on the coordinates after dedistortion processing and the calibrated top view coordinates to obtain the perspective transformation matrix. The stitching and fusion module will calibrate the overlapping area of ​​the four top views, calculate and save the weight value corresponding to each pixel in the overlapping area, and obtain the stitching weight parameter.

[0106] Please refer to Fig.10Based on the same inventive concept, the second aspect of the embodiment of the present application provides a vehicle-mounted panoramic image calibration device, and the vehicle-mounted panoramic image calibration device 10 includes:

[0107] The calibration point selection module 101 is used to obtain four view images collected by the vehicle to be calibrated in a preset calibration site, and determine the target calibration point in each view image in response to the user's calibration operation;

[0108] The first calibration module 102 is used to determine the coordinate conversion parameters corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead image corresponding to the view image; the coordinate conversion parameters are used to process the view image to obtain the overhead image corresponding to the view image;

[0109] The second calibration module 103 is used to calibrate the overlapping area of ​​the four overhead images and determine the stitching weight parameter of the overlapping area between any two adjacent overhead images; the stitching weight parameter is used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

[0110] Optionally, the calibration point selection module 101 includes:

[0111] an initial identification submodule, configured to identify a plurality of original calibration points in a view image in response to a first selection instruction of a user for any view image;

[0112] an enlargement submodule, configured to enlarge a preset area including the original calibration point in response to a second selection instruction of the user for any original calibration point, to obtain a local enlarged image;

[0113] The adjustment submodule is used to move the original calibration point in the local magnified image in response to the user's movement instruction for the original calibration point, so as to obtain the target calibration point corresponding to the original calibration point.

[0114] Optionally, the calibration point selection module 101 further includes:

[0115] The unit selection submodule is used to determine the unit movement distance corresponding to the original calibration point in the local magnified image in response to the user's movement coefficient selection instruction for the original calibration point.

[0116] Optionally, the coordinate transformation parameters include a distortion mapping matrix and a perspective transformation matrix; the first calibration module 102 includes:

[0117] A dedistortion submodule, configured to determine a distortion mapping matrix corresponding to the view image based on original coordinates of the target calibration points in the view image and calibration coordinates of the target calibration points in the dedistorted image corresponding to the view image;

[0118] The perspective transformation submodule is used to determine the perspective transformation matrix corresponding to the view image based on the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image and the calibration coordinates of the target calibration point in the overhead image corresponding to the view image.

[0119] Optionally, the second calibration module 103 includes:

[0120] A custom contour submodule is used to mark the to-be-joined areas of the four overhead images in response to the user's joint command;

[0121] The overlapping submodule is used to determine the overlapping area between two overhead images based on the to-be-joined area corresponding to any two adjacent overhead images.

[0122] Optionally, the overlapping area is a polygonal area containing an acute angle; the second calibration module 103 further includes:

[0123] A weight determination submodule, for determining a weight value corresponding to any pixel point in the overlapping area based on the ratio of the distance between the pixel point and the two acute angle edges of the overlapping area;

[0124] The weight preservation submodule is used to form a splicing weight parameter based on the weight values ​​corresponding to all pixels in the overlapping area.

[0125] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment of the first aspect.

[0126] The third aspect of the present application provides a method for generating a vehicle-mounted panoramic image. Fig.11 , Fig.11 FIG. 1 is a flow chart of a method for generating a vehicle-mounted panoramic image according to an embodiment of the present application. Fig.11 As shown, the method comprises the following steps:

[0127] S301: Acquire four target view images of the target vehicle;

[0128] S302: for any target view image, processing the target view image based on the coordinate conversion parameters corresponding to the target view image to obtain a target overhead view image corresponding to the target view image;

[0129] S303: based on the stitching weight parameter corresponding to the overlapping area between any two adjacent target bird's-eye view images, stitching and fusing the four target bird's-eye view images to obtain a target vehicle-mounted panoramic image;

[0130] The coordinate conversion parameters and the stitching weight parameters are obtained by the vehicle-mounted panoramic image calibration method in the first aspect.

[0131] In this embodiment, after the calibration conversion is performed based on the selected calibration points in the above manner and the coordinate conversion parameters and the stitching weight parameters are obtained, in actual application, the four target view images of the target vehicle can be directly processed using the coordinate conversion parameters to obtain the target overhead view image corresponding to the target view image, and the four target overhead view images can be stitched and fused using the stitching weight parameters to obtain the target vehicle-mounted panoramic image. This ensures the conversion accuracy and stitching fusion effect of the four view images and improves the quality of the final vehicle-mounted panoramic image.

[0132] A fourth aspect of the present application provides a vehicle-mounted panoramic image generation device, the vehicle-mounted panoramic image generation device 12 comprising:

[0133] A view acquisition module 121 is used to acquire four target view images of the target vehicle;

[0134] The image processing module 122 is used to process any target view image based on the coordinate conversion parameters corresponding to the target view image to obtain a target bird's-eye view image corresponding to the target view image;

[0135] An image stitching module 123 is used to stitch and fuse the four target overhead images based on a stitching weight parameter corresponding to an overlapping area between any two adjacent target overhead images to obtain a target vehicle-mounted panoramic image;

[0136] The coordinate conversion parameters and the stitching weight parameters are obtained by the vehicle-mounted panoramic image calibration method in the first aspect.

[0137] In the fifth aspect, based on the same inventive concept, an embodiment of the present application provides a storage medium, which stores machine executable instructions. When the machine executable instructions are executed by a processor, the vehicle-mounted panoramic image calibration method proposed in the first aspect of the present application is implemented.

[0138] It should be noted that the specific implementation of the storage medium of the embodiment of the present application refers to the specific implementation of the vehicle-mounted panoramic image calibration method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.

[0139] In the sixth aspect, based on the same inventive concept, refer to Fig.13 An embodiment of the present application provides a vehicle 13, including a processor 131 and a memory 132; the memory 132 stores machine executable instructions that can be executed by the processor 131, and the processor 131 is used to execute the machine executable instructions to implement the vehicle-mounted panoramic image generation method proposed in the third aspect of the present application.

[0140] It should be noted that the specific implementation of the vehicle 13 in the embodiment of the present application refers to the specific implementation of the vehicle-mounted panoramic image generation method proposed in the second aspect of the embodiment of the present application, and will not be repeated here.

[0141] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0142] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0146] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0147] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0148] The above is a detailed introduction to the vehicle-mounted panoramic image calibration method, generation method, device, medium and vehicle provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A vehicle-mounted panoramic image calibration method, characterized in that: The method comprises: Acquire four view images collected by the vehicle to be calibrated in a preset calibration site, and determine a target calibration point in each of the view images in response to a user's calibration operation; Determine, based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image, coordinate conversion parameters corresponding to the view image; the coordinate conversion parameters are used to process the view image to obtain the overhead view image corresponding to the view image; The overlapping area of ​​the four overhead images is calibrated to determine the stitching weight parameters of the overlapping area between any two adjacent overhead images; the stitching weight parameters are used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

2. The method according to claim 1, characterized in that In response to a calibration operation by a user, determining a target calibration point in each of the view images comprises: In response to a first selection instruction of a user for any of the view images, identifying a plurality of original calibration points in the view image; In response to a second selection instruction of the user for any of the original calibration points, a preset area including the original calibration point is enlarged to obtain a local enlarged image; In response to a user's movement instruction for the original calibration point, the original calibration point is moved in the partial magnified image to obtain the target calibration point corresponding to the original calibration point.

3. The method according to claim 2, characterized in that Before moving the original calibration point in the local enlarged image, the method further includes: In response to a user's instruction to select a movement coefficient for the original calibration point, a unit movement distance corresponding to the original calibration point in the partial magnified image is determined.

4. The method according to claim 1, characterized in that The coordinate transformation parameters include a distortion mapping matrix and a perspective transformation matrix; Determining coordinate conversion parameters corresponding to the view image based on original coordinates of the target calibration point in the view image and calibration coordinates of the target calibration point in the overhead view image corresponding to the view image includes: Determine a distortion mapping matrix corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image; Based on the calibration coordinates of the target calibration point in the dedistorted image corresponding to the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image, a perspective transformation matrix corresponding to the view image is determined.

5. The method according to claim 1, characterized in that Calibrate the overlapping area of ​​the four overhead images, including: In response to a stitching instruction from a user, respectively marking the to-be-stitched areas of the four overhead images; Based on the to-be-joined areas corresponding to any two adjacent overhead images, an overlapping area between the two overhead images is determined.

6. The method according to claim 5, characterized in that The overlapping area is a polygonal area containing acute angles; Determine the stitching weight parameters of the overlapping area between any two adjacent overhead images, including: For any pixel point in the overlapping area, determine a weight value corresponding to the pixel point based on a ratio of distances between the pixel point and two acute-angle edges of the overlapping area; The splicing weight parameter is formed based on the weight values ​​corresponding to all the pixels in the overlapping area.

7. A method for generating a vehicle-mounted panoramic image, characterized in that: The method comprises: Acquire four target view images of the target vehicle; For any target view image, the target view image is processed based on the coordinate conversion parameters corresponding to the target view image to obtain a target overhead view image corresponding to the target view image; Based on the stitching weight parameter corresponding to the overlapping area between any two adjacent target overhead images, the four target overhead images are stitched and fused to obtain a target vehicle-mounted panoramic image; Wherein, the coordinate conversion parameters and the stitching weight parameters are obtained by the vehicle-mounted panoramic image calibration method according to any one of claims 1 to 6.

8. A vehicle-mounted panoramic image calibration device, characterized in that: The device comprises: A calibration point selection module, used to obtain four view images collected by the vehicle to be calibrated in a preset calibration site, and determine a target calibration point in each of the view images in response to a user's calibration operation; A first calibration module, configured to determine a coordinate conversion parameter corresponding to the view image based on the original coordinates of the target calibration point in the view image and the calibration coordinates of the target calibration point in the overhead view image corresponding to the view image; the coordinate conversion parameter is used to process the view image to obtain the overhead view image corresponding to the view image; The second calibration module is used to calibrate the overlapping area of ​​the four overhead images and determine the stitching weight parameters of the overlapping area between any two adjacent overhead images; the stitching weight parameters are used to stitch and fuse the four overhead images to obtain a vehicle-mounted panoramic image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle-mounted panoramic image calibration method as described in any one of claims 1 to 6 is implemented.

10. A vehicle, characterized in that: It includes a processor and a memory; the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the vehicle-mounted panoramic image generation method as described in claim 7.

Images