A panoramic surround view image stitching method and device, electronic equipment and storage medium

Abstract

The application discloses a panoramic surround view image splicing method and device, electronic equipment and a storage medium. The method comprises the following steps: collecting images around a vehicle through cameras installed around the vehicle; performing distortion correction, perspective transformation and image splicing processing on the images collected by the cameras according to camera calibration parameters by using an OpenGL shader to obtain fused panoramic surround view images; when the system works for a period of time, if a splicing seam is found in the panoramic surround view images, the camera calibration parameters are corrected, and the corrected camera calibration parameters are input into the OpenGL shader; and the images collected by the cameras are processed, such as image splicing, according to the corrected camera calibration parameters by using the OpenGL shader to obtain fused panoramic surround view images. The application has high splicing speed, small limitation, good universality, supports four or more camera image splicing, and can guarantee the consistency and stability of the splicing effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a panoramic panoramic image stitching method, apparatus, electronic device, and storage medium. Background Technology

[0002] A panoramic surround view system is an advanced vision-assistance technology used in automotive, security, and industrial fields to provide a comprehensive surround view. It stitches together images from multiple wide-angle cameras in real time to present a 360-degree panoramic view of the vehicle's surroundings, allowing drivers or operators to better understand their environment and improving safety and efficiency.

[0003] Currently, panoramic surround view systems typically consist of four to six wide-angle cameras mounted at the front, rear, left, right, and top of the vehicle. These wide-angle cameras capture images of the vehicle's surroundings, each covering a specific field of view, thus enabling omnidirectional monitoring of the vehicle's environment. Each wide-angle camera transmits its captured image to the main control unit via an interface. The main control unit uses image processing algorithms to process and stitch the received images. Stitching algorithms typically include distortion correction, perspective transformation, and image stitching. These algorithms can stitch images captured by multiple wide-angle cameras into a continuous panoramic surround view image.

[0004] As camera resolution increases, the computational load for image stitching increases significantly, making real-time image stitching impossible with traditional CPUs. To address this, some researchers have proposed using the CUDA parallel computing architecture to implement image stitching algorithms, leveraging the high processing power of image processors to dramatically improve computational efficiency. However, this method only considers four-camera image stitching, and CUDA is a computing architecture specific to NVIDIA graphics cards, which has certain limitations. Furthermore, existing panoramic image stitching methods primarily focus on optimizing camera calibration before stitching, neglecting the issue of poor stitching results caused by camera pose shifts during operation.

[0005] In summary, given the aforementioned problems encountered in the process of panoramic image stitching and calibration correction, it is essential to research a more universal and efficient image stitching method and to ensure that the stitching effect still meets the requirements after camera position shift. Summary of the Invention

[0006] This application provides a preferred embodiment of a panoramic image stitching method to solve the technical problems of existing image stitching algorithms, which rely on hardware performance, resulting in certain limitations, poor versatility, and difficulty in ensuring the consistency and stability of the stitching effect after camera position shift.

[0007] The technical solution adopted in this application is as follows:

[0008] A method for stitching panoramic images, comprising the following steps:

[0009] Images of the area around the vehicle are captured by cameras installed around the vehicle.

[0010] Using OpenGL shaders, distortion correction, perspective transformation, and image stitching are performed on the images acquired by each camera according to the camera calibration parameters to obtain a fused panoramic view image.

[0011] If stitching seams are found in the panoramic view image after the system has been working for a period of time, the camera calibration parameters are corrected and the corrected camera calibration parameters are input into the OpenGL shader.

[0012] Using OpenGL shaders, distortion correction, perspective transformation, and image stitching are performed on the images captured by the camera based on the corrected camera calibration parameters to obtain a fused panoramic view image.

[0013] Furthermore, before acquiring images of the area around the vehicle using cameras installed around the vehicle, the process includes the following steps:

[0014] The camera is calibrated, and the camera calibration parameters are calculated. The camera calibration parameters include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information.

[0015] Further, the camera is calibrated, and the camera calibration parameters are calculated. These parameters include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information. Specifically, this includes the following steps:

[0016] Cameras installed around the vehicle capture images of the calibration board from different angles. The corner points of the calibration board in the images are detected to obtain the pixel coordinates of each corner point. The physical coordinates of the corner points of the calibration board are calculated based on the size of the checkerboard and the origin of the world coordinate system.

[0017] The camera intrinsic parameter matrix and distortion parameters are calibrated using the acquired calibration board images;

[0018] Place a black and white checkerboard pattern in front of the camera mounting position, measure its coordinates in the world coordinate system, select the corner image coordinates in the image containing the checkerboard pattern, establish the mapping relationship from the corrected image to the bird's-eye view, and solve the homography matrix.

[0019] The camera pose information is obtained by using the coordinates in the world coordinate system and the image coordinate system.

[0020] Furthermore, the step of calibrating the camera intrinsic parameter matrix and distortion parameters using the acquired calibration board image specifically includes the following steps:

[0021] For each calibration board image, the homography matrix between physical coordinates and pixel coordinates is solved by considering their relationship.

[0022] The camera intrinsic parameter matrix is ​​solved by inversely solving the homography matrix.

[0023] Ignoring tangential distortion and intrinsic parameter matrix errors, the distortion parameters are solved by inversely using the obtained intrinsic parameter matrix;

[0024] The parameters are optimized using the nonlinear least squares method to obtain the final intrinsic parameter matrix and distortion parameters.

[0025] Furthermore, when calibrating the camera, a general camera model, the KB model, is used. This model describes the correspondence between distorted pixels and corrected pixels, expressed by the formula:

[0026] u = f x ·x·scale+c x v = f y ·y·scale+c y , where scale = θ d / r

[0027] Where (u,v) represents the coordinates of the corrected pixel, (x,y) represents the coordinates of the distorted pixel in the camera coordinate system, and (f x ,f y (c) represents the pixel length of the focal length in the x and y directions. x ,c y ) represents the distortion center, r represents the coordinates of the distorted pixel from the center in the camera coordinate system, and θ represents the distortion center. d The distorted incident angle is calculated using the following formula:

[0028] θ d =θ(1+k1θ) 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0029] Where (k1, k2, k3, k4) are the calculated distortion parameters.

[0030] Furthermore, when the distances from the camera center to the overlapping region in two adjacent image regions are equal, the image fusion region between the two adjacent image regions is processed using a trigonometric function weight fusion algorithm for image stitching. The calculation formula is as follows:

[0031]

[0032] In the formula, θ is the trigonometric function value of the distance from the pixel to the boundary of the overlapping region, and d i For pixel to region I A The distance to the boundary, d is the distance of the overlapping region, I A (i,j) represents the pixel region I. A The pixel value in I B (i,j) represents the pixel region I. B The pixel values ​​in the matrix, where I(i,j) are the pixel values ​​after pixel fusion;

[0033] When the difference in distance from the camera center to the overlapping region between two adjacent image regions exceeds a set threshold, the image fusion region between the two adjacent image regions is processed using a dual-distance weighted fusion algorithm for image stitching. The dual distances are: the distance from a pixel to the boundary of the overlapping region and the distance from a pixel to the camera center. The calculation formula is as follows:

[0034]

[0035] In the formula, w(i,j) represents the distance from pixel to region I. A The weight value, d A1 ,d A2 For pixels to coincident regions I A The distance between the boundary and the camera center, d B1 ,d B2 For pixels to coincident regions I B The distance between the boundary and the camera center, I A (i,j) represents the pixel region I. A The pixel value in I B (i,j) represents the pixel region I. B The pixel values ​​in the graph are I(i,j), where I(i,j) is the pixel value after pixel fusion.

[0036] Furthermore, if a stitching seam is detected in the panoramic view image after the system has been working for a period of time, the camera calibration parameters are corrected, and the corrected camera calibration parameters are input into the OpenGL shader. This specifically includes the following steps:

[0037] When there are stitching seams exceeding the threshold in the panoramic panoramic image display area, measure the coordinates of the black and white checkerboard placed in front of the camera in the world coordinate system, and calculate the offset camera pose information.

[0038] The transformation matrix between the offset camera pose information and the initial camera pose information is obtained, and the camera calibration parameters are corrected using the transformation matrix.

[0039] Finally, the corrected camera calibration parameters are input into the OpenGL shader.

[0040] Another preferred embodiment of this application provides a panoramic surround view image stitching device, including:

[0041] The image acquisition module is used to acquire images of the area around the vehicle using cameras installed around the vehicle.

[0042] The panoramic image fusion module is used to perform distortion correction, perspective transformation and image stitching on the images acquired by each camera according to the camera calibration parameters using OpenGL shaders to obtain the fused panoramic image.

[0043] The camera calibration parameter correction module is used to correct the camera calibration parameters if stitching seams are found in the panoramic view image after the system has been working for a period of time, and then input the corrected camera calibration parameters into the OpenGL shader.

[0044] The panoramic panoramic image correction and fusion module uses OpenGL shaders to perform distortion correction, perspective transformation, and image stitching on the images captured by the camera based on the corrected camera calibration parameters, in order to obtain the fused panoramic panoramic image.

[0045] Another preferred embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the panoramic surround view image stitching method.

[0046] Another preferred embodiment of this application also provides a storage medium including a stored program that, when the program is executed, controls the device where the storage medium is located to perform the steps of the panoramic surround view image stitching method.

[0047] Compared with the prior art, this application has the following advantages:

[0048] 1. This application has a high stitching speed. Because it is based on OpenGL image processing technology, it is not limited to NVIDIA cards and can realize real-time panoramic image stitching on a wider range of image processor platforms, with good versatility.

[0049] 2. This application supports image stitching from four or more fisheye cameras, expanding its application scope and flexibility;

[0050] 3. This application employs different fusion algorithms for different overlapping regions, including trigonometric function weighted fusion algorithm and dual distance weighted fusion algorithm, thereby improving the quality of the fused panoramic surround view image.

[0051] 4. This application provides calibration parameter correction measures for camera position offset to ensure the consistency and stability of subsequent stitching effects.

[0052] In addition to the purposes, features, and advantages described above, this application provides other purposes, features, and advantages. The application will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0053] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0054] Figure 1 This is a schematic flowchart of a panoramic surround view image stitching method according to a preferred embodiment of this application.

[0055] Figure 2 This is a schematic diagram of the installation position of the fisheye camera according to a preferred embodiment of this application.

[0056] Figure 3 This is a schematic diagram of a panoramic surround view image stitching method according to another preferred embodiment of this application.

[0057] Figure 4 This is a flowchart illustrating a sub-step of step S1 in a preferred embodiment of this application.

[0058] Figure 5 This is a flowchart illustrating a sub-step of step S12 in a preferred embodiment of this application.

[0059] Figure 6 This is a schematic diagram of the division of each region in the panoramic view image in a preferred embodiment of this application.

[0060] Figure 7 This is a flowchart illustrating a sub-step of step S4 in a preferred embodiment of this application.

[0061] Figure 8 This is a schematic diagram of a panoramic surround view image stitching device module according to a preferred embodiment of this application.

[0062] Figure 9 This is a schematic diagram of a panoramic surround view image stitching device module according to another preferred embodiment of this application.

[0063] Figure 10 This is a schematic block diagram of an electronic device according to a preferred embodiment of this application.

[0064] Figure 11 This is an internal structural diagram of a computer device according to a preferred embodiment of this application. Detailed Implementation

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0066] like Figure 1 As shown, a preferred embodiment of this application provides a panoramic surround view image stitching method, including the following steps:

[0067] S2. Images of the area around the vehicle are captured by cameras installed around the vehicle, such as... Figure 2 As shown, in this embodiment, six fisheye cameras are installed around the vehicle to capture images of the surrounding area, ensuring that there are no blind spots around the vehicle.

[0068] S3. Using OpenGL shaders, distortion correction, perspective transformation, and image stitching are performed on the images acquired by each camera according to the camera calibration parameters to obtain the fused panoramic view image.

[0069] S4. After the system has been working for a period of time, if a stitching seam is found in the panoramic view image, the camera calibration parameters are corrected and the corrected camera calibration parameters are input into the OpenGL shader.

[0070] S5. Using OpenGL shaders, the images captured by the camera are processed for distortion correction, perspective transformation, and image stitching based on the corrected camera calibration parameters to obtain a fused panoramic view image.

[0071] Compared with the prior art, this embodiment has the following beneficial effects:

[0072] 1. This application has a high stitching speed. Because it is based on OpenGL image processing technology, it is not limited to NVIDIA cards and can realize real-time panoramic image stitching on a wider range of image processor platforms, with good versatility.

[0073] 2. This application supports image stitching from four or more fisheye cameras, expanding its application scope and flexibility;

[0074] 3. This application employs different fusion algorithms for different overlapping regions, including trigonometric function weighted fusion algorithm and dual distance weighted fusion algorithm, thereby improving the quality of the fused panoramic surround view image.

[0075] 4. This application provides calibration parameter correction measures for camera position offset to ensure the consistency and stability of subsequent stitching effects.

[0076] Preferably, such as Figure 3 As shown, before acquiring images of the area around the vehicle using cameras installed around the vehicle, the process also includes the following steps:

[0077] S1. The camera is calibrated, and the camera calibration parameters are calculated. The camera calibration parameters include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information.

[0078] Preferably, such as Figure 4 As shown, in step S1, the camera is calibrated, and the camera calibration parameters are calculated. These parameters include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information. Specifically, this includes the following steps:

[0079] S11. Use cameras installed around the vehicle to take images of the calibration board from different angles, detect the corner points of the calibration board in the images, obtain the pixel coordinate values ​​of each corner point, and calculate the physical coordinates of the corner points of the calibration board based on the size of the checkerboard and the origin of the world coordinate system.

[0080] S12. Use the acquired calibration board images to calibrate the camera intrinsic parameter matrix and distortion parameters;

[0081] S13. Place a black and white checkerboard pattern in front of the camera mounting position, measure its coordinates in the world coordinate system, select the corner point image coordinates in the image containing the checkerboard pattern, establish the mapping relationship from the corrected image to the bird's-eye view, and solve the homography matrix.

[0082] S14. Obtain the camera pose information based on the coordinates in the world coordinate system and the image coordinate system.

[0083] Preferably, such as Figure 5 As shown, step S12, which involves calibrating the camera intrinsic parameter matrix and distortion parameters using the acquired calibration board image, specifically includes the following steps:

[0084] S121. For each calibration board image, solve the homography matrix between the physical coordinates and pixel coordinates.

[0085] S122. Solve the camera intrinsic parameter matrix using the homography matrix;

[0086] S123. Ignore tangential distortion and intrinsic parameter matrix error, and use the obtained intrinsic parameter matrix to solve for the distortion parameters.

[0087] S124. The above parameters are optimized using the nonlinear least squares method to obtain the final intrinsic parameter matrix and distortion parameters.

[0088] Preferably, when calibrating the camera, a general camera model, the KB model, is used. This model describes the correspondence between distorted pixels and corrected pixels, expressed by the formula:

[0089] u = f x ·x·scale+c x v = fy ·y·scale+c y , where scale = θ d / r

[0090] Where (u,v) represents the coordinates of the corrected pixel, (x,y) represents the coordinates of the distorted pixel in the camera coordinate system, and (f x ,f y (c) represents the pixel length of the focal length in the x and y directions. x ,c y ) represents the distortion center, r represents the coordinates of the distorted pixel from the center in the camera coordinate system, and θ represents the distortion center. d The distorted incident angle is calculated using the following formula:

[0091] θ d =θ(1+k1θ) 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0092] Where (k1, k2, k3, k4) are the calculated distortion parameters.

[0093] Preferably, since the distances from the camera's mounting position to the overlapping area vary, if the camera's center is closer to the overlapping area, the image captured by that camera will be superior to the image captured by the other camera. Therefore, considering the distance from the camera's center to the overlapping area when performing image stitching and fusion will significantly improve the quality of the stitched image. Figure 6 As shown, there are two types of image fusion areas in the panoramic view image: the angle between the front-view area and the left-view or right-view area is 90°, and the angle between left-view area 1# and left-view area 2# is 180°. Generally, the distances from the camera center of left-view area 1# and left-view area 2# to the overlapping area are equal (if the difference is large, a dual-distance weighted fusion algorithm can be used). Therefore, this embodiment uses different fusion algorithms for different overlapping areas, wherein:

[0094] When the distances from the camera center to the overlapping area in two adjacent image regions are equal, the image fusion region between the two adjacent image regions (i.e., the image fusion region between left view region 1# and left view region 2#, and the image fusion region between right view region 1# and right view region 2#) is processed using a trigonometric function weight fusion algorithm to eliminate the stitching seams. The calculation formula is as follows:

[0095]

[0096] In the formula, θ is the trigonometric function value of the distance from the pixel to the boundary of the overlapping region, and d i For pixel to region I AThe distance to the boundary, d is the distance of the overlapping region, I A (i,j) represents the pixel region I. A The pixel value in I B (i,j) represents the pixel region I. B The pixel values ​​in the matrix, where I(i,j) are the pixel values ​​after pixel fusion;

[0097] When the difference in distance from the camera center to the overlapping area between two adjacent image regions exceeds a set threshold, the image fusion region between the two adjacent image regions (i.e., the image fusion region between the front view region and left view region 1#, the image fusion region between the front view region and right view region 1#, the image fusion region between the rear view region and left view region 2#, and the image fusion region between the rear view region and right view region 2#) is processed using a dual-distance weighted fusion algorithm to eliminate the stitching seam. The dual distances are: the distance from the pixel to the boundary of the overlapping area and the distance from the pixel to the camera center, calculated as follows:

[0098]

[0099] In the formula, w(i,j) represents the distance from pixel to region I. A The weight value, d A1 ,d A2 For pixels to coincident regions I A The distance between the boundary and the camera center, d B1 ,d B2 For pixels to coincident regions I B The distance between the boundary and the camera center, I A (i,j) represents the pixel region I. A The pixel value in I B (i,j) represents the pixel region I. B The pixel values ​​in the graph are I(i,j), where I(i,j) is the pixel value after pixel fusion.

[0100] This embodiment employs different fusion algorithms to eliminate stitching seams for different overlapping areas, including trigonometric function weighted fusion algorithm and dual-distance weighted fusion algorithm, thereby obtaining better panoramic surround view image quality.

[0101] Preferably, such as Figure 7 As shown, in step S4, if a stitching seam is found in the panoramic view image after the system has been working for a period of time, the camera calibration parameters are corrected, and the corrected camera calibration parameters are input into the OpenGL shader. This specifically includes the following steps:

[0102] S41. When there are stitching seams exceeding the threshold in the panoramic view image display area, measure the coordinates of the black and white checkerboard placed in front of the camera in the world coordinate system, and calculate the camera pose information after offset.

[0103] S42. Based on the offset camera pose information and the initial camera pose information, obtain the transformation matrix between the two, and use the transformation matrix to correct the camera calibration parameters.

[0104] S43. Finally, input the corrected camera calibration parameters into the OpenGL shader.

[0105] Because the working environment of a panoramic surround-view system may experience significant vibrations, the camera's posture may shift after a period of operation due to insecure camera mounting. When the operator observes a large stitching seam in the panoramic bird's-eye view display area, it indicates that the camera posture has shifted, requiring calibration parameter correction. This embodiment uses the following correction method: A black and white checkerboard pattern is placed in front of the camera, and its coordinates in the world coordinate system are measured to calculate the shifted camera pose information. Based on the existing and initial camera pose information, a transformation matrix between them can be obtained, and this transformation matrix is ​​used to correct the calibration parameters. Finally, the corrected parameters are input into the OpenGL shader for image distortion correction, perspective transformation, and image stitching. This embodiment, by providing calibration parameter correction measures for camera position shift, ensures the consistency and stability of the subsequent stitching effect, avoiding large stitching seams after image stitching due to camera displacement.

[0106] like Figure 8 As shown, another preferred embodiment of this application also provides a panoramic surround view image stitching device, including:

[0107] The image acquisition module is used to acquire images of the area around the vehicle using cameras installed around the vehicle.

[0108] The panoramic image fusion module is used to perform distortion correction, perspective transformation and image stitching on the images acquired by each camera according to the camera calibration parameters using OpenGL shaders to obtain the fused panoramic image.

[0109] The camera calibration parameter correction module is used to correct the camera calibration parameters if stitching seams are found in the panoramic view image after the system has been working for a period of time, and input the corrected camera calibration parameters into the OpenGL shader.

[0110] The panoramic panoramic image correction and fusion module uses OpenGL shaders to perform distortion correction, perspective transformation, and image stitching on the images captured by the camera based on the corrected camera calibration parameters, in order to obtain the fused panoramic panoramic image.

[0111] like Figure 9 As shown, unlike the embodiments described above, the panoramic surround view image stitching device further includes:

[0112] The camera calibration module is used to calibrate the camera and calculate the camera calibration parameters, which include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information.

[0113] like Figure 10 As shown, a preferred embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the panoramic surround view image stitching method in the above embodiments.

[0114] like Figure 11 As shown in the preferred embodiment of this application, a computer device is also provided. This computer device may be a terminal or a liveness detection server, and its internal structure diagram may be as follows. Figure 11 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with other external computer devices via a network connection. When the computer program is executed by the processor, it implements the steps of the panoramic surround-view image stitching method described above.

[0115] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] A preferred embodiment of this application also provides a storage medium, the storage medium including a stored program, which, when the program is executed, controls the device where the storage medium is located to perform the steps of the panoramic surround view image stitching method in the above embodiments.

[0117] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0118] If the functions described in this embodiment are implemented as software functional units and sold or used as independent products, they can be stored in one or more computing device-readable storage media. Based on this understanding, the parts of this application's embodiments that contribute to the prior art or the technical solutions can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, mobile computing device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language C++ and the interpreted scripting language Python.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0124] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for stitching panoramic panoramic images, characterized in that, Including the following steps: Images of the area around the vehicle are captured by cameras installed around the vehicle. Using OpenGL shaders, distortion correction, perspective transformation, and image stitching are performed on the images acquired by each camera according to the camera calibration parameters to obtain a fused panoramic view image. If stitching seams are found in the panoramic view image after the system has been working for a period of time, the camera calibration parameters are corrected and the corrected camera calibration parameters are input into the OpenGL shader. Using OpenGL shaders, distortion correction, perspective transformation, and image stitching are performed on the images captured by the camera based on the corrected camera calibration parameters to obtain a fused panoramic view image. When the distances from the camera center to the overlapping area are equal in two adjacent image regions, the image fusion region between the two adjacent image regions is processed using a trigonometric function weighted fusion algorithm for image stitching. The calculation formula is as follows: ； In the formula, θ This is the trigonometric function value of the distance from the pixel to the boundary of the overlapping region. d i pixel to region I A Distance to the boundary d The distance between the overlapping regions. I A (i,j) pixel area I A Pixel values ​​in I B (i,j) pixel area I B Pixel values ​​in I(i,j) The pixel value after pixel fusion; When the difference in distance from the camera center to the overlapping region between two adjacent image regions exceeds a set threshold, the image fusion region between the two adjacent image regions is processed using a dual-distance weighted fusion algorithm for image stitching. The dual distances are: the distance from a pixel to the boundary of the overlapping region and the distance from a pixel to the camera center. The calculation formula is as follows: ； In the formula, w(i,j) pixel to region I A The weight value, d A1 ,d A2 For pixels to overlapping areas I A The distance between the boundary and the camera center d B1 ,d B2 For pixels to overlapping areas I B The distance between the boundary and the camera center I A (i,j) For pixel region I A Pixel values ​​in I B (i,j) pixel area I B Pixel values ​​in I(i,j) The pixel value after pixel fusion; If, after the system has been operating for a period of time, stitching seams are found in the panoramic view image, the camera calibration parameters are corrected, and the corrected camera calibration parameters are input into the OpenGL shader. The specific steps include: When there are stitching seams exceeding the threshold in the panoramic panoramic image display area, measure the coordinates of the black and white checkerboard placed in front of the camera in the world coordinate system, and calculate the offset camera pose information. The transformation matrix between the offset camera pose information and the initial camera pose information is obtained, and the camera calibration parameters are corrected using the transformation matrix. Finally, the corrected camera calibration parameters are input into the OpenGL shader.

2. The panoramic surround view image stitching method according to claim 1, characterized in that, Before acquiring images of the area around the vehicle using cameras mounted around the vehicle, the process includes the following steps: The camera is calibrated, and the camera calibration parameters are calculated. The camera calibration parameters include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information.

3. The panoramic surround view image stitching method according to claim 2, characterized in that, The camera calibration process involves calculating the camera calibration parameters, which include the camera intrinsic parameter matrix, distortion parameters, homography matrix, and camera pose information. The specific steps include: Cameras installed around the vehicle capture images of the calibration board from different angles. The corner points of the calibration board in the images are detected to obtain the pixel coordinates of each corner point. The physical coordinates of the corner points of the calibration board are calculated based on the size of the checkerboard and the origin of the world coordinate system. The camera intrinsic parameter matrix and distortion parameters are calibrated using the acquired calibration board images; Place a black and white checkerboard pattern in front of the camera mounting position, measure its coordinates in the world coordinate system, select the corner image coordinates in the image containing the checkerboard pattern, establish the mapping relationship from the corrected image to the bird's-eye view, and solve the homography matrix. The camera pose information is obtained by using the coordinates in the world coordinate system and the image coordinate system.

4. The panoramic surround view image stitching method according to claim 3, characterized in that, The calibration of the camera intrinsic parameter matrix and distortion parameters using the acquired calibration board images specifically includes the following steps: For each calibration board image, the homography matrix between physical coordinates and pixel coordinates is solved by considering their relationship. The camera intrinsic parameter matrix is ​​solved by inversely solving the homography matrix. Ignoring tangential distortion and intrinsic parameter matrix errors, the distortion parameters are solved by inversely using the obtained intrinsic parameter matrix; The parameters are optimized using the nonlinear least squares method to obtain the final intrinsic parameter matrix and distortion parameters.

5. The panoramic surround view image stitching method according to claim 2, characterized in that, When calibrating the camera, a general camera model, the KB model, is used. This model describes the correspondence between distorted pixels and corrected pixels, expressed by the formula: ； Where (u,v) represents the coordinates of the corrected pixel, (x,y) represents the coordinates of the distorted pixel in the camera coordinate system, and (f x ,f y (c) represents the pixel length of the focal length in the x, y directions. x ,c y () represents the distortion center, and r represents the coordinates of the distorted pixel from the center in the camera coordinate system. θ d The distorted incident angle is calculated using the following formula: ； Where (k1, k2, k3, k4) are the calculated distortion parameters.

6. A panoramic surround view image stitching device, used to implement the panoramic surround view image stitching method as described in any one of claims 1 to 5, characterized in that, The image acquisition module is used to acquire images of the area around the vehicle using cameras installed around the vehicle. The panoramic image fusion module is used to perform distortion correction, perspective transformation and image stitching on the images acquired by each camera according to the camera calibration parameters using OpenGL shaders to obtain the fused panoramic image. The camera calibration parameter correction module is used to correct the camera calibration parameters if stitching seams are found in the panoramic view image after the system has been working for a period of time, and input the corrected camera calibration parameters into the OpenGL shader. The panoramic panoramic image correction and fusion module uses OpenGL shaders to perform distortion correction, perspective transformation, and image stitching on the images captured by the camera based on the corrected camera calibration parameters, in order to obtain the fused panoramic panoramic image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the panoramic surround view image stitching method as described in any one of claims 1 to 5.

8. A storage medium comprising a stored program that, when the program is executed, controls a device in which the storage medium resides to perform the steps of the panoramic surround view image stitching method as described in any one of claims 1 to 5.

Images