Panoramic image stitching method, electronic device, storage medium and program product

Abstract

The invention discloses a panoramic image stitching method, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a first image shot by a first camera and a second image shot by a second camera, determining a first overlapping area overlapped with the second image in the first image and a second overlapping area overlapped with the first image in the second image; calculating a first average RGB value of the first overlapping region and a second average RGB value of the second overlapping region; respectively calculating correction coefficients of the first camera and the second camera according to the first average RGB value and the second average RGB value; and splicing the shot images of the first camera and the second camera based on the correction coefficients of the first camera and the second camera. According to the method, the images shot by the first camera and the second camera are spliced based on the correction coefficient obtained through calculation, and the problem that the color difference at the splicing position is obvious when the overlapping areas of the images shot by the adjacent cameras are spliced is solved.

Description

Technical Field

[0001] This application relates to the field of panoramic imaging technology, and in particular to a panoramic image stitching method, electronic device, storage medium, and program product. Background Technology

[0002] 360° surround-view imaging for vehicles is gradually becoming a research hotspot in the automotive field. It typically involves stitching together multiple images output from cameras distributed around the vehicle to generate a panoramic view around the vehicle. Most in-vehicle panoramic imaging systems support rendering and stitching together bird's-eye views and 3D views. However, during vehicle operation, differences in the physical parameters of each camera and the surrounding environment lead to color variations between the original images. This results in noticeable boundary differences in the stitched view, severely impacting the user experience. Summary of the Invention

[0003] This application provides a panoramic image stitching method, electronic device, storage medium, and program product to at least solve one of the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a panoramic image stitching method, including:

[0005] Acquire a first image captured by a first camera and a second image captured by a second camera, and determine a first overlapping region in the first image that overlaps with the second image, and a second overlapping region in the second image that overlaps with the first image;

[0006] Calculate the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region;

[0007] The correction coefficients for the first camera and the second camera are calculated based on the first average RGB value and the second average RGB value, respectively.

[0008] The images captured by the first camera and the second camera are stitched together based on the correction coefficients of the first camera and the second camera.

[0009] In some embodiments, before determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, the method further includes:

[0010] Determine the overlapping area in the top view corresponding to the first camera and the second camera;

[0011] Based on the intrinsic and extrinsic parameters of the first and second cameras and the world coordinates of the overlapping area, a first pixel mapping table and a second pixel mapping table corresponding to the first and second cameras are calculated.

[0012] Determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, includes:

[0013] The first image is sampled according to the first pixel mapping table to obtain the first overlapping region, and the second image is sampled according to the second pixel mapping table to obtain the second overlapping region.

[0014] In some embodiments, the method further includes, before calculating the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region:

[0015] Identify obstacle pixels in the first and second overlapping regions;

[0016] The obstacle pixels are corrected.

[0017] In some embodiments, identifying obstacle pixels in the first overlapping region and the second overlapping region includes:

[0018] Compare whether the color difference between pixels at the same position in the first overlapping region and the second overlapping region is greater than a set threshold.

[0019] If so, then the pixels at the same position are determined to be obstacle pixels;

[0020] If not, then the pixels at the same location are determined to be non-obstacle pixels.

[0021] In some embodiments, the correction process for the obstacle pixels includes:

[0022] Obtain the first color value of the obstacle pixel in the first overlapping area;

[0023] Obtain the second color value of the obstacle pixel in the second overlapping region;

[0024] The target color value of the obstacle pixel is determined based on the first color value and the second color value;

[0025] Set the color value of the obstacle pixel to the target pixel value.

[0026] In some embodiments, the first camera is a front-view fisheye camera of the vehicle, and the second camera is a right-view or left-view fisheye camera of the vehicle; or,

[0027] The first camera is a left-view fisheye camera for the vehicle, and the second camera is a front-view or rear-view fisheye camera for the vehicle; or,

[0028] The first camera is a right-view fisheye camera of the vehicle, and the second camera is a front-view or rear-view fisheye camera of the vehicle; or,

[0029] The first camera is a rear-view fisheye camera of the vehicle, and the second camera is a left-view or right-view fisheye camera of the vehicle.

[0030] In some embodiments, when a front-view fisheye camera, a right-view fisheye camera, a rear-view fisheye camera, and a left-view fisheye camera are included in pairs, the correction coefficients for the four cameras (front, rear, left, and right) can be calculated using the following formula:

[0031]

[0032] Among them, K 前 K represents the correction factor for the front fisheye camera. 右 K is the correction factor for the right-view fisheye camera. 后 K is the correction factor for the rear-view fisheye camera. 左 Correction coefficients for left-view fisheye cameras; RGB 左 A represents the average RGB value of region A in the image captured by the left-view fisheye camera. 前 A represents the average RGB value of region A in the image captured by the forward-looking fisheye camera. 前 B represents the average RGB value of region B in the image captured by the forward-looking fisheye camera. 右 B represents the average RGB value of region B in the image captured by the right-view fisheye camera. 右 C represents the average RGB value of region C in the image captured by the right-view fisheye camera. 后 C represents the average RGB value of region C in the image captured by the rear-view fisheye camera. 后 D represents the average RGB value of region D in the image captured by the rear-view fisheye camera. 左 D represents the average RGB value of the corresponding region D in the image captured by the left-view fisheye camera.

[0033] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the panoramic image stitching methods described above in this application.

[0034] Thirdly, embodiments of this application provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the panoramic image stitching methods described above.

[0035] Fourthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above panoramic image stitching methods.

[0036] This application embodiment calculates a first overlapping region and a second overlapping region between adjacent images captured by a first camera and a second camera, and further calculates a first average RGB value of the first overlapping region and a second average RGB value of the second overlapping region. Then, it calculates correction coefficients for the first camera and the second camera based on the first average RGB value and the second average RGB value, and finally performs stitching processing on the images captured by the first camera and the second camera based on the correction coefficients. This solves the problem of obvious color differences at the stitching point when stitching overlapping regions of images captured by adjacent cameras. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of an embodiment of the panoramic image stitching method of this application;

[0039] Figure 2 A flowchart of another embodiment of the panoramic image stitching method of this application;

[0040] Figure 3 A flowchart of another embodiment of the panoramic image stitching method of this application;

[0041] Figure 4 A flowchart of another embodiment of the panoramic image stitching method of this application;

[0042] Figure 5 A flowchart of another embodiment of the panoramic image stitching method of this application;

[0043] Figure 6 This is a schematic diagram of images captured by the four fisheye cameras used in this application;

[0044] Figure 7 A flowchart illustrating an embodiment of system initialization in this application;

[0045] Figure 8 This is a flowchart of another embodiment of the panoramic image stitching method in this application;

[0046] Figure 9 For the purposes of this application Figure 6 A schematic diagram of eight overlapping area images obtained from the top-down view V;

[0047] Figure 10 for Figure 9 A schematic diagram of the eight overlapping area images using the average RGB values;

[0048] Figure 11 This is a schematic diagram of the process for solving the correction coefficients using Jacobi SVD in this application;

[0049] Figure 12 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0051] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0052] This application proposes a panoramic image stitching method for vehicles. This method can be executed by the vehicle's infotainment system, which can project the stitched panoramic image onto the vehicle's display interface (which can be a physical display screen or a virtual display interface; this application does not limit this). Multiple cameras can be arranged around the vehicle body, each shooting from different directions and angles. There will be overlapping areas between images captured by adjacent cameras. The multiple cameras can be four fisheye cameras or other types of cameras; this application does not limit this. The four fisheye cameras can be respectively installed at the front of the vehicle, the rear of the vehicle, the left rearview mirror, and the right rearview mirror, etc. It should be noted that the above installation positions of the fisheye cameras are only examples; this application does not limit the specific installation positions of the fisheye cameras, as long as the shooting range of the four fisheye cameras can cover the entire circumference of the vehicle.

[0053] like Figure 1 The diagram shown is a flowchart illustrating an embodiment of the panoramic image stitching method of this application. In this embodiment, the panoramic image stitching method includes the following steps:

[0054] S10. Acquire a first image captured by a first camera and a second image captured by a second camera, and determine a first overlapping region in the first image that overlaps with the second image, and a second overlapping region in the second image that overlaps with the first image.

[0055] S20. Calculate the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region.

[0056] S30. Calculate the correction coefficients of the first camera and the second camera based on the first average RGB value and the second average RGB value, respectively.

[0057] S40. The images captured by the first camera and the second camera are stitched together based on the correction coefficients of the first camera and the second camera.

[0058] For step S10, for example, the first camera can be a front-view fisheye camera of the vehicle, and the second camera can be a left-view fisheye camera of the vehicle. Then, there is an overlap between the left front shooting area of ​​the front-view fisheye camera and the right front shooting area of ​​the left-view fisheye camera. The portion of the first image captured by the front-view fisheye camera that overlaps with the second image captured by the left-view fisheye camera is determined as the first overlapping region; the portion of the second image captured by the left-view fisheye camera that overlaps with the first image captured by the front-view fisheye camera is determined as the second overlapping region.

[0059] For example, the first camera and the second camera are two adjacent fisheye cameras mounted on the vehicle with overlapping shooting ranges. Specifically, the first camera is a front-view fisheye camera of the vehicle, and the second camera is a right-view or left-view fisheye camera of the vehicle; or, the first camera is a left-view fisheye camera of the vehicle, and the second camera is a front-view or rear-view fisheye camera of the vehicle; or, the first camera is a right-view fisheye camera of the vehicle, and the second camera is a front-view or rear-view fisheye camera of the vehicle; or, the first camera is a rear-view fisheye camera of the vehicle, and the second camera is a left-view or right-view fisheye camera of the vehicle.

[0060] For step S20, for example, the first overlapping region is downsampled to obtain the first average RGB value of the first overlapping region; the second overlapping region is downsampled to obtain the second average RGB value of the second overlapping region.

[0061] For step S30, for example, the first correction coefficient of the first camera and the second correction coefficient of the second camera are calculated based on the first average RGB value and the second average RGB value, respectively.

[0062] In this embodiment, only two adjacent first cameras and second cameras are used as examples. When there are multiple cameras (e.g., 4 cameras), the panoramic image stitching method of this application embodiment is generally applicable between other pairs of adjacent cameras.

[0063] This application embodiment calculates a first overlapping region and a second overlapping region between adjacent images captured by a first camera and a second camera, and further calculates a first average RGB value of the first overlapping region and a second average RGB value of the second overlapping region. Then, it calculates correction coefficients for the first camera and the second camera based on the first average RGB value and the second average RGB value, and finally performs stitching processing on the images captured by the first camera and the second camera based on the correction coefficients. This solves the problem of obvious color differences at the stitching point when stitching overlapping regions of images captured by adjacent cameras.

[0064] like Figure 2 The diagram shown is a flowchart illustrating another embodiment of the panoramic image stitching method of this application. In this embodiment, before determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, the method further includes:

[0065] S01. Determine the overlapping area (e.g., the first overlapping area and the second overlapping area) in the top view corresponding to the first camera and the second camera.

[0066] For example, the first camera can be a front-view fisheye camera of the vehicle, and the second camera can be a left-view fisheye camera of the vehicle. There is an overlap between the left front-view shooting area of ​​the front-view fisheye camera and the right front-view shooting area of ​​the left-view fisheye camera. The image captured by the front-view fisheye camera is projected as a first top view, and the image captured by the left-view fisheye camera is projected as a second top view. There is an overlap between the first top view and the second top view, that is, the overlapping area in the top views corresponding to the first camera and the second camera.

[0067] S02. Based on the intrinsic and extrinsic parameters of the first camera and the second camera, and the world coordinates of the overlapping area, calculate the first pixel mapping table and the second pixel mapping table corresponding to the first camera and the second camera.

[0068] Further, determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, includes: sampling the first image according to the first pixel mapping table to obtain the first overlapping region; and sampling the second image according to the second pixel mapping table to obtain the second overlapping region.

[0069] This embodiment can complete initialization during the initial stage of vehicle system power-on, determining the first pixel mapping table of the first camera and the second pixel mapping table of the second camera. This allows for a more accurate calculation of the first and second overlapping regions corresponding to the first and second cameras based on the calculated pixel mapping tables.

[0070] In developing this application, the inventors discovered that when obstacles exist in the overlapping area captured by adjacent cameras, the obstacles cause the pixels projected from the same point into two adjacent areas to not overlap. Conversely, if the difference between two pixels at the same position in the overlapping area is too large, it can be deduced that the obstacle is the cause.

[0071] To avoid the aforementioned problems caused by obstacles affecting the correction of overlapping areas captured by adjacent cameras, the inventors proposed the following... Figure 3 The improvement scheme is shown. For example... Figure 3 In the illustrated scheme, the panoramic image stitching method further includes, before calculating the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region:

[0072] S001. Identify obstacle pixels in the first overlapping region and the second overlapping region.

[0073] S002, Correct the pixels of the obstacle.

[0074] In this embodiment, before calculating the average RGB value of the overlapping area captured by adjacent cameras, obstacle pixels in the overlapping area are identified and corrected, thus avoiding the influence of obstacle pixels on the accuracy of the average RGB value calculation.

[0075] like Figure 4 The diagram shown is a flowchart illustrating another embodiment of the panoramic image stitching method of this application. In this embodiment, step S001 includes:

[0076] S0011. Compare whether the color difference between pixels at the same position in the first overlapping region and the second overlapping region is greater than a set threshold.

[0077] S0012. If the color difference is greater than a set threshold, then the pixels at the same position are determined to be obstacle pixels.

[0078] S0013. If the color difference is not greater than a set threshold, then the pixels at the same position are determined to be non-obstacle pixels.

[0079] For example, in the panoramic image stitching method of this embodiment, the color difference comparison and determination of steps S0011 to S0013 above are performed on each pixel in the overlapping area to identify obstacle pixels. For example, the pixel at the same position in the first overlapping area is p1, and the pixel at the same position in the second overlapping area is p2. The color difference is obtained by subtracting the color value of pixel p1 from the color value of pixel p2 and taking the absolute value.

[0080] Taking the color value as the RGB value of a pixel as an example, if the color value of pixel p1 is (r1, g1, b1) and the color value of pixel p2 is (r2, g2, b2), then the color difference between pixels p1 and p2 is: |p1-p2|+|g1-g2|+|b1-b2|.

[0081] If |p1-p2|+|g1-g2|+|b1-b2|> the set threshold, then pixels at the same position are determined to be obstacle pixels; otherwise, pixels at the same position are determined to be either obstacle pixels or non-obstacle pixels. The set threshold can be 0.9.

[0082] In this embodiment, obstacle pixels are identified by comparing the difference between pixels at the same position in the overlapping area with a set threshold. Furthermore, using the sum of the absolute values ​​of the differences between the RGB values ​​of pixels as the color difference between pixels further improves the accuracy of obstacle pixel identification.

[0083] like Figure 5The diagram shown is a flowchart illustrating another embodiment of the panoramic image stitching method of this application. In this embodiment, step S002 includes:

[0084] S0021. Obtain the first color value of the obstacle pixel in the first overlapping area. For example, the first color value is the color value (r1, g1, b1) of pixel p1.

[0085] S0022. Obtain the second color value of the obstacle pixel in the second overlapping area. For example, the second color value is the color value (r2, g2, b2) of pixel p2.

[0086] S0023. Determine the target color value of the obstacle pixel based on the first color value and the second color value.

[0087] For example, the weighted sum of the first color value and the second color value is used as the target color value of the obstacle pixel. Taking a weighting coefficient of 0.5 for the first color value and the second color value respectively as an example, the target color value is: 0.5*(first color value + second color value) = 0.5*(r1+r2, g1+g2, b1+b2).

[0088] S0024. Set the color value of the obstacle pixel to the target pixel value. For example, set the color value of the obstacle pixel to 0.5*(r1+r2, g1+g2, b1+b2).

[0089] In this embodiment, after identifying the obstacle pixels, the color value of the obstacle pixels is corrected by weighted summation of the color values ​​of the corresponding two pixels in the first and second overlapping regions, thereby avoiding the influence of the obstacle pixel color values ​​on the overlapping regions.

[0090] In some embodiments, the fisheye camera includes four consecutively adjacent front-view fisheye cameras, a rear-view right-view fisheye camera, a rear-view fisheye camera, and a left-view fisheye camera. The correction coefficients for the four cameras (front, rear, left, and right) can be calculated using Jacobi SVD with the following formula:

[0091]

[0092] Among them, K 前 K represents the correction factor for the front fisheye camera. 右 K is the correction factor for the right-view fisheye camera. 后 K is the correction factor for the rear-view fisheye camera. 左 For the left-view fisheye camera;

[0093] like Figure 6As shown, A represents the overlapping area captured by the front-view fisheye camera and the left-view fisheye camera, B represents the overlapping area captured by the front-view fisheye camera and the right-view fisheye camera, C represents the overlapping area captured by the right-view fisheye camera and the rear-view fisheye camera, and D represents the overlapping area captured by the rear-view fisheye camera and the left-view fisheye camera.

[0094] RGB 左 A represents the average RGB value of region A in the image captured by the left-view fisheye camera. 前 A represents the average RGB value of region A in the image captured by the forward-looking fisheye camera. 前 B represents the average RGB value of region B in the image captured by the forward-looking fisheye camera. 右 B represents the average RGB value of region B in the image captured by the right-view fisheye camera. 右 C represents the average RGB value of region C in the image captured by the right-view fisheye camera. 后 C represents the average RGB value of region C in the image captured by the rear-view fisheye camera. 后 D represents the average RGB value of region D in the image captured by the rear-view fisheye camera. 左 D represents the average RGB value of the corresponding region D in the image captured by the left-view fisheye camera.

[0095] The following is an example of a panoramic image stitching method when four fisheye cameras (front-view fisheye camera, right-view fisheye camera, rear-view fisheye camera and left-view fisheye camera) are installed on a vehicle.

[0096] like Figure 6 The diagram shows images captured by the four fisheye cameras in this application. The left side includes a front view F captured by the front-view fisheye camera, a right view R captured by the right-view fisheye camera, a rear view H captured by the rear-view fisheye camera, and a left view L captured by the left-view fisheye camera. Figure 6 As shown, the right side is a top view V obtained based on the front view F, right view R, rear view H, and left view L. The top view V includes four overlapping regions A, B, C, and D.

[0097] The panoramic image stitching method in this embodiment aims to improve the smoothness of in-vehicle panoramic image stitching and eliminate problems such as obvious stitching boundaries and the influence of obstacles. This embodiment mainly includes the following two stages: the system initialization stage and the real-time operation stage. The system can be the vehicle's in-vehicle infotainment system or a processor configured on the vehicle for executing the panoramic image stitching method. The two stages are described in detail below.

[0098] I. System Initialization Phase:

[0099] like Figure 7The diagram shown is a flowchart of an embodiment of system initialization in this application. This embodiment includes the following steps:

[0100] 1. Read camera intrinsic and extrinsic parameters. For example, read the intrinsic and extrinsic parameters of the front-view fisheye camera, right-view fisheye camera, rear-view fisheye camera, and left-view fisheye camera.

[0101] 2. Select the overlapping area. For example, select the valid overlapping area in the top view (such as...). Figure 6 (A, B, C, and D regions in the diagram) and their size (e.g., 64*64).

[0102] 3. Calculate the sampling map tables. For example, based on the intrinsic and extrinsic parameters of the four fisheye cameras and the world coordinates of the overlapping area, eight map tables (i.e., pixel mapping tables) for the overlapping area are obtained through inverse projection. These eight map tables include the first and second pixel mapping tables corresponding to region A, the third and fourth pixel mapping tables corresponding to region B, the fifth and sixth pixel mapping tables corresponding to region C, and the seventh and eighth pixel mapping tables corresponding to region D. The width and height of the map tables are the same as the width and height of the final output image. Map table calculation depends on the camera's intrinsic and extrinsic parameters, and the system initializes it once every time it powers on.

[0103] II. Real-time Operation Phase:

[0104] like Figure 8 The diagram shows a flowchart of another embodiment of the panoramic image stitching method in this application. It is applied to an in-vehicle infotainment system or a processor for executing the panoramic image stitching method. The in-vehicle infotainment system or processor is equipped with a GPU and a CPU. In this embodiment, the panoramic image stitching method includes the following steps:

[0105] 1. Input Images. For example, four images captured in real time by four fisheye cameras are imported into the GPU.

[0106] 2. Generate overlapping region images. For example, based on 8 map tables, sample 4 images from four fisheye cameras to generate 8 overlapping region images, named [A front, A left], [B front, B right], [C back, C right], [D back, D left]. Figure 9 The image shown is based on this application. Figure 6 A schematic diagram showing eight overlapping area images obtained from the top-down view V. Among them,

[0107] A represents the overlapping portion of an image taken by a forward-looking fisheye camera within region A.

[0108] A represents the overlapping portion of the image taken by the left-view fisheye camera within region A;

[0109] The area before B is the overlapping portion of the image taken by the forward-looking fisheye camera in region B;

[0110] B represents the overlapping portion of the image taken by the right-view fisheye camera within region B;

[0111] C represents the overlapping portion of the image taken by the rear-view fisheye camera within region C;

[0112] C represents the overlapping portion of the image taken by the right-view fisheye camera within region C;

[0113] The image after D is the overlapping part of the image taken by the rear-view fisheye camera in region D;

[0114] D (left) represents the overlapping portion of the image taken by the left-view fisheye camera within region D.

[0115] When generating the final output of 8 overlapping region images, the values ​​of the same UV positions are first read from the map table based on the pixel position UV of each image. This value records the pixel coordinates XY of the fisheye camera image, and the color value corresponding to the pixel coordinates XY is obtained, thereby obtaining the color value of the pixels on the overlapping region image.

[0116] 3. Obstacle Pixel Correction. The same physical point captured by adjacent cameras (e.g., front and left) in overlapping areas should ideally maintain geometric and color consistency. Ideally, after projection transformation, the coordinates of these points in the images from both cameras should be aligned, with RGB values ​​differing only slightly due to differences in camera parameters. However, this consistency is significantly disrupted when obstacles are present around the vehicle.

[0117] When an obstacle partially obstructs a camera's field of view, two typical scenarios occur. The first is unilateral obstruction. For example, a pillar on the left front of the vehicle might block the front camera's view, while the left camera can still observe the background. In this case, the overlapping area of ​​the front camera displays the pixel values ​​of the pillar's surface, while the corresponding location on the left camera displays the pixel values ​​of the background, resulting in a significant difference in their RGB values. The second scenario is a difference in viewing angle. When an obstacle appears in the field of view of both cameras simultaneously, due to different shooting angles, the same obstacle may present different side characteristics in different cameras. For example, a traffic cone directly in front might appear as a red front view in the front camera but as a yellow side view in the left camera, causing incorrect matching of physical points that should be aligned.

[0118] Traditional color equalization algorithms often fail to compensate for pixel differences caused by obstacles. As mentioned earlier, obstacles can cause pixels projected from the same point into two adjacent regions to not overlap. Therefore, it can be inferred that excessive differences indicate the presence of an obstacle. To address this issue, this method proposes a correction strategy based on image spatial projection: First, four fisheye camera images are projected to generate eight overlapping region images. Then, two pixels at corresponding positions within the same overlapping region are compared pixel by pixel. When the sum of the absolute values ​​of the RGB differences exceeds a preset threshold (e.g., 0.9), it is determined that an obstacle is present at that location, and the two pixel values ​​are corrected to their average. This method effectively isolates the interference of obstacles in color equalization calculations through preprocessing. Furthermore, these pixels with excessive differences can be used for obstacle detection, providing more reliable color consistency and environmental perception capabilities for panoramic imaging systems.

[0119] For example, the color value is the three components (R, G, B) under the RGB color gamut, such as (1.0, 0.0, 0.0) red. Each pixel in the overlapping area image has an RGB value. An example of modifying the color value of pixels in two overlapping area images at the same location to be the same is as follows: Taking A front and A left as examples, if the color difference of the same point in A front and A left is found to be too large, then the color value of both is rewritten as 0.5 * (A left + A front).

[0120] Traditional color equalization algorithms often fail to compensate properly or fail when there are obstacles around the vehicle. However, the method proposed in this application, which corrects obstacle pixels based on image spatial projection (e.g., projecting four fisheye camera images into eight overlapping area images), can better handle the sector color difference problem in panoramic images.

[0121] 4. Downsampling to calculate the average RGB value. For example, the 8 corrected images obtained in step 3 are downsampled to calculate the average RGB value, denoted as [RGB...]. A前 RGB A左 ],[RGB B前 RGB B右 ],[RGB C后 RGB C右 ],[RGB D后 RGB D左 ].like Figure 10 As shown Figure 9 The diagram shows the eight overlapping area images with average RGB values.

[0122] 5. Solve for the compensation coefficients. For example, the average RGB values ​​and other data calculated in step 4 are read back to the CPU, and the following equation 1 is solved using the least squares method to obtain the compensation coefficients for the four cameras: K前 {KR1,KG1,KB1}, K 后 {KR2,KG2,KB2}, K 左 {KR3,KG3,KB3}, K 右 {KR4,KG4,KB4}. For example, Jacobi SVD code is used to solve this problem, taking eight average RGB values ​​as input to calculate the compensation coefficients K for four camera groups. 前 K 后 K 左 and K 右 .

[0123]

[0124] Among them, K 前 K is the correction factor for the forward-looking fisheye camera. 右 K is the correction factor for the right-view fisheye camera. 后 K is the correction factor for the rear-view fisheye camera. 左 For the left-view fisheye camera;

[0125] A represents the overlapping area captured by the front-view fisheye camera and the left-view fisheye camera; B represents the overlapping area captured by the front-view fisheye camera and the right-view fisheye camera; C represents the overlapping area captured by the right-view fisheye camera and the rear-view fisheye camera; and D represents the overlapping area captured by the rear-view fisheye camera and the left-view fisheye camera.

[0126] RGB 左 A represents the average RGB value of region A in the image captured by the left-view fisheye camera. 前 A represents the average RGB value of region A in the image captured by the forward-looking fisheye camera. 前 B represents the average RGB value of region B in the image captured by the forward-looking fisheye camera. 右 B represents the average RGB value of region B in the image captured by the right-view fisheye camera. 右 C represents the average RGB value of region C in the image captured by the right-view fisheye camera. 后 C represents the average RGB value of region C in the image captured by the rear-view fisheye camera. 后 D represents the average RGB value of region D in the image captured by the rear-view fisheye camera. 左 D represents the average RGB value of the corresponding region D in the image captured by the left-view fisheye camera.

[0127] like Figure 11The diagram shows the flowchart of the Jacobi SVD process for solving the correction coefficients in this application. In this embodiment, the Jacobi SVD code is used to solve the least squares method to determine the correction coefficients for the four fisheye cameras. Specifically, the steps include:

[0128] Input the average RGB values ​​of the overlapping region (i.e., the eight average RGB values ​​in the above embodiment).

[0129] By calibrating the R channel, we construct X_r = A_r / B_r, then solve X_r using SVD, and finally normalize X_r to obtain the compensation coefficients of the R channel; where A_r is a matrix composed of the color values ​​of 8 overlapping regions in the R channel of the images captured by 4 fisheye cameras, and B_r is a vector of all zeros.

[0130] By calibrating the G channel, we construct X_g = A_g / B_g, then solve X_g using SVD, and finally normalize X_g to obtain the compensation coefficients of the G channel; where A_g is a matrix composed of the color values ​​of the 8 overlapping regions in the G channel of the images captured by the 4 fisheye cameras, and B_g is an all-zero vector.

[0131] By calibrating the B channel, we construct X_bA_b / B_b, then solve X_b using SVD, and finally normalize X_b to obtain the compensation coefficients for the B channel. Here, A_b is a matrix composed of the color values ​​of the eight overlapping regions in the B channel of the images captured by the four fisheye cameras, and B_b is an all-zero vector.

[0132] Finally, the three compensation coefficients for the R, G, and B channels are output.

[0133] 6. Real-time panoramic image stitching. For example, the correction coefficients of the current frame are passed to the GPU for real-time color correction of the camera, and then panoramic image stitching is performed to obtain an optimized stitched image.

[0134] For example, the GPU obtains the compensation coefficients (K) of the above four cameras. 前 K 后 K 左 After K (right), color correction processing is performed on the images captured by the four fisheye cameras according to the compensation coefficients of the four cameras. For example, using compensation coefficient K... 前 Compensated image 1 is obtained from the image captured by the forward-looking fisheye camera; compensation coefficient K is used. 后 Compensated image 2 is obtained from the image captured by the rear-view fisheye camera; compensation coefficient K is used. 左 Image 3 is obtained from the image captured by the left-view fisheye camera; compensation coefficient K is used. 右 Compensated image 4 is obtained from the image captured by the right-view fisheye camera. Then, panoramic image stitching is performed based on compensated images 1 to 4 to obtain the optimized panoramic image.

[0135] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0136] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the panoramic image stitching methods described above.

[0137] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above panoramic image stitching methods.

[0138] In some embodiments, this application also provides an electronic device, which includes: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a panoramic image stitching method.

[0139] Figure 12 This is a schematic diagram of the hardware structure of an electronic device for performing a panoramic image stitching method according to another embodiment of this application, as shown below. Figure 12 As shown, the device includes:

[0140] One or more processors 1210 and memory 1220, Figure 12 Take the 1210 processor as an example.

[0141] The device for performing the panoramic image stitching method may also include an input device 1230 and an output device 1240.

[0142] The processor 1210, memory 1220, input device 1230, and output device 1240 can be connected via a bus or other means. Figure 12Taking the example of a connection between China and Israel via a bus.

[0143] The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the panoramic image stitching method in the embodiments of this application. The processor 1210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1220, thereby implementing the panoramic image stitching method of the above-described method embodiments.

[0144] The memory 1220 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the panoramic image stitching device. Furthermore, the memory 1220 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include memory remotely located relative to the processor 1210, and these remote memories can be connected to the panoramic image stitching device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] The input device 1230 can receive input digital or character information, and generate signals related to user settings and function control of the panoramic image stitching device. The output device 1240 may include a display device such as a display screen.

[0146] The one or more modules are stored in the memory 1220, and when executed by the one or more processors 1210, the panoramic image stitching method in any of the above method embodiments is executed.

[0147] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A panoramic image stitching method, comprising: Acquire a first image captured by a first camera and a second image captured by a second camera, and determine a first overlapping region in the first image that overlaps with the second image, and a second overlapping region in the second image that overlaps with the first image; Calculate the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region; The correction coefficients for the first camera and the second camera are calculated based on the first average RGB value and the second average RGB value, respectively. The images captured by the first camera and the second camera are stitched together based on the correction coefficients of the first camera and the second camera.

2. The method according to claim 1, characterized in that, Before determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, the method further includes: Determine the overlapping area in the top view corresponding to the first camera and the second camera; Based on the intrinsic and extrinsic parameters of the first and second cameras and the world coordinates of the overlapping area, a first pixel mapping table and a second pixel mapping table corresponding to the first and second cameras are calculated. Determining the first overlapping region in the first image that overlaps with the second image, and the second overlapping region in the second image that overlaps with the first image, includes: The first image is sampled according to the first pixel mapping table to obtain the first overlapping region, and the second image is sampled according to the second pixel mapping table to obtain the second overlapping region.

3. The method according to claim 1, characterized in that, Before calculating the first average RGB value of the first overlapping region and the second average RGB value of the second overlapping region, the following is also included: Identify obstacle pixels in the first and second overlapping regions; The obstacle pixels are corrected.

4. The method according to claim 3, characterized in that, The identification of obstacle pixels in the first overlapping region and the second overlapping region includes: Compare whether the color difference between pixels at the same position in the first overlapping region and the second overlapping region is greater than a set threshold. If so, then the pixels at the same position are determined to be obstacle pixels; If not, then the pixels at the same location are determined to be non-obstacle pixels.

5. The method according to claim 3, characterized in that, The correction process for the obstacle pixels includes: Obtain the first color value of the obstacle pixel in the first overlapping area; Obtain the second color value of the obstacle pixel in the second overlapping region; The target color value of the obstacle pixel is determined based on the first color value and the second color value; Set the color value of the obstacle pixel to the target pixel value.

6. The method according to claims 1-5, characterized in that, The first camera is a forward-facing fisheye camera for the vehicle, and the second camera is either a right-facing or left-facing fisheye camera for the vehicle; or, The first camera is a left-view fisheye camera for the vehicle, and the second camera is a front-view or rear-view fisheye camera for the vehicle; or, The first camera is a right-view fisheye camera of the vehicle, and the second camera is a front-view or rear-view fisheye camera of the vehicle; or, The first camera is a rear-view fisheye camera of the vehicle, and the second camera is a left-view or right-view fisheye camera of the vehicle.

7. The method according to claim 6, characterized in that, When the camera includes four consecutive fisheye cameras—a front-view fisheye camera, a right-view fisheye camera, a rear-view fisheye camera, and a left-view fisheye camera—the correction coefficients for the front, rear, left, and right cameras can be calculated using the following formula: Among them, K 前 K represents the correction factor for the front fisheye camera. 右 K is the correction factor for the right-view fisheye camera. 后 K is the correction factor for the rear-view fisheye camera. 左 Correction coefficients for left-view fisheye cameras; RGB 左,A This represents the average RGB value of region A in the image captured by the left-view fisheye camera. 前, A This represents the average RGB value of region A in the image captured by the forward-looking fisheye camera. 前,B This represents the average RGB value of region B in the image captured by the forward-looking fisheye camera. 右,B This represents the average RGB value of region B in the image captured by the right-view fisheye camera. 右,C This represents the average RGB value of region C in the image captured by the right-view fisheye camera. 后,C This represents the average RGB value of region C in the image captured by the rear-view fisheye camera. 后,D This represents the average RGB value of region D in the image captured by the rear-view fisheye camera. 左,D This represents the average RGB value of the corresponding D region in the image captured by the left-view fisheye camera.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

Images