Panoramic image stitching method

Abstract

The application discloses a panoramic image splicing method, comprising the following steps: matching the pixels of the splicing band area of two images to be spliced; performing optical flow calculation based on the positions of the matched pixels; determining the first camera parameter correction value and the second camera parameter correction value of a first camera based on the optical flow calculation result; performing re-projection processing on the first image to be spliced based on the first camera parameter correction value, and performing re-projection processing on the second image to be spliced based on the second camera parameter correction value; and performing splicing operation on the two re-projected images to be spliced. The application can realize natural and smooth transition of panoramic image splicing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image stitching method. Background Art

[0002] A panoramic image is an image that covers 360 degrees horizontally and 180 degrees vertically. Panoramic images are typically synthesized from several images with overlapping content (hereinafter referred to as "stitching strips"). The algorithm for synthesizing panoramic images is called a panoramic stitching algorithm. Because the input images have different shooting postures and camera parameters, the same object in three-dimensional space will appear at different angles and positions in different two-dimensional images. Therefore, when generating a 360-degree panoramic image, it is necessary to selectively use the overlapping areas of adjacent images to produce a unified synthesis result for that area.

[0003] How to process the stitching band so that the final panoramic image can have a smooth and natural transition effect at the junction of the content of different lenses is the core problem that the panoramic image stitching algorithm needs to solve. Summary of the Invention

[0004] The main purpose of the present invention is to provide a panoramic image stitching method, which aims to solve the problem that the panoramic image finally generated by the conventional stitching process cannot have a smooth and natural transition at the intersection of the contents of different lenses.

[0005] To achieve the above object, the present invention provides a panoramic image stitching method, which includes:

[0006] Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched;

[0007] Calculating, based on the position of the first pixel and the position of the corresponding second pixel, a first horizontal optical flow and a first vertical optical flow for each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow for each pixel in the stitching band area of ​​the second image to be stitched;

[0008] Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow;

[0009] performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value;

[0010] The two images to be spliced after re-projection are spliced.

[0011] The panoramic image splicing method provided by the application has the following advantages: firstly, the content overlap area of adjacent lens images is calculated by using optical flow, then the focal length and rotation parameters of the two adjacent cameras are adjusted by using the result of the optical flow calculation, and the images of the cameras are regenerated by using the adjusted parameters. In order to avoid the abrupt change of the adjusted image and affect the viewing effect, a distance weight coefficient is introduced when the camera images are regenerated, so that the adjustment range of the image part close to the overlap area is larger, and vice versa. After the above adjustment, the horizontal and vertical direction parallax of the content overlap area of the two adjacent camera images can be greatly reduced, and the reduction of the parallax has the following two advantages: 1. Higher accuracy can be obtained in the subsequent optical flow calculation. 2. When the content of the overlap area is generated in the panoramic image, a more natural visual transition effect is obtained. The last step is to perform a second optical flow calculation on the splicing band of the adjusted image, and generate the corresponding area in the panoramic image by combining the splicing band image and the result of the second optical flow calculation. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a system structure schematic diagram of a hardware running environment related to an embodiment scheme of the application;

[0013] Figure 2 is a flowchart of an embodiment of the panoramic image splicing method of the application;

[0014] Figure 3 is an example diagram of a splicing band in an embodiment of the panoramic image splicing method of the application;

[0015] Figure 4 is an explanation diagram of the camera field of view angle and the splicing band in an embodiment of the panoramic image splicing method of the application. DETAILED DESCRIPTION

[0016] It should be understood that the specific embodiments described herein are merely intended to explain the application, and are not intended to limit the application.

[0017] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the application.

[0018] In the prior art, the splicing band is processed, so that it is difficult to generate a smooth and natural transition effect at the content junction of different lenses in the finally generated panoramic image.

[0019] In order to solve the above technical problems, the present application provides a panoramic image splicing method, in which firstly, the content overlapping area of adjacent lens images is calculated by optical flow, then the focal length and rotation parameters of the two adjacent cameras are adjusted respectively by using the result of optical flow calculation, and the images of the cameras are regenerated respectively by using the adjusted parameters. In the process of regenerating the camera images, in order to avoid the relatively obvious changes of the adjusted images affecting the viewing effect, a distance weight coefficient is introduced, so that the adjustment range of the picture part closer to the overlapping area is larger, and vice versa. After the above adjustment, the horizontal and vertical direction parallax of the content overlapping area of the two adjacent camera images can be reduced to a large extent, and the reduction of the parallax has the following two advantages: 1. In the subsequent optical flow calculation, a higher precision result can be obtained. 2. When generating the content of the overlapping area in the panoramic image, a more natural visual transition effect can be obtained. The last step is to perform a second optical flow calculation on the splicing band of the adjusted images, and generate the corresponding area in the panoramic image by combining the splicing band image and the result of the second optical flow calculation.

[0020] As shown in Figure 1 , the system structure diagram of the hardware running environment involved in the embodiment of the present application is shown. Figure 1

[0021] The terminal of the embodiment of the present application can be a terminal device with computing capability, can also be a PC, and can also be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer and other movable terminal devices with display function.

[0022] As shown in Figure 1 , the terminal can include a processor 1001 such as a CPU, a network interface 1004, a user interface 1003, a memory 1005 and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen (Display) and an input unit such as a keyboard (Keyboard). The optional user interface 1003 can also include a standard wired interface and a wireless interface. The network interface 1004 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory) such as a disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.​

[0023] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0024] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0025] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a panoramic image stitching program.

[0026] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0027] Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched;

[0028] Calculating, based on the position of the first pixel and the position of the corresponding second pixel, a first horizontal optical flow and a first vertical optical flow for each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow for each pixel in the stitching band area of ​​the second image to be stitched;

[0029] Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow;

[0030] performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value;

[0031] Perform stitching operation on the two reprojected images to be stitched.

[0032] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0033] Matching a third pixel of the stitching band region of the reprojected first image to be stitched with a fourth pixel of the stitching band region of the reprojected second image to be stitched;

[0034] Calculating a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the reprojected first image to be stitched based on the position of the third pixel and the corresponding matching position of the fourth pixel;

[0035] Calculating a composite coordinate value of each of the third pixels based on the position of the third pixel and the corresponding third horizontal optical flow and third vertical optical flow;

[0036] Calculating a synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel;

[0037] Image stitching is performed based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

[0038] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0039] Determine the first horizontal optical flow based on a difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel;

[0040] Determine the first vertical optical flow based on a difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel;

[0041] Determine the second horizontal optical flow based on a difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel;

[0042] The second vertical optical flow is determined based on a difference value obtained by subtracting the vertical coordinate of the second pixel from the vertical coordinate of the first pixel.

[0043] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0044] Divide the splicing strip into 360 areas in the vertical direction;

[0045] Calculate the first horizontal average optical flow, the first vertical average optical flow, the second horizontal average optical flow, and the second vertical average optical flow corresponding to each spatial region based on the first horizontal optical flow, the first vertical optical flow, the second horizontal optical flow, and the second vertical optical flow;

[0046] The first camera parameter correction value is determined based on the first horizontal average optical flow and the first vertical average optical flow, and the second camera parameter correction value is determined based on the second horizontal average optical flow and the second vertical average optical flow.

[0047] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0048] Calculate the spatial angle between each pixel and the corresponding spatial point;

[0049] Determining a distance weight parameter based on the spatial angle;

[0050] The first camera parameter correction value is determined based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and the second camera parameter correction value is determined based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow.

[0051] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0052] A first focal length correction value and a first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is:

[0053]

[0054] f′0=f 01 *c0

[0055] C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; f 01 is the original value of the first focal length, f′ 01 is the first focal length correction value; w 01 is the first distance weight correction parameter;

[0056] The preset formula for the first rotation matrix correction value is:

[0057]

[0058] Where Φ1 is the first space Euler angle; h is the preset second preset coefficient; is the average optical flow in the first vertical direction.

[0059] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0060] The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is:

[0061]

[0062] f′ 02 =f 02 *c 02

[0063] C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; f 02 is the original value of the second focal length, f′ 02 is the second focal length correction value;

[0064] The preset formula for the second rotation matrix correction value is:

[0065]

[0066] Where Φ2 is the second space Euler angle; h is the preset second preset coefficient; is the average optical flow in the second vertical direction.

[0067] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0068] Based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v, the composite coordinates of the third pixel are calculated.

[0069] Reference Figure 2 , Figure 2 This is a flow chart of an embodiment of a panoramic image stitching method of the present invention. In some embodiments, the panoramic image stitching method includes:

[0070] Step S10 : matching a first pixel of a splicing band region of a first image to be spliced ​​with a second pixel of a splicing band region of a second image to be spliced.

[0071] The first image to be stitched and the second image to be stitched are two images captured by adjacent lenses and used to stitch a panoramic image. Figure 3 , Figure 3 is an example diagram of a splicing band in an embodiment of the present invention. The splicing band area refers to the overlapping area of ​​the first image to be spliced ​​and the second image to be spliced. Figure 3 Regions A and B are shown. Each first pixel in the stitching band of the first image to be stitched can find a corresponding matching second pixel in the stitching band of the second image to be stitched. When executing this method, the matching second pixel corresponding to each first pixel is first determined. In some embodiments, matching points of the stitching regions of the two images to be stitched can be determined based on pixel values, that is, the pixel value differences between all first pixels and the matching second pixels are calculated, and all matching first pixels and second pixels are determined based on the minimum value of the sum of the pixel value differences corresponding to all first pixels.

[0072] Step S20: Based on the position of the first pixel and the position of the corresponding matching second pixel, calculate the first horizontal optical flow and the first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and the second horizontal optical flow and the second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched.

[0073] Optical flow characterizes image changes and includes information about the relative position changes of targets in different images. Optical flow includes vertical optical flow and horizontal optical flow, which respectively characterize the position changes of the image in the vertical and horizontal directions. Since optical flow has directional characteristics, it is necessary to calculate the corresponding optical flow for each image separately. Among them, the first horizontal optical flow and the first vertical optical flow refer to the optical flow determined for the first image to be stitched; the second horizontal optical flow and the second horizontal optical flow refer to the optical flow determined for the second image to be stitched. The position of a pixel can be determined based on the pixel coordinates.

[0074] In some embodiments, step S20 may include:

[0075] Step S21, determining the first horizontal optical flow based on a difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel;

[0076] Step S22, determining the first vertical optical flow based on a difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel;

[0077] Step S23, determining the second horizontal optical flow based on a difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel;

[0078] Step S24 : determining the second vertical optical flow based on a difference between the vertical coordinate of the first pixel and the vertical coordinate of the second pixel.

[0079] Since the common content in adjacent images only exists in the stitching band, it is only necessary to calculate the optical flow for the stitching band area. Assume that the stitching band areas of the first image to be stitched and the second image to be stitched are A and B, respectively. For a first pixel X(x,y) in A, search for the second pixel X'(x',y') whose pixel value is closest to X in the stitching band B. Then the first horizontal optical flow of the first pixel X is u=x'-x, and the first vertical optical flow is v=y'-y. The second horizontal optical flow of the second pixel X' is u=xx', and the second vertical optical flow is v=y'-y.

[0080] Step S30 : determining a first camera parameter correction value of the first camera based on the first horizontal optical flow and the first vertical optical flow, and determining a second camera parameter correction value of the second camera based on the second horizontal optical flow and the second vertical optical flow.

[0081] The first image to be stitched is acquired by a first camera, and the second image to be stitched is acquired by a second camera.

[0082] In some embodiments, the camera parameter correction values ​​may include a focal length correction value and a rotation matrix correction value.

[0083] Calculating the camera parameter correction values ​​and the subsequent rotation matrix correction values ​​requires a series of adjustments to the adjacent images to reduce the horizontal and vertical parallax. The purpose of reducing the horizontal parallax is mainly to improve the consistency of the inside and outside of the stitching strip in the final panorama generation step, and the purpose of reducing the vertical parallax is mainly to reduce the oblique deformation that may occur in the panorama stitching strip (assuming that the two adjacent images are Figure 1 (If one image is positioned too low, and one image is positioned too high, the stitching strip between them will inevitably be deformed diagonally to achieve a smooth transition.) Reducing horizontal and vertical parallax also helps reduce errors in the subsequent second optical flow calculation (a larger parallax increases the search range for corresponding pixels, making it more likely to produce erroneous results). Horizontal parallax can be reduced by adjusting the camera's focal length, while vertical parallax can be reduced by adjusting the camera's shooting angle.

[0084] The rotation matrix is ​​used to compensate for vertical parallax by rotating each lens around the principal optical axis (Z axis) by a certain angle (i.e., adjusting the spatial Euler angles). To achieve the best results, different parts of the image require different rotation angles, with areas with greater vertical optical flow requiring larger angles.

[0085] In some embodiments, step S30 includes:

[0086] Step S31: divide the splicing tape into 360 equal areas in the vertical direction.

[0087] Consider the camera's visual space as a spherical space. Referencing the direction and method of division of the Earth's latitude, with the center of the spherical visual space as the origin, a circle of angles in the vertical direction is 360°. Each angle can determine a dividing line of the spherical visual space (similar to the Earth's latitude lines). All dividing lines divide the spherical visual space into 360 regions, that is, the stitching band is evenly divided (even division means that the corresponding division angle is 1°) into 360 regions. In this step, the stitching band area of ​​the first image to be stitched is divided into 360 regions, and the stitching band area of ​​the second image to be stitched is divided into 360 regions.

[0088] Step S32: Calculate the first horizontal average optical flow, the first vertical average optical flow, the second horizontal average optical flow, and the second vertical average optical flow corresponding to each spatial area based on the first horizontal optical flow, the first vertical optical flow, the second horizontal optical flow, and the second vertical optical flow.

[0089] For each divided splicing band area, the average value can be calculated based on the first horizontal optical flow and the first vertical optical flow corresponding to each first pixel in the divided area of ​​the first image to be stitched, and the first horizontal average optical flow and the first vertical average optical flow of the corresponding area can be determined, ultimately obtaining 360 first horizontal average optical flows and first vertical average optical flows corresponding to each of the 360 ​​areas of the splicing band of the first image to be stitched. Similarly, based on the second horizontal optical flow and the second vertical optical flow of the pixels in the 360 ​​areas of the splicing band of the second image to be stitched, the 360 ​​second horizontal average optical flows and the second vertical average optical flows corresponding to each of the 360 ​​areas of the splicing band of the second image to be stitched can be obtained.

[0090] In some embodiments, since fisheye lens images do not follow the perspective projection imaging principle of ordinary cameras, they have large deformations and are not convenient for direct calculation of optical flow. It is necessary to first perform re-projection operation on the overlapping area of the fisheye image and transform to the perspective projection space. The overlapping area after performing the projection transformation is not necessarily a regular rectangular area and there may be areas with invalid pixels (which can be regarded as pixel values of 0) in the area. These areas will not have effective optical flow when calculating the optical flow, so they are considered to have optical flow values of zero. If these areas with optical flow values of zero due to invalid pixels are allowed to participate in the average optical flow statistical step, the average optical flow value calculated will be too small, so these areas need to be avoided when calculating the average optical flow.

[0091] Step S33, determining the first camera parameter correction value based on the first horizontal direction average optical flow and the first vertical direction average optical flow, and determining the second camera parameter correction value based on the second horizontal direction average optical flow and the second vertical direction average optical flow.

[0092] The basic implementation of the panoramic image stitching algorithm usually does not need to process the part outside the stitching band, but in order to avoid the sudden change inside and outside the stitching band, it is necessary to adjust the certain range outside the stitching band similar to the stitching band, and the closer to the stitching band, the greater the degree of adjustment, and vice versa, so a distance weight parameter needs to be calculated. Referring to Figure 4 , Figure 4 is a camera field of view angle and stitching band explanation diagram in embodiments of the present application. Taking a camera with a field of view angle of 200° as an example, the effective field of view angle of the axis on one side of the picture and the picture axis is about 100 degrees. Since the circumference of a sphere is 360 degrees, the single side field of view angle of each lens is about 90-100 degrees, which is the overlapping area with the other lens, that is, the stitching band. The single side view angle of the central axis of each lens is 100 degrees, and the stitching area of the two lenses is 20 degrees, and the closer to the stitching area, the greater the weight. In some embodiments, step S33 includes the following steps before step S33:

[0093] Step S331, calculating the spatial angle of each pixel corresponding to the spatial point.

[0094] For a point P in the panoramic image close to the spatial coordinates (x, y, z) corresponding to the 0th lens, the spatial angle is calculated as follows:

[0095]

[0096] The distance weight parameter is determined based on the spatial angle.

[0097] The distance weight parameter of the P point for the 0th lens When θ0>45°, the closer θ0 is to 100°, the greater the weight (the maximum weight is 1), and vice versa. When θ0≤45°, the weight is 0 and no adjustment is required. 45° is an empirical parameter obtained after multiple experiments. Based on the above method, the w corresponding to each first pixel of the first image to be stitched can be calculated separately. 01 And w corresponding to each second pixel of the second image to be stitched 02 .in,

[0098]

[0099] where θ 01 ,θ 02 are the spatial angles of the spatial points of the first pixel and the second pixel respectively, w 01 、w 02 are the distance weight parameters corresponding to the first camera and the second camera respectively.

[0100] Step S33 includes:

[0101] Step S331: Determine the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and determine the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow.

[0102] In some embodiments, the step of determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow includes:

[0103] A first focal length correction value and a first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is:

[0104]

[0105] f′0=f 01 *c 01

[0106] C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; f 01 is the original value of the first focal length, f′ 01 is the first focal length correction value;

[0107] The preset formula of the first rotation matrix correction value R1 is:

[0108]

[0109] Wherein, Φ1 is the first space Euler angle; h is the preset second preset coefficient; is the average optical flow in the first vertical direction. In some embodiments, h can be set to 0.27, which is an empirical coefficient obtained based on visual effects after multiple experiments.

[0110] Specifically, for a point P(x,y,z) in the panorama, the corresponding three-dimensional space latitude φ∈[-π,π], according to the latitude value, the first horizontal average optical flow corresponding to point P can be found in the average optical flow result of step S32 And the first vertical average optical flow Where k is a coefficient calculated based on the physical relationship between the average focal length of the lens and parallax. For example, k can be set to 0.001068.

[0111] Similarly, in some embodiments, the step of determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow includes:

[0112] The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is:

[0113]

[0114] f′ 02 =f 02 *c 02

[0115] C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; f 02 is the original value of the second focal length, f′ 02 is the second focal length correction value;

[0116] The preset formula of the second rotation matrix correction value R2 is:

[0117]

[0118] Where Φ2 is the second space Euler angle; h is the preset second preset coefficient; is the average optical flow in the second vertical direction.

[0119] Step S40 : performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value.

[0120] Assume that there is a three-dimensional point X in space, whose two-dimensional pixel coordinates x=PX,P=K[R|t], where K is the camera intrinsic parameter matrix, Where f is the focal length of the camera, p x is the x coordinate of the principal point in the image, p y is the y coordinate of the principal point in the image. R is the camera rotation matrix, and t is the displacement of the camera center relative to the center of the world coordinate system. 01 The first image to be stitched after reprojection can be obtained by adding the new rotation matrix R1. The first focal length correction value f′ obtained in the above embodiment is 02 The reprojected second image to be stitched can be obtained by adding the new rotation matrix R2.

[0121] Step S50 , performing a stitching operation on the two re-projected images to be stitched.

[0122] In some embodiments, step S50 includes:

[0123] Step S51 : matching the third pixel of the splicing band region of the reprojected first image to be spliced ​​with the fourth pixel of the splicing band region of the reprojected second image to be spliced.

[0124] The matching method of the third pixel and the fourth pixel may refer to the matching method of the first pixel and the second pixel in the above embodiment, which will not be described in detail here.

[0125] Step S52 : calculating a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the reprojected first image to be stitched based on the position of the third pixel and the position of the corresponding matched fourth pixel.

[0126] The implementation method of step S52 refers to the optical flow calculation method in the above embodiment and will not be repeated here.

[0127] Step S53 : Calculate the composite coordinate value of each of the third pixels based on the position of the third pixel and the corresponding third horizontal optical flow and third vertical optical flow.

[0128] In some embodiments, step S53 includes: calculating the composite coordinates of the third pixel based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v.

[0129] After optical flow calculation, the corresponding relationship between pixels in the stitching band of adjacent images can be obtained. Assume that the left image of two adjacent images is L and the right image is R, and their respective stitching band areas are A and B respectively. For a pixel X(x,y) in A, its corresponding pixel in B is X′(x′,y′). Assuming that the horizontal optical flow value at X is u and the vertical optical flow value is v, the corresponding pixel in the synthesized panorama is The coordinates are In this step, the coordinates of each synthesized pixel may be determined according to the pixel coordinates of any reprojected image to be stitched and its corresponding third horizontal optical flow and third vertical optical flow.

[0130] It can be understood that, with reference to the above embodiments, in some embodiments, the reprojected image to be stitched can also be divided into 360 regions in the vertical direction, and after calculating the third horizontal average optical flow and the third vertical average optical flow corresponding to each region, the synthetic pixel coordinates are calculated based on the third horizontal average optical flow, the third vertical average optical flow and the third pixel coordinates.

[0131] Step S54 , calculating the synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel.

[0132] In some embodiments, the implementation formula of step S54 is:

[0133] Among them, f(X) is and f(X′) is the pixel value of the third pixel and the corresponding fourth pixel respectively; w and w′ are the position weight parameters corresponding to the third pixel and the corresponding fourth pixel respectively, and the sum of the two is 1. The position weight parameters are determined by the relative position of the synthesized pixel and the central axis of the splicing band of the synthesized image. w and w′ are the position weights corresponding to the two points respectively, and the sum of the two is 1, which is related to the relative position of the synthesized pixel point in the corresponding splicing band area in the synthesized image. Assume that the splicing band width is s, and the synthesized pixel point The distance from the center axis of the splicing belt is when When the point is on the left side of the central axis (d<0), then w′=1-w(if Located on the central axis of the splicing belt, then w=w′=0.5. If the point is located on the left side of the central axis in the splicing strip, then w must be greater than w′, and vice versa).

[0134] Step S55 : performing image stitching based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

[0135] Based on the above steps, after obtaining the synthesized coordinate values ​​of the synthesized pixels and their corresponding pixel values, a corresponding synthesized image is generated based on the synthesized coordinate values ​​and the pixel values ​​to achieve image stitching.

[0136] In the above image stitching method, optical flow calculations are first performed on the overlapping areas of adjacent camera images. The results of these optical flow calculations are then used to adjust the focal lengths and rotation parameters of the two adjacent cameras. The camera images are then regenerated using these adjusted parameters. To avoid abrupt changes in the adjusted images that affect the viewing experience, a distance weighting factor is introduced during the regeneration of the camera images, which increases the adjustment amplitude for areas closer to the overlapping area and decreases it for areas closer to the overlapping area. This adjustment significantly reduces the horizontal and vertical parallax in the overlapping areas of the adjacent camera images. This reduced parallax has two advantages: 1. Higher accuracy is achieved in the subsequent optical flow calculations. 2. A more natural visual transition is achieved when generating the corresponding overlapping areas in the panorama. The final step is to perform a second optical flow calculation on the adjusted image stitching strip. The stitching strip images and the results of the second optical flow calculation are combined to generate the corresponding areas in the panorama.

[0137] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0138] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0139] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0140] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A panoramic image stitching method, characterized in that: include: Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched; Calculating, based on the position of the first pixel and the position of the corresponding second pixel, a first horizontal optical flow and a first vertical optical flow for each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow for each pixel in the stitching band area of ​​the second image to be stitched; Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow; performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value; Perform stitching operation on the two reprojected images to be stitched; The determining a first camera parameter correction value of the first camera based on the first horizontal optical flow and the first vertical optical flow, and determining a second camera parameter correction value of the second camera based on the second horizontal optical flow and the second vertical optical flow includes: Divide the splicing strip into 360 areas in the vertical direction; Calculate the first horizontal average optical flow, the first vertical average optical flow, the second horizontal average optical flow, and the second vertical average optical flow corresponding to each spatial region based on the first horizontal optical flow, the first vertical optical flow, the second horizontal optical flow, and the second vertical optical flow; determining the first camera parameter correction value based on the first horizontal average optical flow and the first vertical average optical flow, and determining the second camera parameter correction value based on the second horizontal average optical flow and the second vertical average optical flow; The step of determining the first camera parameter correction value based on the first horizontal average optical flow and the first vertical average optical flow, and determining the second camera parameter correction value based on the second horizontal average optical flow and the second vertical average optical flow includes: Calculate the spatial angle between each pixel and the corresponding spatial point; Determining a distance weight parameter based on the spatial angle; determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; Determining the distance weight parameter based on the spatial angle includes: where θ 01 ,θ 02 are the spatial angles of the spatial points of the first pixel and the second pixel respectively, w 01 、w 02 are the distance weight parameters corresponding to the first camera and the second camera respectively; The step of determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow includes: A first focal length correction value and a first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is: C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; is the first focal length correction value; The preset formula of the first rotation matrix correction value R1 is: in is the first space Euler angle; h is the second preset coefficient; is the average optical flow in the first vertical direction.

2. The panoramic image stitching method according to claim 1, wherein: The step of performing a stitching operation on the two reprojected images to be stitched comprises: Matching a third pixel of the stitching band region of the reprojected first image to be stitched with a fourth pixel of the stitching band region of the reprojected second image to be stitched; Calculating a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the reprojected first image to be stitched based on the position of the third pixel and the corresponding matching position of the fourth pixel; Calculating a composite coordinate value of each of the third pixels based on the position of the third pixel and the corresponding third horizontal optical flow and third vertical optical flow; Calculating a synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel; Image stitching is performed based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

3. The panoramic image stitching method according to claim 1 or 2, wherein: The calculating, based on the position of the first pixel and the position of the corresponding second pixel, a first horizontal optical flow and a first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched comprises: Determine the first horizontal optical flow based on a difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel; Determine the first vertical optical flow based on a difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel; Determine the second horizontal optical flow based on a difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel; The second vertical optical flow is determined based on a difference value obtained by subtracting the vertical coordinate of the second pixel from the vertical coordinate of the first pixel.

4. The panoramic image stitching method according to claim 1, wherein: The step of determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow includes: The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is: C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; is the second focal length correction value; The preset formula of the second rotation matrix correction value R2 is: in is the second space Euler angle; h is the second preset coefficient; is the average optical flow in the second vertical direction.

5. The panoramic image stitching method according to claim 2, wherein: The step of calculating the composite coordinate value of each third pixel based on the position of the third pixel and the corresponding third horizontal optical flow and third vertical optical flow includes: Based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v, the composite coordinates of the third pixel are calculated. .

6. The panoramic image stitching method according to claim 5, wherein: The formula for calculating the synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel is: in, 、 are pixel values ​​of the third pixel and the corresponding fourth pixel respectively; and They are position weight parameters corresponding to the third pixel and the corresponding fourth pixel respectively, the sum of the two is 1, and the position weight parameters are determined by the relative position of the synthesized pixel and the central axis of the splicing band of the synthesized image.

Images