Panoramic image splicing method

Abstract

The invention discloses a panoramic image splicing method. The panoramic image splicing method comprises the following steps: matching pixels of splicing zone areas of two images to be spliced; performing optical flow calculation based on the position of the matched pixel; determining a first camera parameter correction value and a second camera parameter correction value of the first camera based on the optical flow calculation result; performing re-projection processing on the first to-be-spliced image based on the first camera parameter correction value, and performing re-projection processing on the second to-be-spliced image based on the second camera parameter correction value; and carrying out splicing operation on the two images to be spliced after re-projection. According to the invention, natural smooth transition of panoramic image stitching can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a panoramic image stitching method. Background Art

[0002] A panoramic image is an image that covers 360 degrees horizontally and 180 degrees vertically. Panoramic images are generally generated by synthesizing several images with overlapping content areas (hereinafter referred to as "stitching strips"), and the algorithm for synthesizing panoramic images is called a panoramic stitching algorithm. Since the shooting postures and camera parameters of the input images are not uniform, the same object in three-dimensional space will appear at different angles and positions in different two-dimensional images. When generating a 360-degree panoramic image, it is necessary to selectively use the overlapping areas of adjacent images to generate a unified synthesis result in the area.

[0003] How to process the stitching band so that the final panoramic image can have a smooth and natural transition effect at the junction of the content of different lenses is the core problem that the panoramic image stitching algorithm needs to solve. Summary of the invention

[0004] The main purpose of the present invention is to provide a panoramic image stitching method, aiming to solve the problem that the panoramic image finally generated by the prior art processing stitching band cannot have a smooth and natural transition at the intersection of the contents of different lenses.

[0005] To achieve the above object, the present invention provides a panoramic image stitching method, the panoramic image stitching method comprising:

[0006] Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched;

[0007] Based on the position of the first pixel and the position of the corresponding matched second pixel, calculating a first horizontal optical flow and a first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched;

[0008] Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow;

[0009] Performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value;

[0010] Perform stitching operation on the two images to be stitched after reprojection.

[0011] Compared with the prior art, the panoramic image stitching method provided by the present invention has the following beneficial effects: first, optical flow calculation is performed on the content overlapping area of ​​adjacent lens images, and then the focal length and rotation parameters of the two adjacent cameras are adjusted respectively using the results of the optical flow calculation, and the images of the cameras are regenerated respectively using the adjusted parameters. When regenerating the camera images, in order to avoid the abrupt changes in the adjusted images that affect the viewing effect, a distance weight coefficient is introduced here so that the adjustment amplitude of the part of the picture closer to the overlapping area is larger, and vice versa. After the above adjustment, the horizontal and vertical parallax of the content overlapping area of ​​the two adjacent camera images can be greatly reduced, and the reduction of parallax has the following two advantages: 1. A higher precision result can be obtained in the subsequent optical flow calculation. 2. When generating the corresponding overlapping area content in the panorama, there will be a more natural visual transition effect. The last step is to perform a second optical flow calculation on the adjusted image splicing band, and generate the corresponding area in the panorama by combining the splicing band image and the results of the second optical flow calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention;

[0013] Figure 2 This is a flow chart of an embodiment of a panoramic image stitching method of the present invention;

[0014] Figure 3 This is an example diagram of a stitching band in an embodiment of a panoramic image stitching method of the present invention;

[0015] Figure 4 This is an explanatory diagram of the camera field of view and the stitching band in the embodiment of the panoramic image stitching method of the present invention. DETAILED DESCRIPTION

[0016] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] In the prior art, the processing of the splicing band makes it difficult to produce a smooth and natural transition effect at the interface between the contents of different lenses in the finally generated panoramic image.

[0019] In order to solve the above technical problems, the present invention provides a panoramic image stitching method. In this method, firstly, the optical flow calculation is performed on the content overlapping area of ​​the adjacent lens images, and then the focal length and rotation parameters of the two adjacent cameras are adjusted respectively using the results of the optical flow calculation, and the images of the cameras are regenerated respectively using the adjusted parameters. When regenerating the camera image, in order to avoid the abrupt changes in the adjusted image that affect the viewing effect, a distance weight coefficient is introduced here so that the adjustment amplitude of the part of the picture closer to the overlapping area is larger, and vice versa. After the above adjustment, the horizontal and vertical parallax of the content overlapping area of ​​the two adjacent camera images can be greatly reduced, and the reduction of parallax has the following two advantages: 1. A higher precision result can be obtained in the subsequent optical flow calculation. 2. When generating the corresponding overlapping area content in the panorama, there will be a more natural visual transition effect. The last step is to perform a second optical flow calculation on the adjusted image splicing band, and generate the corresponding area in the panorama by combining the splicing band image and the result of the second optical flow calculation.

[0020] like Figure 1 As shown, Figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention.

[0021] The terminal of the embodiment of the present invention can be a terminal device with computing capability, or a PC, or a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group AudioLayer IV) player, a portable computer, or other mobile terminal devices with display capabilities.

[0022] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0023] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0024] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0025] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a panoramic image stitching program.

[0026] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and perform data communication with the client; and the processor 1001 can be used to call the panoramic image stitching program stored in the memory 1005 and perform the following operations:

[0027] Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched;

[0028] Based on the position of the first pixel and the position of the corresponding matched second pixel, calculating a first horizontal optical flow and a first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched;

[0029] Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow;

[0030] Performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value;

[0031] Perform stitching operation on the two images to be stitched after reprojection.

[0032] Further, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0033] Matching a third pixel of the stitching band area of ​​the reprojected first image to be stitched with a fourth pixel of the stitching band area of ​​the reprojected second image to be stitched;

[0034] Based on the position of the third pixel and the position of the corresponding matched fourth pixel, calculating a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched after reprojection;

[0035] Calculate a composite coordinate value of each of the third pixels based on the position of the third pixels and the corresponding third horizontal optical flow and third vertical optical flow;

[0036] Calculating a synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel;

[0037] Image stitching is performed based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

[0038] Further, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0039] Determine the first horizontal optical flow based on a difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel;

[0040] Determine the first vertical direction optical flow based on the difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel;

[0041] Determine the second horizontal optical flow based on the difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel;

[0042] The second vertical optical flow is determined based on a difference value obtained by subtracting a vertical coordinate of the second pixel from a vertical coordinate of the first pixel.

[0043] Further, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0044] Divide the splicing belt into 360 areas in the vertical direction;

[0045] Calculate the first horizontal direction average optical flow, the first vertical direction average optical flow, the second horizontal direction average optical flow, and the second vertical direction average optical flow corresponding to each spatial area based on the first horizontal direction optical flow, the first vertical direction optical flow, the second horizontal direction optical flow, and the second vertical direction optical flow;

[0046] The first camera parameter correction value is determined based on the first horizontal average optical flow and the first vertical average optical flow, and the second camera parameter correction value is determined based on the second horizontal average optical flow and the second vertical average optical flow.

[0047] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0048] Calculate the spatial angle of each pixel corresponding to the spatial point;

[0049] Determining a distance weight parameter based on the spatial angle;

[0050] The first camera parameter correction value is determined based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and the second camera parameter correction value is determined based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow.

[0051] Further, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0052] The first focal length correction value and the first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is:

[0053]

[0054] f′0=f 01 *c0

[0055] C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; f 01 is the original value of the first focal length, f′ 01 is the first focal length correction value; w 01 is the first distance weight correction parameter;

[0056] The preset formula for the first rotation matrix correction value is:

[0057]

[0058] Where Φ1 is the first space Euler angle; h is the preset second preset coefficient; is the average optical flow in the first vertical direction.

[0059] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0060] The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is:

[0061]

[0062] f′ 02 =f 02 *c 02

[0063] C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; f 02 is the original value of the second focal length, f′ 02 is the second focal length correction value;

[0064] The preset formula for the second rotation matrix correction value is:

[0065]

[0066] Wherein Φ2 is the second space Euler angle; h is the preset second preset coefficient; is the average optical flow in the second vertical direction.

[0067] Furthermore, the processor 1001 may call the panoramic image stitching program stored in the memory 1005, and further perform the following operations:

[0068] Based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v, the composite coordinates of the third pixel are calculated.

[0069] Reference Figure 2 , Figure 2 This is a flow chart of an embodiment of a panoramic image stitching method of the present invention. In some embodiments, the panoramic image stitching method includes:

[0070] Step S10: matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched.

[0071] The first image to be stitched and the second image to be stitched are two images captured by adjacent lenses and used to stitch a panoramic image. Figure 3 , Figure 3 is an example diagram of a splicing band in an embodiment of the present invention. The splicing band area refers to the overlapping area of ​​the first image to be spliced ​​and the second image to be spliced. Figure 3 The A and B areas shown. Each first pixel in the splicing band area of ​​the first image to be spliced ​​can find a corresponding matching second pixel in the splicing band area of ​​the second image to be spliced. When the method is executed, the second pixel corresponding to each first pixel is first determined. In some embodiments, the matching points of the splicing areas of the two images to be spliced ​​can be determined based on the pixel values, that is, the pixel value differences between all the first pixels and the matching second pixels are calculated, and all the matching first pixels and second pixels are determined based on the minimum value of the sum of the pixel value differences corresponding to all the first pixels.

[0072] Step S20, based on the position of the first pixel and the corresponding matching position of the second pixel, calculate the first horizontal optical flow and the first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and the second horizontal optical flow and the second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched.

[0073] Optical flow characterizes the changes in images, including information about the relative position changes of targets in different images. Optical flow includes vertical optical flow and horizontal optical flow, which characterize the position changes of images in the vertical and horizontal directions, respectively. Since optical flow has directional characteristics, it is necessary to calculate the optical flow corresponding to each image separately. Among them, the first horizontal optical flow and the first vertical optical flow refer to the optical flow determined for the first image to be stitched; the second horizontal optical flow and the second horizontal optical flow refer to the optical flow determined for the second image to be stitched. The position of a pixel can be determined based on the coordinates of the pixel.

[0074] In some embodiments, step S20 may include:

[0075] Step S21, determining the first horizontal optical flow based on the difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel;

[0076] Step S22, determining the first vertical direction optical flow based on the difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel;

[0077] Step S23, determining the second horizontal optical flow based on the difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel;

[0078] Step S24: determining the second vertical optical flow based on the difference between the vertical coordinate of the first pixel and the vertical coordinate of the second pixel.

[0079] Since the common content in adjacent images only exists in the stitching band, it is only necessary to calculate the optical flow for the stitching band area. Assume that the stitching band areas of the first image to be stitched and the second image to be stitched are A and B, respectively. For a first pixel X(x, y) in A, search for the second pixel X'(x', y') whose pixel value is closest to X in the stitching band B. Then the first horizontal optical flow of the first pixel X is u=x'-x, and the first vertical optical flow is v=y'-y. The second horizontal optical flow of the second pixel X' is u=xx', and the second vertical optical flow is v=y'-y.

[0080] Step S30: determining a first camera parameter correction value of the first camera based on the first horizontal optical flow and the first vertical optical flow, and determining a second camera parameter correction value of the second camera based on the second horizontal optical flow and the second vertical optical flow.

[0081] The first image to be stitched is acquired by a first camera, and the second image to be stitched is acquired by a second camera.

[0082] In some embodiments, the camera parameter correction values ​​may include a focal length correction value and a rotation matrix correction value.

[0083] Calculating the camera parameter correction values ​​and the subsequent rotation matrix correction values ​​requires a series of adjustments to the adjacent images to reduce the horizontal and vertical parallax. The purpose of reducing the horizontal parallax is mainly to improve the consistency of the inside and outside of the stitching strip in the final panorama generation step, and the purpose of reducing the vertical parallax is mainly to reduce the oblique deformation that may occur in the panorama stitching strip (assuming that the adjacent two images are Figure 1 The position of one content is too low, and the position of another content is too high. If the stitching strip between them is to achieve a smooth transition, it will definitely be deformed diagonally. At the same time, the reduction of horizontal and vertical parallax is conducive to reducing errors in the subsequent second optical flow calculation (the larger the parallax, the larger the search range when searching for corresponding pixels, which is prone to erroneous results). Horizontal parallax can be reduced by adjusting the camera focal length, while vertical parallax can be reduced by adjusting the camera shooting angle.

[0084] For the rotation matrix, the two lenses are rotated around the main optical axis (Z axis) by a certain angle to compensate for the vertical parallax (i.e. adjust the spatial Euler angle). To achieve the best effect, different parts of the picture need to be rotated at different angles. The area with greater vertical optical flow needs to adjust the angle more.

[0085] In some embodiments, step S30 includes:

[0086] Step S31, dividing the splicing tape into 360 regions in the vertical direction.

[0087] The visual space of the camera is regarded as a spherical space. With reference to the division direction and division method of the earth's latitude, the center of the spherical visual space is taken as the origin, and the angle of one circle in the vertical direction is 360°. Each angle can determine a dividing line of the spherical visual space (similar to the latitude line of the earth). All dividing lines divide the spherical visual space into 360 areas, that is, the splicing band is evenly divided (even division means that the corresponding division angles are all 1°) into 360 areas. In this step, the splicing band area of ​​the first image to be spliced ​​is divided into 360 areas, and the splicing band area of ​​the second image to be spliced ​​is divided into 360 areas.

[0088] Step S32, based on the first horizontal optical flow, the first vertical optical flow, the second horizontal optical flow and the second vertical optical flow, calculate the first horizontal average optical flow, the first vertical average optical flow, the second horizontal average optical flow and the second vertical average optical flow corresponding to each spatial area.

[0089] For each divided splicing band area, the average value can be calculated based on the first horizontal optical flow and the first vertical optical flow corresponding to each first pixel in the divided area of ​​the first image to be stitched, and the first horizontal average optical flow and the first vertical average optical flow of the corresponding area can be determined, and finally 360 first horizontal average optical flows and first vertical average optical flows corresponding to each of the 360 ​​areas of the splicing band of the first image to be stitched are obtained. Similarly, based on the second horizontal optical flow and the second vertical optical flow of the pixels in the 360 ​​areas of the splicing band of the second image to be stitched, 360 second horizontal average optical flows and second vertical average optical flows corresponding to each of the 360 ​​areas of the splicing band of the second image to be stitched can be obtained.

[0090] In some embodiments, since the fisheye lens image does not follow the perspective projection imaging principle of an ordinary camera and has a large deformation, it is not convenient to directly calculate the optical flow. It is necessary to first reproject the overlapping area of ​​the fisheye image and transform it to the perspective projection space. The overlapping area after the projection transformation is not necessarily a regular rectangular area and there will be areas with invalid pixels in the area (which can be regarded as having a pixel value of 0). These areas will not have valid optical flow when calculating the optical flow, so the optical flow value is regarded as zero. If these areas with an optical flow value of 0 due to invalid pixels are included in the average optical flow statistics step, the average optical flow value calculated will be smaller, so it is necessary to avoid these areas when calculating the average optical flow.

[0091] Step S33: determining the first camera parameter correction value based on the first horizontal average optical flow and the first vertical average optical flow, and determining the second camera parameter correction value based on the second horizontal average optical flow and the second vertical average optical flow.

[0092] The basic implementation of the panoramic image stitching algorithm usually does not need to process the parts outside the stitching band, but in order to avoid sudden changes inside and outside the stitching band, it is necessary to make adjustments to a certain range outside the stitching band similar to the stitching band. The closer to the stitching band, the greater the degree of adjustment, and vice versa, so a distance weight parameter needs to be calculated. Figure 4 , Figure 4 This is an explanatory diagram of the camera field of view and the stitching zone in an embodiment of the present invention. Taking a camera with a field of view of 200° as an example, the maximum effective field of view angle between one side of the central axis of the picture and the central axis of the picture is approximately 100 degrees. Since a circle of a sphere is 360 degrees, the unilateral field of view of each lens is approximately 90 to 100 degrees, which is the overlapping area with another lens, that is, the stitching zone. The unilateral viewing angle of the central axis of each lens is 100 degrees, and the stitching area of ​​the two lenses is 20 degrees. The closer to the stitching area, the greater the weight. In some embodiments, before step S33, it includes:

[0093] Step S331, calculating the spatial angle between each pixel and the corresponding spatial point.

[0094] For a point P in the panorama close to lens 0, the corresponding spatial coordinates are (x, y, z), and the spatial angle is calculated:

[0095]

[0096] A distance weight parameter is determined based on the spatial angle.

[0097] Distance weight parameter of point P for lens 0 When θ0>45°, the closer θ0 is to 100°, the greater the weight (the maximum weight is 1), otherwise the smaller the weight. When θ0≤45°, the weight is 0 and does not need to be adjusted. 45° is an empirical parameter obtained after multiple experiments. Based on the above method, w corresponding to each first pixel of the first image to be stitched can be calculated respectively. 01 and w corresponding to each second pixel of the second image to be stitched 02 .in,

[0098]

[0099] where θ 01 ,θ 02 are the spatial angles of the spatial points of the first pixel and the second pixel respectively, w 01 、w 02 are the distance weight parameters corresponding to the first camera and the second camera respectively.

[0100] Step S33 includes:

[0101] Step S331: determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow.

[0102] In some embodiments, the step of determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow includes:

[0103] The first focal length correction value and the first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is:

[0104]

[0105] f′0=f 01 *c 01

[0106] C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; f 01 is the original value of the first focal length, f′ 01 is the first focal length correction value;

[0107] The preset formula of the first rotation matrix correction value R1 is:

[0108]

[0109] Wherein, Φ1 is the first space Euler angle; h is the preset second preset coefficient; is the average optical flow in the first vertical direction. In some embodiments, h can be set to 0.27, which is an empirical coefficient obtained based on visual effects after multiple experiments.

[0110] Specifically, for a point P(x,y,z) in the panorama, the corresponding three-dimensional space latitude φ∈[-π,π], according to the latitude value, the first horizontal average optical flow corresponding to point P can be found in the average optical flow result of step S32 And the average optical flow in the first vertical direction Wherein k is a coefficient calculated according to the physical relationship between the average focal length of the lens and the parallax. For example, k can be set to 0.001068.

[0111] Similarly, in some embodiments, the step of determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow includes:

[0112] The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is:

[0113]

[0114] f′ 02 =f 02 *c 02

[0115] C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; f 02 is the original value of the second focal length, f′ 02 is the second focal length correction value;

[0116] The preset formula of the second rotation matrix correction value R2 is:

[0117]

[0118] Wherein Φ2 is the second space Euler angle; h is the preset second preset coefficient; is the average optical flow in the second vertical direction.

[0119] Step S40: performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value.

[0120] Assume that there is a three-dimensional point X in space, whose two-dimensional pixel coordinates x = PX, P = K[R|t], where K is the camera intrinsic parameter matrix, Where f is the focal length of the camera, p x is the x coordinate of the principal point in the image, p y is the y coordinate of the principal point in the image. R is the camera rotation matrix, and t is the displacement of the camera center relative to the center of the world coordinate system. 01 The first image to be stitched after reprojection can be obtained by adding the new rotation matrix R1. The first focal length correction value f′ obtained in the above embodiment is 02 The second image to be stitched after reprojection can be obtained by adding the new rotation matrix R2.

[0121] Step S50, performing a stitching operation on the two reprojected images to be stitched.

[0122] In some embodiments, step S50 includes:

[0123] Step S51 : matching the third pixel of the stitching band area of ​​the reprojected first image to be stitched with the fourth pixel of the stitching band area of ​​the reprojected second image to be stitched.

[0124] The matching method of the third pixel and the fourth pixel may refer to the matching method of the first pixel and the second pixel in the above embodiment, which will not be described in detail here.

[0125] Step S52: Calculate a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched after reprojection based on the position of the third pixel and the position of the corresponding matched fourth pixel.

[0126] The implementation method of step S52 refers to the optical flow calculation method in the above embodiment, which will not be described in detail here.

[0127] Step S53: Calculate the composite coordinate value of each of the third pixels based on the position of the third pixels and the corresponding third horizontal optical flow and third vertical optical flow.

[0128] In some embodiments, step S53 includes: calculating the composite coordinates of the third pixel based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v.

[0129] After the optical flow calculation, the corresponding relationship between the pixels in the stitching band of adjacent images can be obtained. Assume that the left image of the two adjacent images is L and the right image is R, and their respective stitching band areas are A and B respectively. For a pixel X(x,y) in A, its corresponding pixel in B is X′(x′,y′). Assuming that the horizontal optical flow value at X is u and the vertical optical flow value is v, the corresponding pixel in the synthesized panorama is The coordinates of In this step, the coordinates of each synthesized pixel may be determined according to the pixel coordinates of any image to be stitched after reprojection and the corresponding third horizontal optical flow and third vertical optical flow.

[0130] It can be understood that, with reference to the above embodiments, in some embodiments, the reprojected image to be stitched can also be divided into 360 regions in the vertical direction, and after the third horizontal average optical flow and the third vertical average optical flow corresponding to each region are calculated, the synthetic pixel coordinates are calculated based on the third horizontal average optical flow, the third vertical average optical flow and the third pixel coordinates.

[0131] Step S54, calculating the synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel.

[0132] In some embodiments, the implementation formula of step S54 is:

[0133] Among them, f(X) and f(X′) are the pixel values ​​of the third pixel and the corresponding fourth pixel respectively; w and w′ are the position weight parameters corresponding to the third pixel and the corresponding fourth pixel respectively, and the sum of the two is 1, and the position weight parameters are determined by the relative position of the synthesized pixel and the central axis of the splicing band of the synthesized image. w and w′ are the position weights corresponding to the two points respectively, and the sum of the two is 1, which is related to the relative position of the synthesized pixel point in the corresponding splicing band area in the synthesized image. Assume that the splicing band width is s, and the synthesized pixel point The distance from the center axis of the splicing belt is when When the point is on the left side of the central axis (d<0), then w′=1-w(if Located on the central axis of the splicing belt, then w=w′=0.5. If the point is located on the left side of the central axis in the splicing belt, then w must be greater than w′, and vice versa.

[0134] Step S55: performing image stitching based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

[0135] Based on the above steps, after the synthesized coordinate values ​​of the synthesized pixels and their corresponding pixel values ​​are obtained, a corresponding synthesized image is generated based on the synthesized coordinate values ​​and the pixel values ​​to achieve image stitching.

[0136] In the above image stitching method, the optical flow calculation is first performed on the content overlapping area of ​​the adjacent lens images, and then the focal length and rotation parameters of the two adjacent cameras are adjusted using the results of the optical flow calculation, and the camera images are regenerated using the adjusted parameters. When regenerating the camera images, in order to avoid the abrupt changes in the adjusted images that affect the viewing effect, a distance weight coefficient is introduced so that the closer the image is to the overlapping area, the larger the adjustment amplitude is, and vice versa. After the above adjustment, the two adjacent camera images can greatly reduce the horizontal and vertical parallax of the content overlapping area, and the reduction of parallax has the following two advantages: 1. A higher precision result can be obtained in the subsequent optical flow calculation. 2. When generating the corresponding overlapping area content in the panorama, there will be a more natural visual transition effect. The last step is to perform a second optical flow calculation on the adjusted image splicing band, and combine the splicing band image and the results of the second optical flow calculation to generate the corresponding area in the panorama.

[0137] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0138] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0140] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A panoramic image stitching method, characterized in that: include: Matching a first pixel of a stitching band region of a first image to be stitched with a second pixel of a stitching band region of a second image to be stitched; Based on the position of the first pixel and the position of the corresponding matched second pixel, calculating a first horizontal optical flow and a first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched, and a second horizontal optical flow and a second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched; Determine a first camera parameter correction value for a first camera based on the first horizontal optical flow and the first vertical optical flow, and determine a second camera parameter correction value for a second camera based on the second horizontal optical flow and the second vertical optical flow; Performing a reprojection process on the first image to be stitched based on the first camera parameter correction value, and performing a reprojection process on the second image to be stitched based on the second camera parameter correction value; Perform stitching operation on the two images to be stitched after reprojection.

2. The panoramic image stitching method according to claim 1, characterized in that: The step of performing a stitching operation on the two images to be stitched after reprojection comprises: Matching a third pixel of the stitching band area of ​​the reprojected first image to be stitched with a fourth pixel of the stitching band area of ​​the reprojected second image to be stitched; Based on the position of the third pixel and the position of the corresponding matched fourth pixel, calculating a third horizontal optical flow and a third vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched after reprojection; Calculate a composite coordinate value of each of the third pixels based on the position of the third pixels and the corresponding third horizontal optical flow and third vertical optical flow; Calculating a synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel; Image stitching is performed based on the synthesized coordinate values ​​of each synthesized pixel and the corresponding synthesized pixel value.

3. The panoramic image stitching method according to claim 1 or 2, characterized in that: The step of calculating the first horizontal optical flow and the first vertical optical flow of each pixel in the stitching band area of ​​the first image to be stitched and the second horizontal optical flow and the second vertical optical flow of each pixel in the stitching band area of ​​the second image to be stitched based on the position of the first pixel and the corresponding matching position of the second pixel comprises: Determine the first horizontal optical flow based on a difference between the horizontal coordinate of the second pixel and the horizontal coordinate of the first pixel; Determine the first vertical direction optical flow based on the difference between the vertical coordinate of the second pixel and the vertical coordinate of the first pixel; Determine the second horizontal optical flow based on the difference between the horizontal coordinate of the first pixel and the horizontal coordinate of the second pixel; The second vertical optical flow is determined based on a difference value obtained by subtracting a vertical coordinate of the second pixel from a vertical coordinate of the first pixel.

4. The panoramic image stitching method according to claim 3, characterized in that: The determining a first camera parameter correction value of the first camera based on the first horizontal optical flow and the first vertical optical flow, and determining a second camera parameter correction value of the second camera based on the second horizontal optical flow and the second vertical optical flow includes: Divide the splicing belt into 360 areas in the vertical direction; Calculate the first horizontal average optical flow, the first vertical average optical flow, the second horizontal average optical flow, and the second vertical average optical flow corresponding to each spatial area based on the first horizontal optical flow, the first vertical optical flow, the second horizontal optical flow, and the second vertical optical flow; The first camera parameter correction value is determined based on the first horizontal average optical flow and the first vertical average optical flow, and the second camera parameter correction value is determined based on the second horizontal average optical flow and the second vertical average optical flow.

5. The panoramic image stitching method according to claim 4, characterized in that: Also includes: Calculate the spatial angle of each pixel corresponding to the spatial point; Determining a distance weight parameter based on the spatial angle; The step of determining the first camera parameter correction value based on the first horizontal average optical flow and the first vertical average optical flow, and determining the second camera parameter correction value based on the second horizontal average optical flow and the second vertical average optical flow comprises: The first camera parameter correction value is determined based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow, and the second camera parameter correction value is determined based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow.

6. The panoramic image stitching method according to claim 5, characterized in that: Determining the distance weight parameter based on the spatial angle includes: where θ 01 ,θ 02 are the spatial angles of the spatial points of the first pixel and the second pixel respectively, w 01 、w 02 are the distance weight parameters corresponding to the first camera and the second camera respectively.

7. The panoramic image stitching method according to claim 6, characterized in that: The step of determining the first camera parameter correction value based on the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow comprises: The first focal length correction value and the first rotation matrix correction value of the first camera are determined based on a preset calculation formula, the distance weight parameter, the first horizontal average optical flow, and the first vertical average optical flow; wherein the preset formula for the first focal length correction value is: f′0=f 01 *c 01 C 01 is the first focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the first horizontal direction; f 01 is the original value of the first focal length, f′ 01 is the first focal length correction value; The preset formula of the first rotation matrix correction value R1 is: Where Φ1 is the first spatial Euler angle; h is the second preset coefficient; is the average optical flow in the first vertical direction.

8. The panoramic image stitching method according to claim 6, wherein: The step of determining the second camera parameter correction value based on the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow comprises: The second focal length correction value and the second rotation matrix correction value of the second camera are determined based on a preset formula, the distance weight parameter, the second horizontal average optical flow, and the second vertical average optical flow; wherein the preset formula for the second focal length correction value is: f′ 02 =f 02 *c 02 C 02 is the focal length correction coefficient; k is the first preset coefficient; is the average optical flow in the second horizontal direction; f 02 is the original value of the second focal length, f′ 02 is the second focal length correction value; The preset formula of the second rotation matrix correction value R2 is: Where Φ2 is the second space Euler angle; h is the second preset coefficient; is the average optical flow in the second vertical direction.

9. The panoramic image stitching method according to claim 2, characterized in that: The step of calculating the synthetic coordinate value of each of the third pixels based on the position of the third pixel and the corresponding third horizontal optical flow and third vertical optical flow comprises: Based on the coordinates (x, y) of the third pixel and the third horizontal optical flow u and the third vertical optical flow v, the composite coordinates of the third pixel are calculated.

10. The panoramic image stitching method according to claim 9, characterized in that: The formula for calculating the synthesized pixel value of each synthesized pixel based on the pixel value of the third pixel, the pixel value of the corresponding matched fourth pixel, and the synthesized coordinate value of the corresponding synthesized pixel is: Where f(X) is ′ ) are the pixel values ​​of the third pixel and the corresponding fourth pixel respectively; w and w′ are the position weight parameters corresponding to the third pixel and the corresponding fourth pixel respectively, and the sum of the two is 1, and the position weight parameters are determined by the relative position of the synthetic pixel and the central axis of the splicing band of the synthetic image.

Images