Panoramic image stitching method, device and panoramic camera

Abstract

The present application provides a panoramic image stitching method, device and panoramic camera, the method comprising: obtaining a first image obtained by scanning a photosensitive chip of a first camera along a first direction and a second image obtained by scanning a photosensitive chip of a second camera along a direction opposite to the first direction, calculating the optical flow of the overlapping area of ​​the first image and the second image, obtaining the optical flow vector of the feature point of the overlapping area, any feature point on the first image is recorded as x, and the pixel value is recorded as l(x), the feature point on the second image after being mapped by the optical flow vector is recorded as x', and the pixel value is recorded as l'(x'), and according to l(x) and l'(x'), the pixel value x after x and x' are fused is calculated c , according to the fused pixel value x c , the two images are stitched together to obtain a panoramic image. The present application can reduce the exposure difference in the overlapping area of ​​the images in the stitching of the panoramic image.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of panoramic image stitching, and specifically to a panoramic image stitching method, device and panoramic camera. Background Art

[0002] A panoramic image is an image obtained by shooting 360 degrees in all directions with a panoramic camera. Existing panoramic cameras generally include two or more fisheye lenses. A panoramic image is obtained by stitching images taken with multiple fisheye lenses.

[0003] When a panoramic camera with a CMOS sensor uses a rolling shutter, if the object being photographed moves at a high speed or vibrates rapidly relative to the panoramic camera, the CMOS sensor's line-by-line scanning speed is insufficient and the shooting result may appear "tilted", "shaky" or "partially exposed", which is a jello effect.

[0004] The solution of the prior art proposes a scheme of scanning the photosensitive chips of adjacent cameras in opposite directions to solve the problem of jello effect caused by the insufficient line-by-line scanning speed of the CMOS sensor. However, there is still a slight exposure difference in the overlapping area of ​​the two images, which causes the stitching gap of the panoramic image to be more obvious and the stitching effect of the panoramic image to be poor. Summary of the invention

[0005] The embodiments of the present application provide a panoramic image stitching method, device and panoramic camera. The technical problem to be solved is that, in view of the defects of the prior art, there is a slight exposure difference in the overlapping area of ​​the two images in the existing panoramic image stitching, which leads to obvious stitching gaps in the panoramic image and poor stitching effect of the panoramic image.

[0006] In order to solve the above technical problems, a technical solution adopted by the present application is to provide a panoramic image stitching method, device and panoramic camera, wherein the panoramic image stitching method comprises:

[0007] Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0008] Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0009] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0010] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c ;

[0011] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0012] In a specific implementation, the fused pixel value x c satisfy:

[0013] x c =k0l(x)+k1l'(x');

[0014] Among them, k0 and k1 are the normalized fusion coefficients, satisfying:

[0015] k0+k1=1.

[0016] In a specific implementation, k0 satisfies:

[0017] k0=f k (t);

[0018] Wherein, t is the exposure time of the overlapped area, t0≤t≤t1, wherein t0 is the starting exposure time of the overlapped area, and t1 is the ending exposure time of the overlapped area;

[0019] During the exposure time of the overlapped area, the f k The curvature of (t) is smooth and relative to f k (t') is symmetrical, wherein t' is the middle exposure time of the overlapping area, t'=(t0+t1) / 2.

[0020] In a specific embodiment, the f k (t) is a cubic function of one variable, and f is set k (t)=a+bt+ct 2 +dt 3 ;

[0021] At the initial exposure time of the overlapped area, f k (t0) is 0, at the end exposure time of the overlap area, f k (t1)=1, at the middle exposure time of the overlapped area, f k (t') = 0.5;

[0022] The fk(t) also has a fixed value f k (t c ), according to the fk (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t).

[0023] In a specific embodiment, the f k (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t) includes:

[0024] According to the f k (t0)=0, the f k (t1)=1, the f k (t') = 0.5, the f k (t c )=f k (t c ), establish the f k (t)=a+bt+ct 2 +dt 3 The vector representation of the linear equations and the matrix equation is obtained: Ax=B;

[0025] When R(A)=R(A,B), the linear equation system has a unique solution, and the f is obtained by calculation. k The solution of (t).

[0026] In a specific embodiment, a panoramic image stitching device is also included, the device comprising:

[0027] An acquisition module, the acquisition module is used to acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0028] A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0029] A first calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, wherein any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); and the feature point on the second image after being mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0030] A second calculation module, the second calculation module is used to calculate the pixel value x after the fusion of the x and the x' according to the l(x) and the l'(x'). c ;

[0031] A splicing module is used to combine the fused pixel values ​​x c , the first image and the second image are stitched together to obtain a panoramic image.

[0032] In a specific embodiment, a panoramic camera is also included, and the panoramic camera includes:

[0033] A camera module, the camera module comprising a first camera and a second camera, the first camera and the second camera being arranged vertically side by side;

[0034] A control module, the control module comprising a first control module and a second control module, the first control module controls the photosensitive chip of the first camera to scan along a first direction to obtain a first image, and the second control module controls the photosensitive chip of the second camera to scan along a second direction to obtain a second image, the first direction and the second direction are opposite;

[0035] A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0036] A calculation module, the calculation module includes a first calculation module and a second calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, any feature point on the first image is recorded as x, and the pixel value is recorded as l(x), the feature point on the second image after the optical flow vector is mapped is recorded as x', and the pixel value is recorded as l'(x'), the second calculation module is used to calculate the pixel value x after the fusion of x and x' according to l(x) and l'(x'). c ;

[0037] A splicing module is used to combine the fused pixel values ​​x c , the first image and the second image are stitched together to obtain a panoramic image.

[0038] In a specific embodiment, the photosensitive chip of the first camera and the photosensitive chip of the second camera are CMOS image sensors, the first direction is toward or away from the photosensitive chip of the second camera, and the second direction is toward or away from the photosensitive chip of the first camera.

[0039] In a specific embodiment, the first control module is further used to control the photosensitive chip of the first camera to scan along a first direction to obtain a first image when a shooting instruction is obtained; the second control module is further used to control the photosensitive chip of the second camera to scan along a second direction to obtain a second image when the shooting instruction is obtained.

[0040] In a specific embodiment, the camera module also includes a third camera and a fourth camera, and the third camera and the fourth camera are vertically arranged side by side, and are arranged in sequence around the first camera and the second camera, and the scanning directions of the photosensitive chips of the two adjacent cameras are opposite, and the photosensitive chip of the third camera and the photosensitive chip of the fourth camera are CMOS image sensors.

[0041] The beneficial effect of the present application is that, different from the prior art, the present application provides a panoramic image stitching method, device and panoramic camera, wherein the panoramic image stitching method includes:

[0042] Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0043] Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0044] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0045] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c ;

[0046] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0047] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0048] By calculating the optical flow of the overlapping area of ​​the first image and the second image, an optical flow vector of the feature point of the overlapping area is obtained, and the pixel value of the feature point on the first image is smoothly fused with the pixel value of the feature point after being mapped by the optical flow vector to obtain a fused pixel value x c, and according to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image, so that the overlapping area of ​​the first image and the second image can be smoothly transitioned, reducing the problem of obvious stitching gaps in the overlapping area caused by exposure differences, and improving the stitching quality of the panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the implementation modes of the present application, the drawings required for use in the description of the implementation modes will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a schematic diagram of the implementation process of a panoramic image stitching method provided in Example 1 of the present application;

[0051] Figure 2 It is a scanning schematic diagram of a photosensitive chip of a camera in a panoramic image stitching method provided in Example 1 of the present application;

[0052] Figure 3 is a panoramic image stitching method provided in the first embodiment of the present application. k (t) Schematic diagram of function curve;

[0053] Figure 4 This is a schematic diagram of the module composition of a panoramic image stitching device provided in Example 2 of the present application;

[0054] Figure 5 It is a schematic diagram of the internal structure of a panoramic camera provided in Example 3 of the present application.

[0055] Reference numerals:

[0056] 500, panoramic camera; 501, first camera; 502, second camera; 503, third camera; 504, fourth camera. DETAILED DESCRIPTION

[0057] The present application is further described in detail below in conjunction with the accompanying drawings and implementation methods. It is particularly noted that the following implementation methods are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following implementation methods are only some implementation methods of the present application rather than all implementation methods. All other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to these processes, methods, products or devices.

[0059] The above terms are only for the convenience of description and should not be construed as limitations on the present technical solution.

[0060] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] The specific implementation of the present application is described in detail below in conjunction with specific embodiments.

[0062] Embodiment 1.

[0063] Figure 1 The implementation process of a panoramic image stitching method provided in Example 1 of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown, which is described in detail as follows.

[0064] A first image is obtained by scanning a photosensitive chip of a first camera along a first direction, and a second image is obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction.

[0065] It should be noted that a panoramic image (PANORAMIC PHOTO, or PANORAMA) usually refers to an image captured within a 360-degree full scene range, with a normal viewing angle of approximately 90 degrees horizontally and 70 degrees vertically, or including a peripheral vision angle of approximately 180 degrees horizontally and 90 degrees vertically.

[0066] The formation of the jello effect is determined by the characteristics of the camera itself. Most cameras that use complementary metal oxide semiconductor (CMOS) sensors use a rolling shutter, which takes pictures by exposing the image sensor line by line.

[0067] The exposure process of a CMOS sensor can be roughly understood as power off (electrical signal cleared) - exposure - power on (light source converts electrical signal into imaging). Moreover, the exposure process of a CMOS sensor is performed in a row-by-row or column-by-column scanning mode. This working mode of a CMOS sensor is called a rolling shutter working mode.

[0068] When a camera using a CMOS sensor uses a rolling shutter, if the object being photographed moves at high speed or vibrates rapidly relative to the camera, the CMOS sensor will not scan quickly enough, and the shooting result may appear "tilted", "wobbly" or "partially exposed", which is called the jello effect. For example, when a camera shoots a person who moves quickly from right to left, the exposure process of the camera's CMOS sensor uses a top-to-bottom line-by-line scanning method. The relative movement of the person and the camera causes the jello effect, and the CMOS sensor's light-sensitive imaging is tilted.

[0069] like Figure 2 As shown, Figure 2 This is a scanning diagram of the camera's photosensitive chip. Figure 2 a in the figure is a scanning schematic diagram of the photosensitive chip of the first camera, 201 is the first direction, Figure 2 b is a scanning schematic diagram of the photosensitive chip of the second camera, 202 is the second direction, the first direction 201 and the second direction 202 are opposite, the first image is obtained by scanning the photosensitive chip of the first camera along the first direction 201, and the second image is obtained by scanning the photosensitive chip of the second camera along the second direction 202.

[0070] According to preset camera extrinsics, the first image and the second image are aligned in a camera coordinate system.

[0071] It should be noted that the first camera and the second camera have been calibrated before leaving the factory. According to the preset camera extrinsics, the first image and the second image can be aligned in the camera coordinate system, and the coordinate transformation of the pixel points of the first image and the pixel points of the second image can be completed for subsequent image stitching.

[0072] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature points of the overlapping area, where any feature point on the first image is denoted as x, and its pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and its pixel value is denoted as l'(x').

[0073] It should be noted that if Figure 2As shown, the photosensitive chip of the first camera scans the overlapping area at time t0, and scans the boundary of the overlapping area at time t1. The photosensitive chip of the second camera also scans the overlapping area at time t0, and scans the boundary of the overlapping area at time t1. At this time, if the first image and the second image are directly fused and spliced, the pixels scanned at time t1 and the pixels scanned at time t0 will be fused. However, their exposure times differ by about t1-t0. Therefore, if the shooting environment changes slightly during this period, it will cause errors in subsequent optical flow calculations or even failure.

[0074] At the midpoint between t0 and t1, that is, Figure 2 At the moment t' in the figure, the exposure time of the first image and the exposure time of the second image are completely consistent, and during the scanning process from t0 to t', the difference in exposure time of pixels of the first image and the overlapping area of ​​the second image is gradually decreasing, and during the scanning process from t' to t1, the difference in exposure time of pixels of the first image and the overlapping area of ​​the second image is gradually increasing.

[0075] In the overlapped area, the optical flow of the feature points of the first image and the second image is calculated to obtain the optical flow vector of the feature points of the overlapped area, which is the quantitative representation of the time difference between the first image and the second image. Any feature point on the first image is denoted as x, and the pixel value, i.e., the pixel coordinate, is denoted as l(x). The feature point on the second image after the optical flow vector mapping is denoted as x', and the pixel value, i.e., the pixel coordinate, is denoted as l'(x'). The fusion calculation of the feature points in the overlapped area can be performed through l(x) and l'(x').

[0076] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c .

[0077] It should be noted that the fused pixel value x is obtained by fusing the feature point x in the first image and the feature point x' on the second image after optical flow vector mapping. c , the fusion of the pixels in the overlapping area can be achieved, and then the panoramic image can be stitched.

[0078] One specific implementation method is that the fused pixel value x c satisfy:

[0079] x c =k0l(x)+k1l'(x');

[0080] Among them, k0 and k1 are the normalized fusion coefficients, satisfying:

[0081] k0+k1=1.

[0082] It should be noted that the purpose of fusing the feature point x in the first image with the feature point x' on the second image after optical flow vector mapping is to remove the ghost caused by the parallax factor in the overlapped area and the stitching traces caused by the difference in exposure time. The error caused by the parallax factor has been completed in the process of matching the feature points of the first image and the second image in the optical flow calculation of the overlapped area, that is, the feature point x' after optical flow vector mapping. Therefore, for the pixel value x c The fusion algorithm only needs to deal with the difference caused by the exposure time, thereby decoupling the processing flow of the parallax factor and the exposure time difference.

[0083] The x c =k0l(x)+k1l'(x') is the pixel value x c A specific implementation of the fusion algorithm, wherein k0 and k1 are normalized fusion coefficients, satisfying: k0+k1=1.

[0084] One specific implementation method is that k0 satisfies:

[0085] k0=f k (t);

[0086] Wherein, t is the exposure time of the overlapped area, t0≤t≤t1, wherein t0 is the starting exposure time of the overlapped area, and t1 is the ending exposure time of the overlapped area;

[0087] During the exposure time of the overlapped area, the f k The curvature of (t) is smooth and relative to f k (t') is symmetrical, wherein t' is the middle exposure time of the overlapping area, t'=(t0+t1) / 2.

[0088] It should be noted that, from the above analysis, k0 and k1 are parameters related to the exposure time, so k0 is set to satisfy: k0=f k (t).

[0089] During the exposure time of the overlapped area, i.e., the time period t0≤t≤t1, the pixels at the start exposure time t0 of the overlapped area and the end exposure time t1 of the overlapped area are the stitching contour lines. Therefore, in order to obtain a smooth transition stitching, the f k The curvature of (t) should be smooth and relative to f k(t') symmetric.

[0090] One specific implementation method is that the f k (t) is a cubic function of one variable, and f is set k (t)=a+bt+ct 2 +dt 3 ;

[0091] At the initial exposure time of the overlapped area, f k (t0) is 0, at the end exposure time of the overlap area, f k (t1)=1, at the middle exposure time of the overlapped area, f k (t') = 0.5;

[0092] The f k (t) also has a fixed value f k (t c ), according to the f k (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t).

[0093] It should be noted that according to the f k (t), construct the curvature characteristics of f k The functional relationship of (t) is a cubic function of one variable, such as Figure 3 As shown, set the f k (t)=a+bt+ct 2 +dt 3 , so the coordinates of 4 points are needed to calculate the f k (t) is the solution of the equation.

[0094] From the known starting exposure time in the overlapped area, f k (t0) is 0, at the end exposure time of the overlap area, f k (t1)=1, at the middle exposure time of the overlapped area, f k (t')=0.5, according to the f k (t) is the curvature of the function curve. In order to achieve a better splicing effect, a fixed point P is set according to the actual splicing experience, denoted as (t c, f k (t c )), therefore according to the f k (t0), said f k (t1), said f k(t') and f k (t c ), and obtain the f k The solution of (t).

[0095] One specific implementation method is that according to the f k (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t) includes:

[0096] According to the f k (t0)=0, the f k (t1)=1, the f k (t') = 0.5, the f k (t c )=f k (t c ), establish the f k (t)=a+bt+ct 2 +dt 3 The vector representation of the linear equations and the matrix equation is obtained: Ax=B;

[0097] When R(A)=R(A,B), the linear equation system has a unique solution, and the f is obtained by calculation. k The solution of (t).

[0098] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0099] It should be noted that the pixel value x c The corresponding mapping relationship between the feature points in the overlapping area of ​​the first image and the second image is reflected. The coordinates of each fused pixel value in the overlapping area can be obtained according to the corresponding mapping relationship. According to the coordinates of the fused pixel value, the smooth fusion of the overlapping area of ​​the first image and the second image can be achieved to obtain a stitched panoramic image, thereby improving the stitching quality of the panoramic image.

[0100] Embodiment 2.

[0101] The present application embodiment provides a panoramic image stitching device, such as Figure 4 As shown, Figure 4 A schematic diagram of the module composition of a panoramic image stitching device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0102] The panoramic image stitching device comprises:

[0103] An acquisition module, the acquisition module is used to acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0104] A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0105] A first calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, wherein any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); and the feature point on the second image after being mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0106] A second calculation module, the second calculation module is used to calculate the pixel value x after the fusion of the x and the x' according to the l(x) and the l'(x'). c ;

[0107] A splicing module is used to combine the fused pixel values ​​x c , the first image and the second image are stitched together to obtain a panoramic image.

[0108] A panoramic image stitching device according to an embodiment of the present application is used to implement a panoramic image stitching method provided in any specific implementation manner of the first embodiment of the present application:

[0109] Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0110] Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0111] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0112] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c ;

[0113] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0114] It should be noted that each module in the panoramic image stitching device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0115] It should be noted that, for technical details that are not described in detail in this embodiment, reference can be made to the panoramic image stitching method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0116] Embodiment three.

[0117] In an embodiment of the present application, a panoramic camera is provided. Figure 5 Schematic diagram of the internal structure of the panoramic camera.

[0118] The panoramic camera comprises:

[0119] A camera module, the camera module comprising a first camera and a second camera, the first camera and the second camera being arranged vertically side by side;

[0120] A control module, the control module comprising a first control module and a second control module, the first control module controls the photosensitive chip of the first camera to scan along a first direction to obtain a first image, and the second control module controls the photosensitive chip of the second camera to scan along a second direction to obtain a second image, the first direction and the second direction are opposite;

[0121] A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0122] A calculation module, the calculation module includes a first calculation module and a second calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, any feature point on the first image is recorded as x, and the pixel value is recorded as l(x), the feature point on the second image after the optical flow vector is mapped is recorded as x', and the pixel value is recorded as l'(x'), the second calculation module is used to calculate the pixel value x after the fusion of x and x' according to l(x) and l'(x'). c ;

[0123] A splicing module is used to combine the fused pixel values ​​x c, the first image and the second image are stitched together to obtain a panoramic image.

[0124] The panoramic camera is used to implement the panoramic image stitching method provided by any specific implementation method of the first embodiment of the present application:

[0125] Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0126] Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0127] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0128] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c ;

[0129] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0130] It should be noted that, for technical details that are not described in detail in this embodiment, reference can be made to the panoramic image stitching method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0131] In one specific implementation, the photosensitive chip of the first camera and the photosensitive chip of the second camera are CMOS image sensors, the first direction is toward or away from the photosensitive chip of the second camera, and the second direction is toward or away from the photosensitive chip of the first camera.

[0132] One specific implementation is that the first control module is also used to control the photosensitive chip of the first camera to scan along a first direction to obtain a first image when a shooting instruction is obtained; the second control module is also used to control the photosensitive chip of the second camera to scan along a second direction to obtain a second image when the shooting instruction is obtained.

[0133] One specific implementation is that the camera module also includes a third camera and a fourth camera, and the third camera and the fourth camera are vertically arranged side by side, and are arranged in sequence around the first camera and the second camera, and the scanning directions of the photosensitive chips of the two adjacent cameras are opposite, and the photosensitive chip of the third camera and the photosensitive chip of the fourth camera are CMOS image sensors.

[0134] It should be noted that if Figure 5 As shown, the panoramic camera 500 has four cameras, including the first camera 501, the second camera 502, the third camera 503, and the fourth camera 504. The first camera 501, the second camera 502, the third camera 503, and the fourth camera 504 are arranged in sequence along the periphery of the panoramic camera 500, and the scanning directions of the photosensitive chips of two adjacent cameras are opposite. The photosensitive chips of the first camera 501, the second camera 502, the third camera 503, and the fourth camera 504 are all CMOS image sensors.

[0135] Different from the prior art, the embodiments of the present application provide a panoramic image stitching method, device and panoramic camera, wherein the panoramic image stitching method includes:

[0136] Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction;

[0137] Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics;

[0138] Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x');

[0139] According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c ;

[0140] According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

[0141] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0142] By calculating the optical flow of the overlapping area of ​​the first image and the second image, an optical flow vector of the feature point of the overlapping area is obtained, and the pixel value of the feature point on the first image is smoothly fused with the pixel value of the feature point after being mapped by the optical flow vector to obtain a fused pixel value x c , and according to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image, so that the overlapping area of ​​the first image and the second image can be smoothly transitioned, reducing the problem of obvious stitching gaps in the overlapping area caused by exposure differences, and improving the stitching quality of the panoramic image.

[0143] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0145] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A panoramic image stitching method, characterized in that: The method comprises: Acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction; Aligning the first image and the second image in a camera coordinate system according to preset camera extrinsics; Calculate the optical flow of the overlapping area of ​​the first image and the second image to obtain the optical flow vector of the feature point of the overlapping area, where any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); the feature point on the second image mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x'); According to the l(x) and the l'(x'), the pixel value x after the fusion of the x and the x' is calculated c , the fused pixel value x c Satisfies: x c =k0l(x)+k1l'(x'); Among them, k0 and k1 are the fusion coefficients of normalization processing, satisfying: k0+k1=1; The k0 satisfies: k0 = f k (t); Wherein, t is the exposure time of the overlapped area, t0≤t≤t1, wherein t0 is the starting exposure time of the overlapped area, and t1 is the ending exposure time of the overlapped area; During the exposure time of the overlapped area, the f k The curvature of (t) is smooth and relative to f k (t') symmetric, wherein t' is the middle exposure time of the overlapped area, t'=(t0+t1) / 2; According to the fused pixel value x c , the first image and the second image are stitched together to obtain a panoramic image.

2. A panoramic image stitching method according to claim 1, characterized in that: The f k (t) is a cubic function of one variable, and f is set k (t) = a + bt + ct 2 +dt 3 ; At the initial exposure time of the overlapped area, f k (t0) is 0, at the end exposure time of the overlap area, f k (t1) = 1, at the middle exposure time of the overlapped area, f k (t') = 0.5; The f k (t) also has a fixed value f k (t c ), according to the f k (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t).

3. A panoramic image stitching method according to claim 2, characterized in that: According to the f k (t0), said f k (t1), said f k (t') and f k (t c ), and obtain the f k The solution of (t) includes: According to the f k (t0) = 0, the f k (t1)=1, the f k (t') = 0.5, the f k (t c ) = f k (t c ), establish the f k (t) = a + bt + ct 2 +dt 3 The vector representation of the linear equations and the matrix equation is obtained: Ax = B; When R(A)=R(A,B), the linear equation system has a unique solution, and the f is obtained by calculation. k The solution of (t).

4. A panoramic image stitching device, characterized in that: The device comprises: An acquisition module, the acquisition module is used to acquire a first image obtained by scanning a photosensitive chip of a first camera along a first direction, and acquire a second image obtained by scanning a photosensitive chip of a second camera along a second direction, wherein the first direction is opposite to the second direction; A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics; A first calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, wherein any feature point on the first image is denoted as x, and the pixel value is denoted as l(x); and the feature point on the second image after being mapped by the optical flow vector is denoted as x', and the pixel value is denoted as l'(x'); A second calculation module, the second calculation module is used to calculate the pixel value x after the fusion of the x and the x' according to the l(x) and the l'(x'). c , the fused pixel value x c Satisfies: x c =k0l(x)+k1l'(x'); Among them, k0 and k1 are the fusion coefficients of normalization processing, satisfying: k0+k1=1; The k0 satisfies: k0 = f k (t); Wherein, t is the exposure time of the overlapped area, t0≤t≤t1, wherein t0 is the starting exposure time of the overlapped area, and t1 is the ending exposure time of the overlapped area; During the exposure time of the overlapped area, the f k The curvature of (t) is smooth and relative to f k (t') symmetric, wherein t' is the middle exposure time of the overlapped area, t'=(t0+t1) / 2; A splicing module is used to combine the fused pixel values ​​x c , the first image and the second image are stitched together to obtain a panoramic image.

5. A panoramic camera, characterized in that: The panoramic camera comprises: A camera module, the camera module comprising a first camera and a second camera, the first camera and the second camera being arranged vertically side by side; A control module, the control module comprising a first control module and a second control module, the first control module controls the photosensitive chip of the first camera to scan along a first direction to obtain a first image, and the second control module controls the photosensitive chip of the second camera to scan along a second direction to obtain a second image, the first direction and the second direction are opposite; A calibration module, wherein the calibration module is used to align the first image and the second image in a camera coordinate system according to preset camera extrinsics; A calculation module, the calculation module includes a first calculation module and a second calculation module, the first calculation module is used to calculate the optical flow of the overlapping area of ​​the first image and the second image, and obtain the optical flow vector of the feature point of the overlapping area, any feature point on the first image is recorded as x, and the pixel value is recorded as l(x), the feature point on the second image after the optical flow vector is mapped is recorded as x', and the pixel value is recorded as l'(x'), the second calculation module is used to calculate the pixel value x after the fusion of x and x' according to l(x) and l'(x'). c , the fused pixel value x c Satisfies: x c =k0l(x)+k1l'(x'); Among them, k0 and k1 are the fusion coefficients of normalization processing, satisfying: k0+k1=1; The k0 satisfies: k0 = f k (t); Wherein, t is the exposure time of the overlapped area, t0≤t≤t1, wherein t0 is the starting exposure time of the overlapped area, and t1 is the ending exposure time of the overlapped area; During the exposure time of the overlapped area, the f k The curvature of (t) is smooth and relative to f k (t') symmetric, wherein t' is the middle exposure time of the overlapped area, t'=(t0+t1) / 2; A splicing module is used to combine the fused pixel values ​​x c , the first image and the second image are stitched together to obtain a panoramic image.

6. A panoramic camera according to claim 5, characterized in that: The photosensitive chip of the first camera and the photosensitive chip of the second camera are CMOS image sensors, the first direction is toward or away from the photosensitive chip of the second camera, and the second direction is toward or away from the photosensitive chip of the first camera.

7. A panoramic camera according to claim 6, characterized in that: The first control module is also used to control the photosensitive chip of the first camera to scan along the first direction to obtain a first image when a shooting instruction is obtained; the second control module is also used to control the photosensitive chip of the second camera to scan along the second direction to obtain a second image when the shooting instruction is obtained.

8. A panoramic camera according to claim 7, characterized in that: The camera module also includes a third camera and a fourth camera, which are arranged vertically side by side and are arranged in sequence around the first camera and the second camera, and the scanning directions of the photosensitive chips of the two adjacent cameras are opposite, and the photosensitive chip of the third camera and the photosensitive chip of the fourth camera are CMOS image sensors.

Images