Panoramic image generation method and apparatus, terminal, and storage medium

Abstract

The application discloses a panoramic image generation method and device, a terminal and a storage medium. The method comprises the following steps: determining a first spherical image and a second spherical image based on a first image, a second image and a preset spherical coordinate system; determining pixel values of all points on the first spherical image and pixel values of all points on the second spherical image based on the first spherical image and the second spherical image respectively; determining a pitch angle offset and a direction angle offset corresponding to the first spherical image and the second spherical image based on a first preset method, the pixel values of all points on the first spherical image and the pixel values of all points on the second spherical image; and performing splicing and fusion on the first spherical image and the second spherical image based on the pitch angle offset and the direction angle offset to generate a panoramic image. The application selects pixel values of overlapping areas according to the confidence degree of the overlapping areas, thereby ensuring the accuracy of the panoramic image after splicing and fusion and improving the efficiency of image splicing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, terminal, and storage medium for generating panoramic images. Background Technology

[0002] Panoramic stitching is the process of combining images with overlapping areas to create a single panoramic image. By using stitching techniques, the amount of information stored can be compressed to the maximum extent, making the information contained in the resulting panoramic image more effective.

[0003] Currently, panoramic stitching mainly involves calculating the rotation and translation matrix between pairs of images to obtain their relative poses, and then fusing the images based on these relative poses. The calculation of the rotation and translation matrix between pairs of images can be achieved in two ways: First, the two images are cylindrically projected, and then dense region matching is performed on the projected cylindrical images to determine the rotation and translation matrix; Second, SIFT and SURF feature points are extracted, and the rotation and translation matrix is ​​calculated through feature point pairs and homography transformation.

[0004] However, due to the parallax between multiple viewpoints of the camera, when images from different viewpoints are stitched together, ghosting is likely to occur in the boundary areas, reducing the accuracy of image stitching. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, terminal, and storage medium for generating panoramic images, in order to solve the problem of low image stitching accuracy in related technologies.

[0006] To achieve the above objectives, in a first aspect, this application provides a method for generating panoramic images, comprising:

[0007] Based on the first image, the second image, and a preset spherical coordinate system, the first spherical image and the second spherical image are determined, wherein the first image and the second image are obtained by the gimbal camera taking pictures of the target object from different perspectives;

[0008] Based on the first spherical image and the second spherical image, determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image, respectively.

[0009] Based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image, the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image are determined.

[0010] The first and second spherical images are stitched together based on the pitch and azimuth offsets to generate a panoramic image.

[0011] In one possible implementation, based on a first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image, the pitch angle offset and orientation angle offset corresponding to the first spherical image and the second spherical image are determined, including:

[0012] Obtain the mask values ​​of all points on the first spherical image and the mask values ​​of all points on the second spherical image respectively;

[0013] Based on the second preset method and all points on the first spherical image, determine the confidence level of all points on the first spherical image, and based on the second preset method and all points on the second spherical image, determine the confidence level of all points on the second spherical image;

[0014] Based on the first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image, the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image are determined.

[0015] In one possible implementation, based on a second preset method and all points on the first spherical image, the confidence level of all points in the first spherical image is determined, including:

[0016] Select a first target point from all points on the first spherical image and set a first confidence level for the first target point;

[0017] Calculate the distance between each of the remaining points (excluding the first target point) on the first spherical image and the first target point, and obtain the distance value corresponding to each of the remaining points;

[0018] Input the distance value corresponding to each of the remaining points into the first objective function to obtain the confidence score of each of the remaining points;

[0019] The confidence scores of all points on the first spherical image are summed up with the confidence scores of the first confidence score and each of the remaining points.

[0020] In one possible implementation, based on a first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image, the pitch angle offset and orientation angle offset corresponding to the first and second spherical images are determined, including:

[0021] Get the target window;

[0022] The target window is overlaid on the first spherical image to form a first region, and the pixel values, mask values, and confidence levels of all points in the first region are determined based on the pixel values, mask values, and confidence levels of all points in the first spherical image.

[0023] The target window is overlaid on the second spherical image to form a second region, and the pixel values, mask values, and confidence levels of all points in the second region are determined based on the pixel values, mask values, and confidence levels of all points in the second spherical image.

[0024] The pixel values, mask values, and confidence scores of all points in the first region and the pixel values, mask values, and confidence scores of all points in the second region are input into the second objective function to obtain the pitch angle offset and the azimuth angle offset.

[0025] In one possible implementation, the first spherical image and the second spherical image are stitched and fused based on pitch angle offset and azimuth angle offset to generate a panoramic image, including:

[0026] Based on the pitch angle offset and the azimuth angle offset, the first non-overlapping region corresponding to the first spherical image, the second non-overlapping region corresponding to the second spherical image, and the overlapping region of the first spherical image and the second spherical image are determined.

[0027] Based on the confidence scores of all points on the first spherical image and the confidence scores of all points on the second spherical image, the first confidence score and the second confidence score corresponding to the overlapping region are determined respectively.

[0028] The pixel values ​​of the overlapping region are determined based on the magnitudes of the first and second confidence levels.

[0029] A panoramic image is formed by the first non-overlapping region, the second non-overlapping region, the overlapping region, the pixel values ​​of the first non-overlapping region, the pixel values ​​of the second non-overlapping region, and the pixel values ​​of the overlapping region.

[0030] In one possible implementation, determining the first spherical image and the second spherical image based on the first image, the second image, and a preset spherical coordinate system includes:

[0031] Establish a spherical coordinate system with the origin of the camera coordinate system as the center and a preset radius to obtain the preset spherical coordinate system;

[0032] The first image and the second image are projected onto a preset spherical coordinate system to obtain a first spherical image and a second spherical image.

[0033] In one possible implementation, determining the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image, based on the first spherical image and the second spherical image respectively, includes:

[0034] Obtain all points on the first spherical image and all points on the second spherical image;

[0035] The pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image are determined based on all points in the first spherical image and all points in the second spherical image, respectively.

[0036] In one possible implementation, obtaining all points on the first spherical image and all points on the second spherical image includes:

[0037] Obtain the azimuth and pitch angles corresponding to any point in the camera coordinate system;

[0038] The azimuth and elevation angles are divided into equal angles, and based on the divided azimuth and elevation angles and the preset radius, all points on the sphere of the preset spherical coordinate system are determined.

[0039] Select points from all points that fall on the first spherical image and the second spherical image respectively, to obtain all points on the first spherical image and all points on the second spherical image.

[0040] In one possible implementation, determining the pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image based on all points in the first spherical image and the second spherical image respectively includes:

[0041] The spherical coordinates of all points on the first spherical image and the spherical coordinates of all points on the second spherical image are transformed using a preset function to obtain the image coordinates of all points on the first spherical image and the image coordinates of all points on the second spherical image.

[0042] The image coordinates of each point in the first spherical image are identified to obtain a first identification result, and the pixel values ​​of all points in the first spherical image are determined based on the first identification result.

[0043] The image coordinates of each point in the second spherical image are identified to obtain a second identification result. Based on the second identification result, the pixel values ​​of all points in the second spherical image are determined.

[0044] In one possible implementation, the image coordinates of each point among all points on the first spherical image are identified to obtain a first identification result, and based on the first identification result, the pixel values ​​of all points on the first spherical image are determined, including:

[0045] For each point among all points on the first spherical image, if the image coordinates are on the first image and are integers, obtain the pixel value at the image coordinates.

[0046] If the image coordinates are on the first image and are not integers, the image coordinates are processed using a preset interpolation method, and the pixel value at the processed image coordinates is obtained.

[0047] The pixel values ​​at the image coordinates and the pixel values ​​at the processed image coordinates are summarized to obtain the pixel values ​​of all points on the first spherical image.

[0048] Secondly, embodiments of the present invention provide a panoramic image generation apparatus, comprising:

[0049] The spherical image determination module is used to determine a first spherical image and a second spherical image based on a first image, a second image, and a preset spherical coordinate system, wherein the first image and the second image are obtained by the gimbal camera taking pictures of the target object from different perspectives;

[0050] The pixel value determination module is used to determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image, respectively, based on the first spherical image and the second spherical image.

[0051] The offset determination module is used to determine the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image.

[0052] The panoramic image generation module is used to stitch and fuse the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image.

[0053] Thirdly, embodiments of the present invention provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above panoramic image generation methods.

[0054] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the panoramic image generation methods described above.

[0055] This invention provides a method, apparatus, terminal, and storage medium for generating panoramic images, comprising: determining a first spherical image and a second spherical image based on a first image, a second image, and a preset spherical coordinate system; then determining the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image based on the first spherical image and the second spherical image, respectively; further determining the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on a first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image; and finally stitching and fusing the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image. This invention projects the first and second images onto a preset spherical coordinate system, thereby performing image registration on the spherical surface. This ensures that the registration is not affected or interfered with by parallax, improving the accuracy of image stitching. In addition, the overlapping area of ​​the two spherical images is determined by the pitch angle offset and the azimuth angle offset, and the pixel value of the overlapping area is selected by the confidence level of the overlapping area, ensuring the accuracy of the stitched and fused panoramic image and improving the efficiency of image stitching. Attached Figure Description

[0056] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0057] Figure 1 This is a flowchart illustrating the implementation of a panoramic image generation method provided in an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the azimuth and pitch angles in the camera coordinate system provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the projection of the camera coordinate system onto a planar image according to an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the splicing and fusion of planar images provided in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the structure of a panoramic image generation device provided in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the terminal provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0065] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0066] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0067] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "and / or B" can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0068] It should be understood that in this invention, "B corresponding to A", "B corresponding to A", "A and B correspond", or "B and A correspond" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0069] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0070] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0071] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0072] In one embodiment, such as Figure 1 As shown, a method for generating panoramic images is provided, including the following steps:

[0073] Step S101: Determine the first spherical image and the second spherical image based on the first image, the second image, and the preset spherical coordinate system.

[0074] The first image and the second image are obtained by taking pictures of the target object from different perspectives. Because the camera is at different perspectives, there is a common area (i.e., an overlapping area) in the first image and the second image. Therefore, it is necessary to project the first image and the second image onto a sphere first, and then determine the pixel value of the overlapping area based on the generated first spherical image and the second spherical image to finally generate a panoramic image, which can improve the accuracy of the panoramic image.

[0075] To project the first and second images onto the sphere, a spherical coordinate system needs to be established with the origin of the camera coordinate system as the center and a preset radius. Then, the first and second images are projected onto this preset spherical coordinate system to obtain the first and second spherical images respectively. The preset radius can be arbitrarily specified, as long as it is slightly larger than the focal length.

[0076] After establishing a preset spherical coordinate system, the first and second images are projected into the spherical coordinate system respectively, and then the spherical images are matched. This can eliminate the parallax problem caused by the viewing angle and improve the accuracy of image stitching.

[0077] Step S102: Based on the first spherical image and the second spherical image, determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image, respectively.

[0078] Once a preset spherical coordinate system is established, the coordinates of points on that system can be determined as follows: First, obtain the azimuth and pitch angles corresponding to any point in the camera coordinate system. Then, divide the azimuth and pitch angles into equal angles. Based on these divisions and the preset radius, determine all points on the sphere of the preset spherical coordinate system, and their coordinates.

[0079] In one embodiment, combined with Figure 2 The steps for determining the coordinates of a point in a preset spherical coordinate system are described below:

[0080] like Figure 2 As shown, O-XC-YC-ZC is the camera coordinate system. There is a three-dimensional point M(xc, yc, zc) in the camera coordinate system. The intersection of the perpendicular line from the three-dimensional point M to the XOY plane and the XOY plane is defined as N. The angle between ON and the X-axis is defined as the azimuth angle theta, and the angle between NOM is defined as the pitch angle phi.

[0081] The preset spherical coordinate system in this invention is centered on the origin O of the aforementioned camera coordinate system (the center of the sphere) and has a radius d. The azimuth angle theta and pitch angle phi are divided into M and N equal parts (generally M = N), respectively, denoted as θ0, θ1, ... θ i , ..., θ M , Where 0 <= i <= M, 0 <= j <= N. Through the above operations, a preset spherical coordinate system can be established, where the coordinates of points on the sphere of the spherical coordinate system are...

[0082] Since the first and second images are transformed to a preset spherical coordinate system through projection, once all points and their coordinates on the preset spherical coordinate system are determined, it is only necessary to select the points that fall on the first and second spherical images respectively to obtain all points and their coordinates on the first and second spherical images.

[0083] An image is a two-dimensional matrix, where each point coordinate corresponds to a pixel value. Since there's a one-to-one correspondence between point coordinates on a sphere (spherical coordinates) and point coordinates in an image (image coordinates), the pixel value corresponding to a point coordinate in the image is equal to the pixel value of that point on the sphere. To obtain the pixel value corresponding to a point coordinate in the spherical image, the point coordinates on the sphere need to be converted to the corresponding point coordinates in the image. Then, the pixel value of the point coordinate on the sphere can be directly obtained from the pixel value at the corresponding point coordinate in the image.

[0084] Based on the above principles, the process of determining the pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image, respectively, based on all points in the first spherical image and the second spherical image, is described in detail below:

[0085] (1) Use a preset function to transform the spherical coordinates of all points on the first spherical image and the spherical coordinates of all points on the second spherical image to obtain the image coordinates of all points on the first spherical image and the image coordinates of all points on the second spherical image.

[0086] Figure 3 The camera coordinate system is O-XC-YC-ZC, where O1-XY is the image coordinate system, and the distance between O-O1 is the focal length. Once we know the camera intrinsic parameter matrix K (provided by the manufacturer or pre-calibrated) and the point M(xc, yc, zc) in the camera coordinate system, we can use the following formula to obtain the projected coordinates (u, v) of the 3D point M(xc, yc, zc) on the image.

[0087]

[0088]

[0089] In the first formula above, the first matrix to the right of the equals sign is the intrinsic parameter matrix K, and the second matrix to the right of the equals sign is the transformation matrix from the world coordinate system to the camera coordinate system.

[0090] The point coordinates corresponding to the spherical image have been determined using the previous embodiment. For example, let the coordinates of a point on the first spherical image be... First, set the coordinates of the points on the first sphere. The coordinates are converted to three-dimensional Cartesian coordinates (xc, yc, zc), and then transformed back to the image coordinate system using the formula described above, thus obtaining the points on the first spherical image. The image coordinates are (u, v). The coordinate transformation for the second spherical image is similar to that for the first spherical image, and will not be described again here.

[0091] (2) After the conversion from spherical coordinates to image coordinates is realized, the image coordinates of each point need to be identified in order to determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image.

[0092] For the first spherical image, the image coordinates of each point in the first spherical image need to be identified first to obtain a first identification result. Based on the first identification result, the pixel values ​​of all points in the first spherical image are determined. Specifically, for the image coordinates of each point in the first spherical image, if the image coordinates are on the first image and are integers, the pixel value at the image coordinates is obtained. If the image coordinates are on the first image and are not integers, the image coordinates are processed using a preset interpolation method, and the pixel value at the processed image coordinates is obtained. Finally, the pixel values ​​at the original image coordinates and the pixel values ​​at the processed image coordinates are summarized to obtain the pixel values ​​of all points in the first spherical image.

[0093] For the second spherical image, the image coordinates of each point in the second spherical image must first be identified to obtain a second identification result. Based on the second identification result, the pixel values ​​of all points in the second spherical image are determined. Specifically, for the image coordinates of each point in the second spherical image, if the image coordinates are on the second image and are integers, the pixel value at the image coordinates is obtained. If the image coordinates are on the second image and are not integers, the image coordinates are processed using a preset interpolation method, and the pixel value at the processed image coordinates is obtained. Finally, the pixel values ​​at the original image coordinates and the pixel values ​​at the processed image coordinates are summarized to obtain the pixel values ​​of all points in the second spherical image.

[0094] Step S103: Based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image, determine the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image.

[0095] To determine the pitch and azimuth offsets, it is necessary to first obtain the mask values ​​of all points on the first spherical image and the mask values ​​of all points on the second spherical image, respectively. Specifically, let the area of ​​the rectangular window P be M*N. This rectangular window is then applied to the first and second spherical images to form a first coverage area and a second coverage area. Each coverage area contains multiple points. It is determined whether the coordinates of each point are within the image's field of view. If they are, the mask value corresponding to this point is set to 1; otherwise, it is set to 0. In this way, the mask values ​​of all points on the first and second spherical images can be obtained.

[0096] Next, based on the second preset method and all points on the first spherical image, it is necessary to determine the confidence level of all points on the first spherical image, and based on the second preset method and all points on the second spherical image, determine the confidence level of all points on the second spherical image. Specifically, for the first spherical image, a first target point is selected from all points on the first spherical image, and a first confidence level is set for the first target point. Then, the distance between each of the remaining points (excluding the first target point) and the first target point is calculated to obtain the distance value corresponding to each of the remaining points. Then, the distance value corresponding to each of the remaining points is input into the first objective function to obtain the confidence level of each of the remaining points. Finally, the first confidence level and the confidence level of each of the remaining points are summarized to obtain the confidence level of all points on the first spherical image.

[0097] Further, taking the first spherical image as an example, a reference point (i.e., the first target point) is first set on the first spherical image. This reference point can be the center point of the first spherical image or the point where the principal optical axis passes through the first spherical image. When the reference point is the point where the principal optical axis passes through the first spherical image, let the confidence level of this point (d, theta0, phi0) be 1. Then, the confidence level of the remaining points can be determined according to their spherical distance from the point (d, theta0, phi0), such as decreasing the confidence level sequentially according to the distance from small to large. When the reference point is the center point (u0, v0) of the first spherical image, a function of confidence level and distance (i.e., the first target function) is pre-set. For example, if the confidence level of the center point (u0, v0) is 1, the distance between the remaining points and the center point (u0, v0) can be calculated first, and then the distance of each point can be input into the first target function to obtain the confidence level of each of the remaining points. The method for determining the confidence level of all points on the second spherical image and the type of the first spherical image will not be elaborated here.

[0098] Once the pixel values, mask values, and confidence levels of all points on the first spherical image, as well as the pixel values, mask values, and confidence levels of all points on the second spherical image, are determined, the pitch angle offset and orientation angle offset corresponding to the first and second spherical images can be determined based on the first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image. The first preset method can be a region matching method.

[0099] The following uses the SAD (Self-Action Algorithm) region matching method as an example to illustrate the steps for determining the pitch and azimuth offsets:

[0100] (1) Obtain the target window;

[0101] (2) The target window is covered on the first spherical image to form a first region, and the pixel value, mask value and confidence level of all points in the first region are determined based on the pixel value, mask value and confidence level of all points in the first spherical image;

[0102] (3) Cover the target window on the second spherical image to form a second region, and determine the pixel value, mask value and confidence level of all points in the second region based on the pixel value, mask value and confidence level of all points in the second spherical image;

[0103] (4) Input the pixel values, mask values ​​and confidence scores of all points in the first region and the pixel values, mask values ​​and confidence scores of all points in the second region into the second objective function to obtain the pitch angle offset and the azimuth angle offset;

[0104] (5) Move the target window on the second spherical image and repeat the process of (3)-(4) (there is a preset search range here, and it will exit if it exceeds this range);

[0105] (6) After each movement, the pixel values, mask values ​​and confidence scores of the two target windows are input into the target function corresponding to SAD. When the target function value corresponding to SAD is the minimum, the corresponding pitch angle offset and azimuth angle offset are output.

[0106] Specifically, the objective function corresponding to SAD is as follows:

[0107]

[0108] Where A and B are preset search area ranges, A is the first search range corresponding to the first spherical image, B is the second search range corresponding to the second spherical image, P0(i,j) is the pixel value of the point on the first spherical image, P1(i+a,j+b) is the pixel value of the point on the second spherical image, C0(i,j) is the mask value of the point on the first spherical image, C1(i+a,j+b) is the mask value of the point on the second spherical image, B(i,j) is the confidence level of the point on the first spherical image, B(i+a,j+b) is the confidence level of the point on the second spherical image, a is the pitch angle offset, and b is the azimuth angle offset.

[0109] Step S104: Based on the pitch angle offset and azimuth angle offset, the first spherical image and the second spherical image are stitched and fused to generate a panoramic image.

[0110] Specifically, based on the pitch and azimuth offsets, the first non-overlapping region corresponding to the first spherical image, the second non-overlapping region corresponding to the second spherical image, and the overlapping region of the first and second spherical images are first determined. Then, based on the confidence levels of all points on the first and second spherical images, the first and second confidence levels corresponding to the overlapping region are determined. Based on the magnitudes of the first and second confidence levels, the pixel values ​​of the overlapping region are determined. Finally, the panoramic image is constructed from the first non-overlapping region, the second non-overlapping region, the overlapping region, the pixel values ​​of the first non-overlapping region, the second non-overlapping region, and the overlapping region.

[0111] Furthermore, with Figure 4 The steps for generating a panoramic image are explained below using an example:

[0112] In a plane, suppose we first take a picture of image 1 with a camera, then move the camera slightly to the right and take another picture of image 2. We then have two overlapping images, image 1 and image 2, from two different perspectives. The purpose of the first two steps is to calculate the translation between the two images, which is a shift of 5 pixels to the right.

[0113] Next, the two images need to be stitched together into a single large image. The parts where the perspectives don't overlap are copied directly. For the parts where the perspectives overlap (the last three columns of image 1 and the first three columns of image 2), ideally, the corresponding pixel values ​​in these three columns should be identical. For example, the content of column 6 in image 1 should be the same as the content of column 1 in image 2. However, since this isn't ideal, the pixel values ​​at corresponding positions are different, represented by different colors. Since higher confidence levels correspond to smaller pixel value errors, pixel values ​​with higher confidence levels are selected to fill the overlapping parts. For example, for column 6 of the merged image, should we choose column 6 of image 1 or column 1 of image 2? Pixel values ​​closer to the optical center (which can be simply understood as the image center) have higher confidence levels. Therefore, the distance from the center of image 1 to the 6th column is less than the distance from the optical center of image 2 to the 1st column. So the confidence level of the 6th column of image 1 is greater than the confidence level of the 1st column of image 2. Therefore, the green value of the 6th column of image 1 is filled into the corresponding overlapping part. The same applies to the other columns.

[0114] Therefore, by converting the aforementioned plane into a sphere, the pitch and azimuth offsets (i.e., translations in the plane) obtained in the previous embodiment are used to determine the first non-overlapping region corresponding to the first spherical image, the second non-overlapping region corresponding to the second spherical image, and the overlapping region of the first and second spherical images. Since the overlapping region corresponds to the first confidence level of the first spherical image and the second confidence level of the second spherical image, and since pixels at positions with higher confidence levels are more stable, the first and second confidence levels are compared, and the pixel values ​​corresponding to positions with higher confidence levels are filled into the overlapping region, thus determining the pixel values ​​of the overlapping region.

[0115] Then, the first non-overlapping region, the second non-overlapping region, and the overlapping region are stitched together. Then, the pixel values ​​of the first non-overlapping region, the second non-overlapping region, and the overlapping region are filled into the corresponding regions to form a panoramic image.

[0116] This invention provides a method for generating panoramic images, comprising: determining a first spherical image and a second spherical image based on a first image, a second image, and a preset spherical coordinate system; then determining the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image, respectively; further determining the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on a first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image; finally, stitching and fusing the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image. This invention projects the first image and the second image onto a preset spherical coordinate system, thereby performing image registration on the spherical surface. This prevents the registration from being affected or interfered with by parallax, improving the accuracy of image stitching. Furthermore, it determines the overlapping area of ​​the two spherical images through the pitch angle offset and the azimuth angle offset, and selects the pixel values ​​of the overlapping area based on the confidence level of the overlapping area, ensuring the accuracy of the stitched and fused panoramic image and improving the efficiency of image stitching.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0118] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0119] Figure 5The diagram illustrates a panoramic image generation device according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment are shown. The panoramic image generation device includes a spherical image determination module 51, a pixel value determination module 52, an offset determination module 53, and a panoramic image generation module 54, as detailed below:

[0120] The spherical image determination module 51 is used to determine a first spherical image and a second spherical image based on a first image, a second image and a preset spherical coordinate system, wherein the first image and the second image are obtained by the gimbal camera taking pictures of the target object from different perspectives;

[0121] The pixel value determination module 52 is used to determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image based on the first spherical image and the second spherical image, respectively.

[0122] The offset determination module 53 is used to determine the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image.

[0123] The panoramic image generation module 54 is used to stitch and fuse the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image.

[0124] In one possible implementation, the offset determination module 53 includes:

[0125] The mask value acquisition submodule is used to acquire the mask values ​​of all points on the first spherical image and the mask values ​​of all points on the second spherical image, respectively.

[0126] The confidence determination submodule is used to determine the confidence of all points on the first spherical image based on the second preset method and all points on the first spherical image, and to determine the confidence of all points on the second spherical image based on the second preset method and all points on the second spherical image.

[0127] The offset determination submodule is used to determine the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values, mask values ​​and confidence levels of all points on the first spherical image, and the pixel values, mask values ​​and confidence levels of all points on the second spherical image.

[0128] In one possible implementation, the confidence level determination submodule includes:

[0129] The first confidence setting unit is used to select a first target point from all points on the first spherical image and set a first confidence level for the first target point;

[0130] The distance calculation unit is used to calculate the distance between each of the remaining points (excluding the first target point) on the first spherical image and the first target point, and to obtain the distance value corresponding to each of the remaining points.

[0131] The second confidence determination unit is used to input the distance value corresponding to each of the remaining points into the first objective function to obtain the confidence of each of the remaining points;

[0132] The third confidence determination unit is used to summarize the first confidence and the confidence of each point in the remaining points to obtain the confidence of all points on the first spherical image.

[0133] In one possible implementation, the offset determination submodule includes:

[0134] The window acquisition unit is used to acquire the target window;

[0135] The first region determination unit is used to cover the target window onto the first spherical image to form a first region, and to determine the pixel values, mask values, and confidence levels of all points in the first region based on the pixel values, mask values, and confidence levels of all points in the first spherical image.

[0136] The second region determination unit is used to cover the target window onto the second spherical image to form a second region, and to determine the pixel values, mask values, and confidence levels of all points in the second region based on the pixel values, mask values, and confidence levels of all points in the second spherical image.

[0137] The offset determination unit is used to input the pixel values, mask values, and confidence scores of all points in the first region and the pixel values, mask values, and confidence scores of all points in the second region into the second objective function to obtain the pitch angle offset and the azimuth angle offset.

[0138] In one possible implementation, the panoramic image generation module 54 includes:

[0139] The sub-region determination submodule is used to determine the first non-overlapping region corresponding to the first spherical image, the second non-overlapping region corresponding to the second spherical image, and the overlapping region of the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset.

[0140] The confidence calculation submodule is used to determine the first confidence level and the second confidence level corresponding to the overlapping region based on the confidence levels of all points on the first spherical image and the second spherical image, respectively.

[0141] The pixel value selection submodule is used to determine the pixel values ​​of the overlapping region based on the magnitude of the first confidence level and the second confidence level.

[0142] The panoramic image generation submodule is used to construct a panoramic image from a first non-overlapping region, a second non-overlapping region, an overlapping region, the pixel values ​​of the first non-overlapping region, the pixel values ​​of the second non-overlapping region, and the pixel values ​​of the overlapping region.

[0143] In one possible implementation, the spherical image determination module 51 includes:

[0144] The coordinate system establishment submodule is used to establish a spherical coordinate system with the origin of the camera coordinate system as the center and a preset radius, and obtain the preset spherical coordinate system.

[0145] The projection submodule is used to project the first image and the second image onto a preset spherical coordinate system to obtain the first spherical image and the second spherical image.

[0146] In one possible implementation, the pixel value determination module 52 includes:

[0147] The point acquisition submodule is used to acquire all points on the first spherical image and all points on the second spherical image;

[0148] The pixel value determination submodule is used to determine the pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image based on all points in the first spherical image and all points in the second spherical image, respectively.

[0149] In one possible implementation, the point acquisition submodule includes:

[0150] Angle acquisition unit is used to acquire the azimuth and pitch angles corresponding to any point in the camera coordinate system;

[0151] The spherical coordinate point determination unit is used to divide the azimuth angle and elevation angle into equal angles, and based on the divided azimuth angle and elevation angle, as well as the preset radius, determine all points on the spherical surface of the preset spherical coordinate system.

[0152] The spherical point determination unit is used to select points that fall on the first spherical image and the second spherical image respectively from all points, so as to obtain all points on the first spherical image and all points on the second spherical image.

[0153] In one possible implementation, the pixel value determination submodule includes:

[0154] The coordinate transformation unit is used to perform coordinate transformation on the spherical coordinates of all points on the first spherical image and the spherical coordinates of all points on the second spherical image using a preset function, respectively, to obtain the image coordinates of all points on the first spherical image and the image coordinates of all points on the second spherical image;

[0155] The first recognition unit is used to recognize the image coordinates of each point among all points on the first spherical image, obtain a first recognition result, and determine the pixel values ​​of all points on the first spherical image based on the first recognition result.

[0156] The second recognition unit is used to recognize the image coordinates of each point among all points on the second spherical image, obtain a second recognition result, and determine the pixel values ​​of all points on the second spherical image based on the second recognition result.

[0157] In one possible implementation, the first identification unit includes:

[0158] The first pixel value acquisition subunit is used to acquire the pixel value at each point in the first spherical image if the image coordinates are on the first image and are integers.

[0159] The second pixel value acquisition subunit is used to process the image coordinates using a preset interpolation method if the image coordinates are on the first image and are not integers, and to obtain the pixel value at the processed image coordinates.

[0160] The first spherical pixel determination subunit is used to summarize the pixel values ​​at the image coordinates and the pixel values ​​at the processed image coordinates to obtain the pixel values ​​of all points on the first spherical image.

[0161] Figure 6 This is a schematic diagram of a terminal provided in an embodiment of the present invention. Figure 6 As shown, the terminal 6 in this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program 63, it implements the steps in the various panoramic image generation method embodiments described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when processor 61 executes computer program 63, it implements the functions of each module / unit in the above-described embodiments of the panoramic image generation apparatus, for example... Figure 5 The functions of modules / units 51 to 54 shown.

[0162] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the panoramic image generation method provided in the various embodiments described above.

[0163] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0164] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the execution of the executable instructions by the at least one processor causes the device to implement the panoramic image generation method provided in the various embodiments described above.

[0165] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0166] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating panoramic images, characterized in that, include: Based on the first image, the second image, and a preset spherical coordinate system, a first spherical image and a second spherical image are determined, wherein the first image and the second image are obtained by the gimbal camera capturing the target object from different perspectives; Based on the first spherical image and the second spherical image respectively, determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image; Based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image, the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image are determined; The first spherical image and the second spherical image are stitched and fused together based on the pitch angle offset and the azimuth angle offset to generate a panoramic image; The step of determining the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image includes: Obtain the mask values ​​of all points on the first spherical image and the mask values ​​of all points on the second spherical image, respectively; Based on the second preset method and all points on the first spherical image, determine the confidence level of all points on the first spherical image, and based on the second preset method and all points on the second spherical image, determine the confidence level of all points on the second spherical image. Based on the first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image, the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image are determined.

2. The method for generating panoramic images as described in claim 1, characterized in that, The step of determining the confidence level of all points in the first spherical image based on the second preset method and all points in the first spherical image includes: Select a first target point from all points on the first spherical image, and set a first confidence level for the first target point; Calculate the distance between each of the remaining points (excluding the first target point) on the first spherical image and the first target point, and obtain the distance value corresponding to each of the remaining points; Input the distance value corresponding to each of the remaining points into the first objective function to obtain the confidence score of each of the remaining points; The confidence scores of all points on the first spherical image are summed up with the confidence scores of the first confidence score and each of the remaining points.

3. The method for generating panoramic images as described in claim 2, characterized in that, The step of determining the pitch angle offset and orientation angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image, includes: Get the target window; The target window is overlaid on the first spherical image to form a first region, and the pixel values, mask values, and confidence levels of all points in the first region are determined based on the pixel values, mask values, and confidence levels of all points in the first spherical image. The target window is overlaid on the second spherical image to form a second region, and the pixel values, mask values, and confidence levels of all points in the second region are determined based on the pixel values, mask values, and confidence levels of all points in the second spherical image. The pixel values, mask values, and confidence scores of all points in the first region and the pixel values, mask values, and confidence scores of all points in the second region are input into the second objective function to obtain the pitch angle offset and the azimuth angle offset.

4. The method for generating panoramic images as described in claim 3, characterized in that, The step of stitching and fusing the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image includes: Based on the pitch angle offset and the azimuth angle offset, the first non-overlapping region corresponding to the first spherical image, the second non-overlapping region corresponding to the second spherical image, and the overlapping region of the first spherical image and the second spherical image are determined. Based on the confidence scores of all points on the first spherical image and the confidence scores of all points on the second spherical image, the first confidence score and the second confidence score corresponding to the overlapping region are determined respectively. The pixel values ​​of the overlapping region are determined based on the magnitudes of the first confidence level and the second confidence level. The panoramic image is composed of the first non-overlapping region, the second non-overlapping region, the overlapping region, the pixel values ​​of the first non-overlapping region, the pixel values ​​of the second non-overlapping region, and the pixel values ​​of the overlapping region.

5. The method for generating panoramic images as described in any one of claims 1-4, characterized in that, The determination of the first spherical image and the second spherical image based on the first image, the second image, and a preset spherical coordinate system includes: A spherical coordinate system is established with the origin of the camera coordinate system as the center of the sphere and a preset radius, thus obtaining the preset spherical coordinate system; The first image and the second image are projected onto the preset spherical coordinate system respectively to obtain the first spherical image and the second spherical image.

6. The method for generating panoramic images as described in claim 5, characterized in that, The step of determining the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image based on the first spherical image and the second spherical image respectively includes: Obtain all points on the first spherical image and all points on the second spherical image; The pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image are determined based on all points in the first spherical image and all points in the second spherical image, respectively.

7. The method for generating panoramic images as described in claim 6, characterized in that, The step of obtaining all points on the first spherical image and all points on the second spherical image includes: Obtain the azimuth and elevation angles corresponding to any point in the camera coordinate system; The azimuth and elevation angles are divided into equal angles, and based on the divided azimuth and elevation angles and the preset radius, all points on the sphere of the preset spherical coordinate system are determined. Select points from all the points that fall on the first spherical image and the second spherical image respectively to obtain all points on the first spherical image and all points on the second spherical image.

8. The method for generating panoramic images as described in claim 7, characterized in that, The step of determining the pixel values ​​of all points in the first spherical image and the pixel values ​​of all points in the second spherical image based on all points in the first spherical image and the second spherical image, respectively, includes: The spherical coordinates of all points on the first spherical image and the spherical coordinates of all points on the second spherical image are transformed using a preset function to obtain the image coordinates of all points on the first spherical image and the image coordinates of all points on the second spherical image. The image coordinates of each point in the first spherical image are identified to obtain a first identification result, and the pixel values ​​of all points in the first spherical image are determined based on the first identification result. The image coordinates of each point in the second spherical image are identified to obtain a second identification result, and the pixel values ​​of all points in the second spherical image are determined based on the second identification result.

9. The method for generating panoramic images as described in claim 8, characterized in that, The step of identifying the image coordinates of each point among all points on the first spherical image to obtain a first identification result, and determining the pixel values ​​of all points on the first spherical image based on the first identification result, includes: For each point among all points on the first spherical image, if the image coordinates are on the first image and are integers, obtain the pixel value at the image coordinates. If the image coordinates are on the first image and are not integers, the image coordinates are processed using a preset interpolation method, and the pixel value at the processed image coordinates is obtained. The pixel values ​​at the image coordinates and the pixel values ​​at the processed image coordinates are summarized to obtain the pixel values ​​of all points on the first spherical image.

10. A panoramic image generation apparatus, characterized in that, include: The spherical image determination module is used to determine a first spherical image and a second spherical image based on a first image, a second image, and a preset spherical coordinate system, wherein the first image and the second image are obtained by the gimbal camera capturing the target object from different perspectives; The pixel value determination module is used to determine the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image based on the first spherical image and the second spherical image, respectively. The offset determination module is used to determine the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values ​​of all points on the first spherical image and the pixel values ​​of all points on the second spherical image; A panoramic image generation module is used to stitch and fuse the first spherical image and the second spherical image based on the pitch angle offset and the azimuth angle offset to generate a panoramic image; The step of determining the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image based on the first preset method, the pixel values ​​of all points on the first spherical image, and the pixel values ​​of all points on the second spherical image includes: Obtain the mask values ​​of all points on the first spherical image and the mask values ​​of all points on the second spherical image, respectively; Based on the second preset method and all points on the first spherical image, determine the confidence level of all points on the first spherical image, and based on the second preset method and all points on the second spherical image, determine the confidence level of all points on the second spherical image. Based on the first preset method, the pixel values, mask values, and confidence levels of all points on the first spherical image, and the pixel values, mask values, and confidence levels of all points on the second spherical image, the pitch angle offset and azimuth angle offset corresponding to the first spherical image and the second spherical image are determined.

11. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the panoramic image generation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the panoramic image generation method as described in any one of claims 1 to 9.

Images