Panoramic image generation method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a panoramic image generation method, device, equipment and storage medium, comprising: acquiring a plurality of scene images of a target scene and a depth value corresponding to each pixel point in each scene image. The plurality of panoramic images are obtained by a camera shooting at a plurality of different angles around the rotation axis axis center, and the depth value is the distance of the corresponding object point of each pixel point on the plurality of scene images in the three-dimensional space relative to the rotation axis axis center. Taking the rotation axis axis center as the projection center and the depth value corresponding to each pixel point as the radius, a target projection sphere corresponding to each pixel point is determined; according to the internal and external parameters of the camera calibrated in advance, the target longitude and latitude coordinates of each pixel point projected onto the corresponding target projection sphere are determined. The target longitude and latitude coordinates are converted into two-dimensional plane coordinates in the panoramic coordinate system to obtain a plurality of target planar development images corresponding to the plurality of scene images. The plurality of target planar development images are fused to obtain a target panoramic image of the target scene.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image generation method, apparatus, device and storage medium. Background Art

[0002] When generating panoramic images, the panoramic imaging model is usually simplified to an ideal model: a projection center and an ideal projection sphere. The panoramic imaging model expresses the collinear relationship between the object point, the image point on the sphere, and the projection center (i.e., the center of the ideal sphere) through a collinear equation. In a specific application process, in order to convert multiple two-dimensional images of a target scene obtained by a shooting device into a panoramic image, it is usually necessary to first determine the spherical coordinates corresponding to the pixel coordinates of each pixel in the multiple two-dimensional images on the ideal projection sphere based on the internal and external parameters of the shooting device corresponding to each two-dimensional image. Then, the spherical coordinates are mapped to panoramic coordinates based on the collinear equation to expand the image, and finally, the expanded images are spliced ​​to obtain a panoramic image.

[0003] However, in actual applications, whether a camera rotates around a rotation axis to capture a target scene, or multiple cameras are arranged around a rotation axis to capture the target scene at fixed angles, there will be a problem where the camera node and the axis of the rotation axis do not coincide. In other words, there are multiple projection centers, and the projection center C corresponding to the camera node and the projection center S corresponding to the axis of the rotation axis do not coincide. For any image point u on the projection sphere with the projection center S as the sphere center, the misalignment of the projection center C and the projection center S will result in an error between the ideal object point P determined from the projection center S through the image point u and the actual object point P' determined from the projection center C through the image point u. This error can cause problems such as cracks and misalignments in the stitched panoramic image.

[0004] Adjusting the camera node to coincide with the axis of rotation requires repeated testing, which is difficult and requires high engineering error, making it inconvenient for users to operate. Therefore, a convenient and fast method for generating panoramic images with high image stitching quality is urgently needed. Summary of the Invention

[0005] Embodiments of the present invention provide a panoramic image generation method, apparatus, device, and storage medium for generating high-quality panoramic images.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a panoramic image, the method comprising:

[0007] Acquire multiple scene images of a target scene and a depth value corresponding to each pixel in each scene image, wherein the multiple panoramic images are obtained by capturing the camera at multiple different angles around a rotation axis, and the depth value is the distance between the object point corresponding to each pixel in three-dimensional space and the rotation axis when capturing the multiple scene images;

[0008] Taking the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius, determining the target projection sphere corresponding to each pixel point;

[0009] Determine the target latitude and longitude coordinates of each pixel point projected onto the corresponding target projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera;

[0010] Converting the target latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane expanded images corresponding to the multiple scene images;

[0011] The multiple target plane expanded images are fused to obtain a target panoramic image of the target scene.

[0012] In a second aspect, an embodiment of the present invention provides a panoramic image generation device, the device comprising:

[0013] an acquisition module, configured to acquire multiple scene images of a target scene and a depth value corresponding to each pixel point in each scene image, wherein the multiple panoramic images are obtained by capturing the camera at multiple different angles around a rotation axis, and the depth value is the distance between the object point corresponding to each pixel point in three-dimensional space and the rotation axis when capturing the multiple scene images;

[0014] The processing module is used to determine the target projection sphere corresponding to each pixel point with the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius; determine the target longitude and latitude coordinates of each pixel point projected onto the corresponding target projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera; convert the target longitude and latitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane expanded images corresponding to the multiple scene images; and fuse the multiple target plane expanded images to obtain a target panoramic image of the target scene.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the panoramic image generation method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the panoramic image generation method described in the first aspect.

[0017] In the solution provided by an embodiment of the present invention, first, multiple scene images of the target scene and the depth values ​​corresponding to each pixel point in each scene image are obtained. The multiple panoramic images are obtained by shooting the camera at multiple different angles around the axis of the rotation axis, and the depth value is the distance between the object point corresponding to each pixel point in three-dimensional space and the axis of the rotation axis when shooting the multiple scene images. Then, with the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius, the target projection sphere corresponding to each pixel point is determined, and based on the pre-calibrated intrinsic and extrinsic parameters of the camera, the target latitude and longitude coordinates of each pixel point projected onto the corresponding target projection sphere are determined. Thereafter, the target latitude and longitude coordinates corresponding to each pixel point are converted into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane unfolded images corresponding to the multiple scene images. Finally, the multiple target plane unfolded images are fused to obtain a target panoramic image of the target scene.

[0018] In this solution, the point on the sphere represented by the latitude and longitude coordinates of any pixel on its target projection sphere is its image point. Since the radius corresponding to the target projection sphere is the depth value corresponding to the pixel point, the image point is also the object point corresponding to the pixel point. Based on this, regardless of whether the projection center C corresponding to the camera node coincides with the projection center S corresponding to the axis of rotation, the object point P determined from the projection center S through the image point u is the same as the object point P' determined from the projection center C through the image point u, because at this time the object point P and the object point P' are both the image point u. This solution determines the projection sphere with the depth value corresponding to the pixel point as the radius, avoiding the projection error caused by the misalignment of the camera node and the axis of rotation, and can quickly obtain high-quality panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a panoramic image generation system provided by an embodiment of the present invention;

[0021] Figure 2 A flowchart of a panoramic image generation method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a scene of a panoramic image generation process provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a panoramic imaging model provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of another panoramic imaging model provided by an embodiment of the present invention;

[0025] Figure 6 A flowchart of an image fusion method provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of an image overlapping area provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of a splicing seam provided by an embodiment of the present invention;

[0028] Figure 9 A schematic structural diagram of a panoramic image generation device provided by an embodiment of the present invention;

[0029] Figure 10 For Figure 9 A schematic structural diagram of an electronic device corresponding to the panoramic image generation device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one.

[0032] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0033] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0034] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0035] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0036] Before introducing the panoramic image generation method provided by the embodiment of the present invention, the following concepts are first explained.

[0037] The camera nodal point refers to the optical center of a camera or video camera lens. Because light passing through this nodal point doesn't refract, and objects (near or far) don't shift when the lens rotates, the nodal point is also called the lens's zero-parallax point. In related art, to obtain high-quality panoramic images, photography must be centered around the nodal point. This ensures that the overlapping areas of adjacent photos taken in all directions are free of parallax, resulting in a more natural, seamless, and stitched panoramic image.

[0038] The axis of rotation refers to the corresponding rotation center of the camera in three-dimensional space during panoramic shooting. It is understandable that when determining the panoramic image of the target scene, it is usually necessary to first obtain an image within a 360-degree range of the target scene. In practical applications, images within a 360-degree range of the target scene can be obtained in at least two ways: fixing a camera on a gimbal, driving the camera to rotate by rotating the gimbal, and collecting images within a 360-degree range of the target scene during the rotation process; or placing multiple cameras around a certain center point, so that the multiple cameras shoot the target scene at different fixed angles, and the shooting ranges of the multiple cameras cover the target scene. In the above two methods, the rotation center of the gimbal and the center point around which the cameras revolve can both be considered as the axis of rotation in the embodiments of the present invention. It should be noted that in this embodiment, the rotation axis can be a real one, such as the rotation axis of the gimbal, or it can be a virtual one, such as the vertical line corresponding to the center point around which the cameras revolve.

[0039] In practice, if the camera node coincides with the axis of rotation, the image is actually taken with the camera node as the center point, and the overlapping areas of adjacent photos in each shooting direction do not have any parallax. However, adjusting the camera node so that it coincides with the axis of rotation requires repeated testing for the user, and the adjustment is difficult and requires high engineering error, making it difficult to implement. Therefore, when acquiring images used to generate panoramic images, there are often errors between the camera node and the axis of rotation. These errors can cause cracks and misalignment in the stitched panoramic image, resulting in low-quality panoramic images.

[0040] To solve at least one of the above technical problems, an embodiment of the present invention provides a panoramic image generation method. In this solution, depth information is introduced to reduce stitching errors caused by the misalignment of camera nodes and rotation axis centers.

[0041] Figure 1 A schematic diagram of a panoramic image generation system provided by an embodiment of the present invention is shown in FIG. Figure 1 The system shown includes: a combined computing unit consisting of an image acquisition unit and an image transmission unit, and a mobile device.

[0042] The image acquisition unit may be a fisheye camera or a non-fisheye camera, configured to capture multiple scene images of a target scene from multiple different angles around a rotation axis at a specific shooting point. In a specific implementation, the camera may optionally be fixed to a gimbal, which rotates the camera around the rotation axis, capturing multiple scene images of the target scene from multiple different angles during the rotation process. Alternatively, multiple cameras may be placed around the rotation axis, capturing multiple scene images of the target scene from different fixed angles.

[0043] The image transmission unit includes a communication chip for communicating with the mobile device to transmit the multiple scene images acquired by the image acquisition unit to the mobile device, so that the mobile device generates a panoramic image of the target scene based on the multiple scene images received.

[0044] The mobile device can be an electronic device such as a smart phone or a laptop computer, and is used to execute the panoramic image generation method provided by the embodiment of the present invention. In addition, the mobile device is also used to interact with the user during the panoramic image generation process, such as displaying the panoramic image.

[0045] Optionally, the combined calculation unit may also include a panoramic image generation unit configured to execute the panoramic image generation method provided in an embodiment of the present invention, and generate a panoramic image of the target scene based on the multiple scene images acquired by the image acquisition unit. In this embodiment, a mobile device is used as an example for illustration.

[0046] Figure 2 A flowchart of a panoramic image generation method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the following steps may be included:

[0047] 201. Acquire multiple scene images of the target scene and depth values ​​corresponding to each pixel point in each scene image. The multiple panoramic images are obtained by shooting the camera at multiple different angles around the axis of the rotation axis. The depth value is the distance between the object point corresponding to each pixel point in the three-dimensional space and the axis of the rotation axis when shooting the multiple scene images.

[0048] 202. Determine the target projection sphere corresponding to each pixel point by taking the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius.

[0049] 203. Determine the target latitude and longitude coordinates of each pixel point projected onto the corresponding target projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera.

[0050] 204. Convert the target latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane expanded images corresponding to the multiple scene images.

[0051] 205. Fuse multiple target plane unfolded images to obtain a target panoramic image of the target scene.

[0052] For ease of understanding, first combine Figure 3 An overview of the panoramic image generation process.

[0053] Figure 3 A schematic diagram of a panoramic image generation process provided by an embodiment of the present invention. Figure 3As shown, first, N (N is an integer greater than or equal to 2) scene images of the target scene are acquired by a camera, such as: image 1, image 2, ..., image N. The scene images can be fisheye images or non-fisheye images, and the type of scene images is not limited in this embodiment. Then, based on the camera intrinsic parameters and extrinsic parameters corresponding to the N scene images, the spherical coordinates corresponding to the pixel coordinates of each pixel in the N scene images on the projection sphere are determined, that is, the longitude and latitude coordinates of the pixel coordinates of each pixel in the N scene images are determined. The camera intrinsic parameters include the camera's focal length (fx, fy) in the x and y directions, the aperture center (cx, cy), the distortion coefficient, etc., and the camera extrinsic parameters include the camera pose information when acquiring the N scene images. Afterwards, the longitude and latitude coordinates corresponding to each pixel point are converted into two-dimensional plane coordinates in a panoramic coordinate system to obtain N planar unfolded images corresponding to the multiple scene images, and the image overlapping areas of the N planar unfolded images are fused to obtain a panoramic image of the target scene.

[0054] like Figure 3 As shown in Figure 2, in the process of generating panoramic images, "projection" and "expansion" are two important image processing links, and both links are related to the projection sphere. Figure 4 This paper describes the influence of the projection sphere on the quality of panoramic image generation.

[0055] Figure 4 A schematic diagram of a panoramic imaging model provided by an embodiment of the present invention. Figure 4 The left figure is an ideal panoramic imaging model, corresponding to the situation where the camera node coincides with the axis of rotation; Figure 4 The right figure in the figure shows a panoramic imaging model with a multi-lens combination, corresponding to the case where the camera node and the rotation axis do not coincide. The multi-lens combination can be understood as the multiple cameras mentioned above placed around a central point (the rotation axis), so that each camera captures the target scene at different fixed angles.

[0056] In this embodiment, Figure 4 The imaging geometry of the illustrated panoramic imaging model is based on the collinear relationship between the image point, object point, and projection center. This collinear relationship can be expressed using a collinearity equation. Here, the image point corresponds to the latitude and longitude coordinates of the pixel point on the projection sphere, the object point corresponds to the two-dimensional coordinates of the pixel point in the panoramic coordinate system, and the projection center corresponds to the rotation axis or camera nodal point. In this embodiment, the center of the projection sphere refers to the projection center corresponding to the rotation axis.

[0057] exist Figure 4In the ideal panoramic imaging model shown in the left figure, since the rotation axis coincides with the camera nodal point, the projection center S corresponding to the rotation axis coincides with the projection center C corresponding to the camera nodal point. Consequently, the object point P, which is collinear with the projection center S and the image point u, and the object point P' (not shown) which is collinear with the projection center C and the image point u, also coincide. In this case, regardless of the value of the projection sphere's radius r, SuP and CuP' always coincide, so object points P and P' always coincide, resulting in the absence of parallax as described above.

[0058] And in Figure 4 In the panoramic imaging model of the multi-lens combination shown in the right figure, since the axis of the rotation axis does not coincide with the camera node, the projection center S corresponding to the axis of the rotation axis does not coincide with the projection center C corresponding to the camera node. Based on the intrinsic and extrinsic parameters of the camera, the image point u in the camera coordinate system can be C Convert to the image point u in the projection sphere with the projection center S as the sphere center. Figure 4 As shown in the right figure, the object point P, which is collinear with the projection center S and the image point u, and the object point P', which is collinear with the projection center C and the image point u, do not coincide, indicating parallax. As mentioned earlier, parallax can cause cracks and misalignment in the stitched panoramic image.

[0059] Figure 4 In the illustrated case, the parallax is affected by the positional relationship between the rotation axis and the camera node. However, in actual applications, the parallax is also affected by the radius of the projection sphere.

[0060] In this embodiment, Figure 2 The panoramic image generation method illustrated in the example introduces depth information so that when the axis of the rotation axis does not coincide with the camera node, the object point P (the object point collinear with the projection center S and the image point u) also coincides with the object point P' (the object point collinear with the projection center C and the image point u).

[0061] Specifically, when generating a target panoramic image of a target scene, in addition to obtaining multiple scene images of the target scene, the depth value corresponding to each pixel in each scene image is also obtained. The depth value is the depth information mentioned above. Figure 1 The image acquisition unit in the panoramic image generation system shown captures the image and transmits it to the mobile device by the image transmission unit. It should be noted that the depth value corresponding to each pixel point in this embodiment is the distance between the object point corresponding to each pixel point in three-dimensional space and the axis of rotation when capturing multiple scene images.

[0062] Optionally, the depth value corresponding to each pixel in each scene image can be obtained in at least the following two ways.

[0063] The first acquisition method involves acquiring multiple sets of point cloud data corresponding to multiple scene images captured by a laser sensor. The point cloud data contains depth information between the laser sensor and object points in three-dimensional space. The depth value corresponding to each pixel in each scene image is determined based on a pre-calibrated first positional relationship between the laser sensor and the camera, and a second positional relationship between the camera and the axis of rotation. The camera's extrinsic parameters include the second positional relationship between the camera and the axis of rotation, which can be obtained through pre-calibration.

[0064] Optionally, to improve the accuracy of depth values, after acquiring multiple sets of point cloud data, preprocessing operations such as densification and hole filling can be performed on the point cloud data. For example, upsampling and smoothing the point cloud data or densifying the point cloud data using scene images; or using the Point Cloud Library (PCL) to fill holes in the point cloud data. Object detection can also be performed on the scene image to complete the point cloud data.

[0065] The second acquisition method is to obtain multiple scene images of the target scene, determine the initial projection sphere corresponding to the multiple panoramic images with the axis of the rotation axis as the projection center and a preset value as the radius; determine the initial latitude and longitude coordinates of each pixel point in the multiple scene images projected onto the initial projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera; convert the initial latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates under the panoramic coordinate system to obtain multiple initial plane unfolded images corresponding to the multiple scene images; fuse the multiple initial plane unfolded images to obtain the initial panoramic image of the target scene; input the initial panoramic image into a pre-trained neural network model to output the depth value corresponding to each pixel point in the initial panoramic image; and determine the depth value corresponding to each pixel point in each scene image based on the depth value output by the neural network model.

[0066] Of the two acquisition methods mentioned above, the first is more efficient and accurate, but requires an additional sensor, a laser sensor. The second does not require additional sensor equipment, but requires pre-training a neural network model and generating an initial panoramic image, making it relatively inefficient. In actual applications, users can choose different depth value acquisition methods based on their actual needs.

[0067] Afterwards, the target projection sphere corresponding to each pixel point is determined with the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius. Based on the pre-calibrated intrinsic and extrinsic parameters of the camera, the target latitude and longitude coordinates of each pixel point projected onto the corresponding target projection sphere are determined.

[0068] It can be understood that since the depth value is the distance between the object point corresponding to each pixel point in the three-dimensional space and the axis center of the rotation axis when taking multiple scene images, for any pixel point, its target latitude and longitude coordinates (corresponding to the image point) on the target projection sphere with the axis center of the rotation axis as the projection center and the depth value as the radius are actually also the object point corresponding to the pixel point.

[0069] For ease of understanding, combined Figure 5 Provide explanation. Figure 5 Corresponding to Figure 3 The situation shown in the right figure, Figure 5 Schematic diagram of another panoramic imaging model provided by an embodiment of the present invention. Figure 5 As shown, the dotted circle is the target projection sphere with a depth value and a radius r, as shown in Figure 5 As shown in the figure, the latitude and longitude coordinates of any pixel point on the target projection sphere, i.e., the image point u', coincide with the object point P', that is, the image point coincides with the actual object point. In this case, regardless of whether the projection center C corresponding to the camera node coincides with the projection center S corresponding to the rotation axis, the object point P (not shown) determined from the projection center S through the image point u' is the same as the object point P' determined from the projection center C through the image point u', because in this case, the object point P and the object point P' are both the image point u', that is, there is no parallax.

[0070] Therefore, in this solution, for any pixel in multiple scene images, the radius of the corresponding target projection sphere is set to its corresponding depth value, so that the pixel's image point on the target projection sphere coincides with the corresponding object point, eliminating parallax caused by the misalignment of the rotation axis and the camera node. Furthermore, when converting the target latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system, there is no error, resulting in multiple, more accurate, target plane unfolded images. Finally, by fusing multiple target plane unfolded images, the target panoramic image of the target scene is also of higher quality.

[0071] based on Figure 2To further improve the quality of panoramic images, the illustrated panoramic image generation method can also perform image preprocessing before projecting multiple scene images. For example, the multiple scene images can be uniformly illuminated. Specifically, based on the texture information of the multiple scene images, the image overlap regions between each pair are determined. The scene images are then color-space converted to a format with luminance channel values, such as HSV or YUV. The luminance channel values ​​corresponding to the image overlap regions are then uniformly adjusted by adjusting relevant parameters. For example, this adjustment can be performed using the formula I' = g*I + b, where I' represents the adjusted luminance channel value of the image overlap region, I represents the actual luminance channel value of the image overlap region in the scene image, g represents the gain, and b represents the bias. By adjusting g and b, the luminance channel values ​​of the image overlap regions are made consistent.

[0072] When the textures of multiple scene images are relatively rich, the texture information contained in the multiple scene images can also be used to correct the pre-calibrated camera extrinsics. For example, the camera pose information corresponding to the multiple scene images can be corrected, wherein the camera pose information includes rotation information and translation information. Specifically, feature matching can be performed through texture information to determine the pose transformation information between the multiple scene images, including rotation transformation information and translation transformation information; then, the pose transformation information is nonlinearly optimized through the bundle adjustment (BA) algorithm; finally, based on the optimized pose transformation information, more accurate camera pose information corresponding to the multiple scene images is determined. It can be understood that based on accurate camera pose information, the latitude and longitude coordinates of the pixel coordinate projection of each pixel in the multiple scene images are also more accurate, so that a more accurate target plane unfolded image can be obtained.

[0073] In the aforementioned embodiments, the quality of a panoramic image is improved by obtaining a more accurate target flat image. It is understood that when fusing multiple target flat images, the fusion method can also affect the quality of the target panoramic image. The following embodiments will illustrate the fusion process of multiple flat images.

[0074] Figure 6 A flowchart of an image fusion method provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the following steps may be included:

[0075] 601. Perform feature point matching on multiple target plane expanded images to determine an image overlap region between any two adjacent target plane expanded images in the multiple target plane expanded images, wherein similarities between multiple pairs of feature points corresponding to the image overlap region are greater than a first set threshold.

[0076] Optionally, feature points in the multiple target plane expanded images may be extracted and described using, for example, an ORB (Oriented FAST and Rotated BRIEF) algorithm. In this embodiment, there is no limitation on the extraction method and description method of the feature points.

[0077] It is understandable that the multiple target plane expanded images obtained by projecting multiple scene images onto a projection sphere are actually image sequences with a certain splicing order. Image overlapping areas for image fusion may exist between adjacent target planes. Therefore, when determining the image overlapping areas in multiple target plane expanded images, the similarity between the features of two adjacent target plane expanded images is calculated, where the similarity can be expressed by Ming distance, Euclidean distance, etc. If the similarity between the feature point i in the target plane expanded image 1 and the feature point j in the target plane expanded image 2 is greater than a first set threshold, then the feature point i and the feature point j are considered to be a pair of feature points; if there are multiple pairs of feature points greater than a preset number between the target plane expanded image 1 and the target plane expanded image 2, then the image overlapping area between the target plane expanded image 1 and the target plane expanded image 2 corresponding to these multiple pairs of feature points is determined.

[0078] It can be understood that the image overlap area between the two target plane expanded images actually corresponds to two partial images, the first partial image is located in the target plane expanded image 1, and the second partial image is located in the target plane expanded image 2, as shown in FIG. Figure 7 As shown, Figure 7 A schematic diagram of an image overlap region provided by an embodiment of the present invention. For ease of description, in this embodiment, the region corresponding to the first partial image and the second partial image is collectively referred to as the image overlap region.

[0079] 602. Determine a first stitching seam on the first target plane expanded image and a second stitching seam on the second target plane expanded image based on the similarity between multiple pairs of feature points corresponding to the image overlapping area; wherein the first target plane expanded image and the second target plane expanded image are any two target plane expanded images with image overlapping areas, and the first stitching seam and the second stitching seam are lines connecting pixel points corresponding to feature points in the multiple pairs of feature points whose similarity is greater than a second set threshold.

[0080] The seam is the line connecting the most similar pixels in the overlapping region of the first target plane expanded image and the second target plane expanded image. Most similar can be understood as having the smallest difference in color intensity or structure. Alternatively, the seam can be found using a point-by-point method, a dynamic programming method, or a graph cut method.

[0081] In this embodiment, by setting a second threshold, feature point pairs corresponding to the overlapping regions of the images are screened for similarity greater than the second threshold. The line connecting the corresponding pixels of the feature points in the screened feature point pairs is the stitching seam. Since the two feature points in each pair correspond to the first target plane expanded image and the second target plane expanded image, respectively, for ease of distinction, the line connecting the pixels corresponding to the feature points in the first target plane expanded image is referred to as the first stitching seam, and the line connecting the pixels corresponding to the feature points in the second target plane expanded image is referred to as the second stitching seam. In theory, the first stitching seam and the second stitching seam coincide.

[0082] 603. Determine a homography matrix between the first target plane expanded image and the second target plane expanded image based on multiple pairs of feature points corresponding to the image overlapping areas.

[0083] To improve the accuracy of the homography matrix, an optional method is to first determine the target image overlap region from the image overlap region corresponding to the first target plane unfolded image and the second target plane unfolded image. The target image overlap region includes the first stitching seam and the second stitching seam and is smaller than the image overlap region. Subsequently, the homography matrix between the first target plane unfolded image and the second target plane unfolded image is determined based on multiple pairs of target feature points corresponding to the target image overlap region. Alternatively, the homography matrix can be determined using the Lucas–Kanade optical flow algorithm based on a smaller target image overlap region.

[0084] 604. Based on the homography matrix, the first stitching seam, and the second stitching seam, stitch the multiple target plane unfolded images to obtain a target panoramic image of the target scene.

[0085] Specifically, based on the homography matrix, the positions of the first target plane expanded image and the second target plane expanded image are adjusted so that the first stitching seam and the second stitching seam coincide, completing the stitching of the first target plane expanded image and the second target plane expanded image. Similarly, multiple target plane expanded images are stitched together to obtain a target panoramic image of the target scene.

[0086] In practical applications, if the target panoramic image only uses the pixel values ​​corresponding to the expanded image of a certain target plane near the stitching seam position, there may be excessive and unnatural problems.

[0087] For example, Figure 8 A schematic diagram of a splicing seam provided by an embodiment of the present invention, Figure 8 Successor Figure 7 The situation shown. Figure 8As shown, assuming that the first target plane expanded image is target plane expanded image 1 and the second target plane expanded image is target plane expanded image 2, the multiple pairs of feature points in the image overlap region whose similarity is greater than a second set threshold are A1A2, B1B2, and C1C2, where the line connecting the pixels corresponding to A1B1C1 is the first stitching seam, and the line connecting the pixels corresponding to A2B2C2 is the first stitching seam. In the panoramic image 12 obtained after stitching, if the pixel values ​​to the left of the stitching seam in panoramic image 12 are all taken from target plane expanded image 1, and the pixel values ​​to the right of the stitching seam are all taken from target plane expanded image 2, the stitching seam in panoramic image 12 will be more obvious and unnatural.

[0088] Therefore, based on the homography matrix, the first stitching seam and the second stitching seam, multiple target plane unfolded images are stitched together to obtain a target panoramic image of the target scene, including:

[0089] Based on the homography matrix, the pixel coordinates of the first target plane unfolded image and the second target plane unfolded image are aligned. A first distance weight for the first pixel is determined based on the distance between the first pixel and the first boundary of the image overlap region. The first pixel is a pixel within the image overlap region in the first target plane unfolded image. A second distance weight for the second pixel is determined based on the distance between the second pixel and the second boundary of the image overlap region. The second pixel is a pixel within the image overlap region in the second target plane unfolded image. After alignment, the first and second pixels correspond to the same pixel coordinates. A weighted sum is performed on the pixel value of the first pixel, the first distance weight, the pixel value of the second pixel, and the second distance weight to obtain the pixel value of each pixel within the image overlap region in the target panoramic image.

[0090] It can be understood that for the pixel coordinates after alignment, if the pixel coordinates correspond to the image overlapping area, the same pixel coordinates will correspond to two pixels, which are located in different target plane expanded images, respectively, and are represented by the first pixel and the second pixel in this embodiment.

[0091] When determining the first distance weight and the second distance weight corresponding to the first pixel and the second pixel, respectively, different boundaries are selected as reference boundaries. In this embodiment, the first boundary and the second boundary are two relative boundaries of the image overlapping area and are distributed on both sides of the splicing seam. Figure 8 As shown, assuming that the first pixel point is a pixel point in the first partial image and the first boundary is the left boundary of the image overlapping area, the second boundary corresponding to the second pixel point in the second partial image is the right boundary of the image overlapping area.

[0092] When determining the distance weight of a target pixel, it is determined based on the distance between the pixels on each row of the reference boundary and the target pixel (e.g., Euclidean distance, Manhattan distance, etc.). The closer the target pixel is to its corresponding reference boundary, the greater its corresponding distance weight. For example, if the reference boundary corresponding to the target pixel x is the first boundary, and the target pixel x is a pixel on the first boundary, then the first distance weight corresponding to the target pixel is 1; if the reference boundary corresponding to the target pixel y is the first boundary, and the target pixel is a pixel on the second boundary, then the first distance weight of the target pixel y is 0.

[0093] After determining the first distance weight of the first pixel point and the second distance weight of the second pixel point, the pixel value of the same pixel point coordinates corresponding to the first pixel point and the second pixel point in the target panoramic image is: the pixel value of the first pixel point * the first distance weight + the pixel value of the second pixel point * the second distance weight.

[0094] In another optional embodiment, in order to improve the image fusion effect, the gray value error may also be used as an influencing factor for calculating the pixel value of the pixel point in the target panoramic image.

[0095] During the specific implementation process, first, the grayscale value error between the first pixel and the second pixel is determined. Then, based on the grayscale value error, the first grayscale weight corresponding to the first pixel and the second grayscale weight corresponding to the second pixel are determined. After that, the first weight of the first pixel is determined based on the first distance weight and the first grayscale weight; the second weight of the second pixel is determined based on the second distance weight and the second grayscale weight. For example: the first weight = the first distance weight + the first grayscale weight, the second weight = the second distance weight + the second grayscale weight. Finally, the pixel value of the first pixel, the first weight, the pixel value of the second pixel, and the second weight are weighted and summed to obtain the pixel value of each pixel in the overlapping area of ​​the image in the target panoramic image. The pixel value of the same pixel coordinate corresponding to the first pixel and the second pixel in the target panoramic image is: the pixel value of the first pixel * the first weight + the pixel value of the second pixel * the second weight.

[0096] In this embodiment, by setting different weights, the first pixel point and the second pixel point corresponding to the same pixel point coordinates in the image overlapping area are fused to determine the pixel value of the pixel point in the image overlapping area in the target panoramic image, so that the generated panoramic image splicing is more natural and of higher quality.

[0097] The following describes in detail the panoramic image generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0098] Figure 9A schematic structural diagram of a panoramic image generation device provided by an embodiment of the present invention is shown in FIG. Figure 9 As shown, the device includes: an acquisition module 11 and a processing module 12.

[0099] The acquisition module 11 is used to obtain multiple scene images of the target scene and the depth value corresponding to each pixel point in each scene image. The multiple panoramic images are obtained by shooting the camera at multiple different angles around the axis of the rotation axis. The depth value is the distance between the object point corresponding to each pixel point in the three-dimensional space and the axis of the rotation axis when shooting the multiple scene images.

[0100] The processing module 12 is used to determine the target projection sphere corresponding to each pixel point with the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius; determine the target longitude and latitude coordinates of each pixel point projected onto the corresponding target projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera; convert the target longitude and latitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane expanded images corresponding to the multiple scene images; and fuse the multiple target plane expanded images to obtain a target panoramic image of the target scene.

[0101] Optionally, the acquisition module 11 is specifically used to obtain multiple groups of point cloud data corresponding to the multiple scene images collected by the laser sensor; determine the depth value corresponding to each pixel point in each scene image based on a pre-calibrated first position relationship between the laser sensor and the camera, and a second position relationship between the camera and the axis of the rotation axis, wherein the external parameters of the camera include the second position relationship.

[0102] Optionally, the acquisition module 11 is further specifically used to determine, after acquiring multiple scene images of the target scene, an initial projection sphere corresponding to the multiple panoramic images with the axis of the rotation axis as the projection center and a preset value as the radius; determine the initial latitude and longitude coordinates of each pixel point in the multiple scene images projected onto the initial projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera; convert the initial latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple initial plane unfolded images corresponding to the multiple scene images; fuse the multiple initial plane unfolded images to obtain an initial panoramic image of the target scene; input the initial panoramic image into a pre-trained neural network model to output the depth value corresponding to each pixel point in the initial panoramic image; and determine the depth value corresponding to each pixel point in each scene image based on the depth value output by the neural network model.

[0103] Optionally, the multiple target plane unfolded images are an image sequence, and the processing module 12 is specifically used to perform feature point matching on the multiple target plane unfolded images to determine the image overlapping area between any two adjacent target plane unfolded images in the multiple target plane unfolded images, and the similarity between the multiple pairs of feature points corresponding to the image overlapping area is greater than a first set threshold; based on the similarity between the multiple pairs of feature points, determine the first stitching seam on the first target plane unfolded image and the second stitching seam on the second target plane unfolded image; wherein, the first target plane unfolded image and the second target plane unfolded image are any two target plane unfolded images with the image overlapping area, and the first stitching seam and the second stitching seam are lines connecting the pixel points corresponding to the feature points in the multiple pairs of feature points whose similarity is greater than a second set threshold; based on the multiple pairs of feature points, determine the homography matrix between the first target plane unfolded image and the second target plane unfolded image; based on the homography matrix, the first stitching seam and the second stitching seam, stitch the multiple target plane unfolded images to obtain a target panoramic image of the target scene.

[0104] Optionally, the processing module 12 is further specifically used to determine a target image overlapping area from the image overlapping area corresponding to the first target plane expanded image and the second target plane expanded image, wherein the target image overlapping area includes the first stitching seam and the second stitching seam and is smaller than the image overlapping area; and determine the homography matrix between the first target plane expanded image and the second target plane expanded image based on multiple pairs of target feature points corresponding to the target image overlapping area.

[0105] Optionally, the processing module 12 is further specifically used to perform pixel coordinate alignment on the first target plane unfolded image and the second target plane unfolded image based on the homography matrix; determine the first distance weight of the first pixel point according to the distance between the first pixel point and the first boundary of the image overlapping area, the first pixel point being the pixel point within the image overlapping area in the first target plane unfolded image; determine the second distance weight of the second pixel point according to the distance between the second pixel point and the second boundary of the image overlapping area, the second pixel point being the pixel point within the image overlapping area in the second target plane unfolded image, the first pixel point and the second pixel point corresponding to the same pixel coordinates after the alignment; perform weighted summation on the pixel value of the first pixel point, the first distance weight, the pixel value of the second pixel point and the second distance weight to obtain the pixel value of each pixel point in the image overlapping area in the target panoramic image.

[0106] Optionally, the processing module 12 is further specifically used to determine the grayscale value error between the first pixel point and the second pixel point; determine the first grayscale weight corresponding to the first pixel point and the second grayscale weight corresponding to the second pixel point based on the grayscale value error; determine the first weight of the first pixel point based on the first distance weight and the first grayscale weight; determine the second weight of the second pixel point based on the second distance weight and the second grayscale weight; and perform weighted summation on the pixel value of the first pixel point, the first weight, the pixel value of the second pixel point, and the second weight to obtain the pixel value of each pixel point in the image overlapping area in the target panoramic image.

[0107] Figure 9 The device shown can execute the steps introduced in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0108] In one possible design, the above Figure 9 The structure of the panoramic image generating device shown can be realized as an electronic device, such as Figure 10 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable code, and when the executable code is executed by the processor 22, the processor 22 can at least implement the panoramic image generation method provided in the above embodiments.

[0109] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the panoramic image generation method provided in the aforementioned embodiment.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0111] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating a panoramic image, characterized in that: include: Acquire multiple scene images of a target scene, and a depth value corresponding to each pixel in each scene image, wherein the multiple scene images are captured by a camera at multiple different angles around a rotation axis, and the depth value is the distance between the object point corresponding to each pixel in three-dimensional space and the rotation axis when the multiple scene images are captured; Taking the axis of the rotation axis as the projection center and the depth value corresponding to each pixel point as the radius, determining the target projection sphere corresponding to each pixel point; Determine the target latitude and longitude coordinates of each pixel point projected onto the corresponding target projection sphere based on the pre-calibrated intrinsic and extrinsic parameters of the camera; Converting the target latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain multiple target plane expanded images corresponding to the multiple scene images; The multiple target plane expanded images are fused to obtain a target panoramic image of the target scene.

2. The method according to claim 1, characterized in that The depth values ​​corresponding to each pixel in each scene image are obtained by: Acquire multiple sets of point cloud data corresponding to the multiple scene images collected by the laser sensor; Determine the depth value corresponding to each pixel point in each scene image based on a pre-calibrated first positional relationship between the laser sensor and the camera, and a second positional relationship between the camera and the axis of the rotation axis, wherein the external parameters of the camera include the second positional relationship.

3. The method according to claim 1, characterized in that The depth values ​​corresponding to each pixel in each scene image are obtained by: After acquiring multiple scene images of the target scene, determining the initial projection sphere corresponding to the multiple scene images with the axis of the rotation axis as the projection center and a preset value as the radius; Determining initial latitude and longitude coordinates of each pixel point in the plurality of scene images projected onto the initial projection sphere based on pre-calibrated intrinsic and extrinsic parameters of the camera; Converting the initial latitude and longitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain a plurality of initial plane expanded images corresponding to the plurality of scene images; fusing the multiple initial planar expanded images to obtain an initial panoramic image of the target scene; Inputting the initial panoramic image into a pre-trained neural network model to output a depth value corresponding to each pixel in the initial panoramic image; According to the depth value output by the neural network model, the depth value corresponding to each pixel in each scene image is determined.

4. The method according to any one of claims 1 to 3, characterized in that The plurality of target plane expanded images are an image sequence, and fusing the plurality of target plane expanded images to obtain a target panoramic image of the target scene includes: performing feature point matching on the plurality of target plane expanded images to determine an image overlap region between any two adjacent target plane expanded images among the plurality of target plane expanded images, wherein similarities between a plurality of pairs of feature points corresponding to the image overlap region are greater than a first set threshold; Determining, based on the similarities between the multiple pairs of feature points, a first stitching seam on the first target plane unfolded image and a second stitching seam on the second target plane unfolded image; wherein the first target plane unfolded image and the second target plane unfolded image are any two target plane unfolded images having the image overlapping area, and the first stitching seam and the second stitching seam are lines connecting the corresponding pixels of the feature points in the multiple pairs of feature points whose similarity is greater than a second set threshold; determining a homography matrix between the first target plane expanded image and the second target plane expanded image according to the plurality of pairs of feature points; Based on the homography matrix, the first stitching seam and the second stitching seam, the multiple target plane unfolded images are stitched together to obtain a target panoramic image of the target scene.

5. The method according to claim 4, characterized in that The determining, based on the multiple pairs of feature points, a homography matrix between the first target plane expanded image and the second target plane expanded image includes: Determine a target image overlapping area from image overlapping areas corresponding to the first target plane expanded image and the second target plane expanded image, where the target image overlapping area includes the first stitching seam and the second stitching seam and is smaller than the image overlapping area; A homography matrix between the first target plane expanded image and the second target plane expanded image is determined according to a plurality of pairs of target feature points corresponding to the overlapping areas of the target images.

6. The method according to claim 4, characterized in that The step of stitching the plurality of target plane expanded images based on the homography matrix, the first stitching seam, and the second stitching seam to obtain a target panoramic image of the target scene includes: Based on the homography matrix, performing pixel coordinate registration on the first target plane expanded image and the second target plane expanded image; determining a first distance weight of a first pixel point based on a distance between the first pixel point and a first boundary of the image overlapping region, the first pixel point being a pixel point within the image overlapping region in the expanded image of the first target plane; determining a second distance weight of the second pixel point based on a distance between the second pixel point and a second boundary of the image overlapping region, the second pixel point being a pixel point within the image overlapping region in the second target plane expanded image, the first pixel point and the second pixel point corresponding to the same pixel coordinates after the registration; Perform weighted summation on the pixel value of the first pixel point, the first distance weight, the pixel value of the second pixel point, and the second distance weight to obtain the pixel value of each pixel point in the image overlapping area in the target panoramic image.

7. The method according to claim 6, characterized in that The method further comprises: Determining a grayscale value error between the first pixel and the second pixel; Determining a first grayscale weight corresponding to the first pixel and a second grayscale weight corresponding to the second pixel according to the grayscale value error; Determining a first weight of the first pixel according to the first distance weight and the first grayscale weight; determining a second weight of the second pixel point according to the second distance weight and the second grayscale weight; Perform a weighted sum of the pixel value of the first pixel point, the first weight, the pixel value of the second pixel point, and the second weight to obtain the pixel value of each pixel point in the image overlapping area in the target panoramic image.

8. A panoramic image generating device, characterized in that: include: an acquisition module, configured to acquire multiple scene images of a target scene and a depth value corresponding to each pixel point in each scene image, wherein the multiple scene images are obtained by capturing the multiple scene images at multiple different angles around the axis of a rotation axis, and the depth value is the distance between the object point corresponding to each pixel point in three-dimensional space and the axis of the rotation axis when the multiple scene images are captured; a processing module configured to determine a target projection sphere corresponding to each pixel point using the axis of the rotation axis as a projection center and the depth value corresponding to each pixel point as a radius; determine target longitude and latitude coordinates of each pixel point projected onto the corresponding target projection sphere based on pre-calibrated intrinsic and extrinsic parameters of the camera; and convert the target longitude and latitude coordinates corresponding to each pixel point into two-dimensional plane coordinates in a panoramic coordinate system to obtain a plurality of target plane expanded images corresponding to the plurality of scene images; The multiple target plane expanded images are fused to obtain a target panoramic image of the target scene.

9. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the panoramic image generation method according to any one of claims 1 to 7.

10. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the panoramic image generation method according to any one of claims 1 to 7.

Images