Three-dimensional image generation method, apparatus and device based on panoramic image, and storage medium

Abstract

The present invention relates to the field of image data processing. Disclosed are a three-dimensional image generation method, apparatus and device based on a panoramic image, and a storage medium, which are used for solving the technical problem in the prior art of a generated three-dimensional image being inaccurate due to relatively large errors in space coordinate points included in the three-dimensional image when the three-dimensional image is generated on the basis of a panoramic picture. The method comprises: acquiring from a panoramic picture a picture selection area of a three-dimensional image to be generated, and extracting the original pixel coordinates of each pixel point in the picture selection area; acquiring a depth image corresponding to the picture selection area; according to the original pixel coordinates and the depth image, calculating the three-dimensional coordinates corresponding to each pixel point in the picture selection area; identifying and removing background pixel points from the picture selection area, so as to obtain the three-dimensional coordinates corresponding to target pixel points, and forming a target three-dimensional coordinate set; and generating a bounding box on the basis of the target three-dimensional coordinate set, constructing a three-dimensional image surface, and generating, on the basis of the three-dimensional image surface, a three-dimensional image corresponding to the picture selection area.

Description

Method, device, equipment and storage medium for generating three-dimensional images based on panoramic images Technical Field

[0001] The present invention relates to the field of image data processing, and in particular to a method, device, equipment and storage medium for generating a three-dimensional image based on a panoramic image. Background Art

[0002] With the development of 3D visualization technology, solutions that generate 3D images with 3D spatial coordinates based on panoramic images for viewing or performing operations are increasingly being applied in various fields. For example, a terminal device such as a drone or inspection robot can use a camera to capture a panoramic photo of the current scene or object and generate the corresponding 3D image.

[0003] In existing technologies, pixels at specific locations in the scanned panoramic image are generally marked and the plane coordinates of the pixels are obtained. After the plane coordinates are converted into three-dimensional vectors, the collision point is calculated as the output result through ray detection. Relying on the panoramic image as the only source of input, the lack of depth data support leads to large errors in calculating the three-dimensional coordinates, which further causes inaccurate shapes of the subsequently generated three-dimensional images.

[0004] Summary of the Invention

[0005] The main purpose of the present invention is to solve the technical problem in the prior art that when generating a three-dimensional image based on a panoramic picture, the spatial coordinate points contained in the three-dimensional image have large errors and the generated three-dimensional image is inaccurate.

[0006] A first aspect of the present invention provides a method for generating a three-dimensional image based on a panoramic image, comprising: obtaining a picture selection in a panoramic picture for which a three-dimensional image is to be generated, and extracting the original pixel coordinates of each pixel point in the picture selection; obtaining a depth image corresponding to the picture selection; calculating the three-dimensional coordinates corresponding to each pixel point in the picture selection based on the original pixel coordinates and the depth image; identifying and eliminating background pixels in the picture selection to obtain the three-dimensional coordinates corresponding to target pixels to form a target three-dimensional coordinate set; generating a bounding box based on the target three-dimensional coordinate set, constructing a three-dimensional image surface based on the bounding box position and the target three-dimensional coordinate set, and generating a three-dimensional image corresponding to the picture selection based on the three-dimensional image surface.

[0007] Optionally, in a first implementation manner of the first aspect of the present invention, the calculating the three-dimensional coordinates corresponding to each pixel point in the image selection area based on the original pixel coordinates and the depth image includes: obtaining the observation origin corresponding to the panoramic image, and using the observation point as the three-dimensional ray origin; obtaining the focal length value of the panoramic image, calculating the ray near point and ray far point corresponding to the original pixel coordinates of each pixel point based on the original pixel coordinates and the focal length value, and determining the ray direction corresponding to each pixel point based on the ray near point and ray far point; generating three-dimensional rays corresponding to each pixel point based on the three-dimensional ray origin and the ray direction; generating UV coordinates corresponding to each pixel point based on the three-dimensional rays, and obtaining the depth value corresponding to each pixel point in the depth image based on the UV coordinates; scaling the depth value to obtain the three-dimensional coordinates corresponding to each pixel point.

[0008] Optionally, in a second implementation of the first aspect of the present invention, the identifying and eliminating background pixel points in the image selection area to obtain the three-dimensional coordinates corresponding to the target pixel points, and forming a target three-dimensional coordinate set includes: traversing the three-dimensional coordinates of each pixel point in the image selection area, and calculating the observation distance of each three-dimensional coordinate to the observation origin; sorting each three-dimensional coordinate according to the observation distance to obtain a three-dimensional coordinate sequence; performing a spatial continuity test on the three-dimensional coordinate sequence according to a pre-set distance threshold to identify background pixel points; deleting the three-dimensional coordinates corresponding to the background pixel points, and forming the remaining three-dimensional coordinates in the three-dimensional coordinate sequence into a target three-dimensional coordinate set.

[0009] Optionally, in a third implementation method of the first aspect of the present invention, constructing a three-dimensional image surface based on the bounding box position and the target three-dimensional coordinate set includes: obtaining the bounding box center coordinates of the bounding box; determining the plane to be constructed, and generating a reference plane based on the plane to be constructed; traversing the three-dimensional coordinates of the pixels contained in the plane to be constructed, converting the three-dimensional coordinates of the pixels contained in the plane to be constructed into the plane coordinate space corresponding to the reference plane, and calculating the extreme values ​​of the coordinates contained in the plane coordinate space; determining the image surface size based on the extreme values ​​of the coordinates, and generating the corresponding three-dimensional image surface based on the image surface size and the pixel depth.

[0010] Optionally, in a fourth implementation of the first aspect of the present invention, generating UV coordinates corresponding to each pixel point based on the three-dimensional ray includes: calculating the pitch angle and yaw angle of the three-dimensional ray; calculating the u coordinate value according to the pitch angle, and calculating the v coordinate value according to the yaw angle, to obtain the UV coordinates corresponding to each pixel point.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, obtaining the depth value corresponding to each pixel point in the depth image based on the UV coordinates includes: determining whether linear interpolation is required to obtain depth data; if so, using the UV coordinates to sample the depth values ​​of the four pixels adjacent to the pixel point in the depth image, and performing interpolation calculation to obtain the depth value corresponding to the pixel point; if not, using the UV coordinates to sample the depth values ​​at the position corresponding to the pixel point in the depth image to obtain the depth value corresponding to each pixel point.

[0012] Optionally, in a sixth implementation of the first aspect of the present invention, generating a reference plane based on the plane to be constructed includes: traversing the three-dimensional coordinates of the pixels contained in the plane to be constructed, and using the covariance matrix characteristic matrix to calculate the normal direction corresponding to the plane to be constructed; determining the sight vector according to the observation origin, and correcting the normal direction based on the sight vector; generating a normal vector according to the center coordinates of the bounding box and the normal direction; making an auxiliary plane perpendicular to the normal vector through the center of the bounding box, and calculating the maximum distance between the plane to be constructed and the auxiliary plane; translating the auxiliary plane along the normal vector by the maximum distance to obtain a reference plane.

[0013] A second aspect of the present invention provides a three-dimensional image generation device, comprising: a first acquisition module, used to acquire a picture selection in a panoramic picture for generating a three-dimensional image, and extract the original pixel coordinates of each pixel in the picture selection; a second acquisition module, used to acquire a depth image corresponding to the picture selection; a coordinate calculation module, used to calculate the three-dimensional coordinates corresponding to each pixel in the picture selection based on the original pixel coordinates and the depth image; a screening module, used to identify and eliminate background pixels in the picture selection, obtain the three-dimensional coordinates corresponding to the target pixel points, and form a target three-dimensional coordinate set; a plane construction module, used to generate a bounding box based on the target three-dimensional coordinate set, construct a three-dimensional image surface based on the bounding box position and the target three-dimensional coordinate set, and generate a three-dimensional image corresponding to the picture selection based on the three-dimensional image surface.

[0014] Optionally, in a first implementation manner of the second aspect of the present invention, the coordinate calculation module includes: a ray origin determination unit, used to obtain the observation origin corresponding to the panoramic image, and use the observation point as the three-dimensional ray origin; a ray direction determination unit, used to obtain the focal length value of the panoramic image, calculate the ray near point and ray far point corresponding to the original pixel coordinates of each pixel point based on the original pixel coordinates and the focal length value, and determine the ray direction corresponding to each pixel point based on the ray near point and ray far point; a three-dimensional ray generation unit, used to generate three-dimensional rays corresponding to each pixel point based on the three-dimensional ray origin and the ray direction; a depth value acquisition unit, used to generate UV coordinates corresponding to each pixel point based on the three-dimensional rays, and obtain the depth value corresponding to each pixel point in the depth image based on the UV coordinates; a ray scaling unit, used to scale the depth value to obtain the three-dimensional coordinates corresponding to each pixel point.

[0015] Optionally, in a second implementation of the second aspect of the present invention, the screening module includes: a distance calculation unit, used to traverse the three-dimensional coordinates of each pixel point in the image selection area, and calculate the observation distance of each three-dimensional coordinate to the observation origin; a sorting unit, used to sort each three-dimensional coordinate according to the observation distance to obtain a three-dimensional coordinate sequence; a background recognition unit, used to perform a spatial continuity test on the three-dimensional coordinate sequence according to a pre-set distance threshold to identify background pixel points; a coordinate elimination unit, used to delete the three-dimensional coordinates corresponding to the background pixel points, and form the remaining three-dimensional coordinates in the three-dimensional coordinate sequence into a target three-dimensional coordinate set.

[0016] Optionally, in a third implementation of the second aspect of the present invention, the plane construction module includes: a center coordinate calculation unit, used to obtain the center coordinates of the bounding box of the bounding box; a reference plane generation unit, used to determine the plane to be constructed, and generate a reference plane based on the plane to be constructed; an extreme value calculation unit, used to traverse the three-dimensional coordinates of the pixels contained in the plane to be constructed, convert the three-dimensional coordinates of the pixels contained in the plane to be constructed into the plane coordinate space corresponding to the reference plane, and calculate the extreme values ​​of the coordinates contained in the plane coordinate space; an image surface generation unit, used to determine the image surface size according to the extreme values ​​of the coordinates, and generate a corresponding three-dimensional image surface according to the image surface size and the pixel depth.

[0017] Optionally, in a fourth implementation method of the second aspect of the present invention, the depth value acquisition unit is further used to: calculate the pitch angle and yaw angle of the three-dimensional ray; calculate the u coordinate value according to the pitch angle, and calculate the v coordinate value according to the yaw angle, to obtain the UV coordinate corresponding to each pixel point.

[0018] Optionally, in the fifth implementation method of the second aspect of the present invention, the depth value acquisition unit is further used to: determine whether linear interpolation is required to obtain depth data; if so, use the UV coordinates to sample the depth values ​​of the positions of four pixels adjacent to the pixel point in the depth image, and perform interpolation calculations to obtain the depth value corresponding to the pixel point; if not, use the UV coordinates to sample the depth values ​​of the positions corresponding to the pixel point in the depth image to obtain the depth values ​​corresponding to each pixel point.

[0019] Optionally, in a sixth implementation method of the second aspect of the present invention, the reference plane generation unit is further specifically used to: traverse the three-dimensional coordinates of the pixels contained in the plane to be constructed, and use the covariance matrix characteristic matrix to calculate the normal direction corresponding to the plane to be constructed; determine the line of sight vector according to the observation origin, and correct the normal direction based on the line of sight vector; generate a normal vector according to the center coordinates of the bounding box and the normal direction; make an auxiliary plane perpendicular to the normal vector through the center of the bounding box, and calculate the maximum distance between the plane to be constructed and the auxiliary plane; translate the auxiliary plane along the normal vector by the maximum distance to obtain a reference plane.

[0020] A third aspect of the present invention provides a panoramic-view-based three-dimensional image generation device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory to enable the panoramic-view-based three-dimensional image generation device to perform the steps of the above-mentioned panoramic-view-based three-dimensional image generation method.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored therein, which, when executed on a computer, causes the computer to execute the steps of the above-mentioned method for generating a three-dimensional image of a panoramic image.

[0022] The technical solution provided by the present invention involves obtaining a selected area of ​​a panoramic image from which a three-dimensional image is to be generated, extracting the original pixel coordinates of each pixel within the selected area, obtaining a depth image corresponding to the selected area, calculating the three-dimensional coordinates corresponding to each pixel within the selected area based on the original pixel coordinates and the depth image, identifying and eliminating background pixels within the selected area to obtain a target three-dimensional coordinate set corresponding to the target pixel, generating a bounding box based on the target three-dimensional coordinate set, constructing a three-dimensional image surface, and generating a three-dimensional image corresponding to the selected area based on the three-dimensional image surface. This method combines the image pixel coordinates and depth information of the panoramic image to calculate the three-dimensional coordinates of each pixel, thereby improving the accuracy of the three-dimensional coordinate calibration and making the generated three-dimensional image more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] FIG1 is a schematic flow chart of a first embodiment of a method for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention;

[0025] FIG2 is a flow chart of a second embodiment of a method for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention;

[0026] FIG3 is a flow chart of a third embodiment of a method for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention;

[0027] FIG4 is a schematic diagram of an embodiment of a device for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention;

[0028] FIG5 is a schematic diagram of another embodiment of a device for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention;

[0029] FIG6 is a schematic diagram of an embodiment of a device for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in various forms, and it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, making it easier to fully convey the inventive concept to those skilled in the art. In the figures, the same reference numerals represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0031] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0032] In the description of specific embodiments, the features, structures, characteristics, or other details of the present invention are described to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from practicing the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0033] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0036] Referring to FIG. 1 , a first embodiment of a method for generating a three-dimensional image based on a panoramic image according to an embodiment of the present invention includes:

[0037] S101, obtaining a selected area of ​​a panoramic image for generating a three-dimensional image, and extracting the original pixel coordinates of each pixel point in the selected area of ​​the image;

[0038] It is understandable that the execution subject of the present invention can be a 3D image generating device based on a panoramic image, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0039] After receiving the request to generate a three-dimensional image, the server first obtains a panoramic image used to generate the three-dimensional image. In this embodiment, the panoramic image contains at least one object. When there are multiple objects, the target area in the panoramic image can be selected by rectangular selection and subsequent processing can be performed on it.

[0040] After obtaining the image selection in the panoramic image, the original pixel coordinates of each pixel in the image selection are extracted.

[0041] S102, obtaining a depth image corresponding to the selected area of ​​the image;

[0042] Before receiving a request to generate a 3D image, a panoramic image of the scene and / or object for which 3D image generation is desired must be obtained. This panoramic image may be captured by an image acquisition device positioned at the observation origin. Specifically, the image acquisition device may be a camera, laser scanner, or the like. In this embodiment, while simultaneously acquiring the panoramic image, the image acquisition device also acquires a depth image of the panoramic image at the time of capture. This depth image contains depth data, specifically the straight-line distance between each pixel in the panoramic image and the capture device.

[0043] In this step, according to the image selection area selected in the panoramic image, a depth image corresponding to the image selection area is obtained, and depth data of each pixel in the image selection area is obtained.

[0044] S103, calculating the three-dimensional coordinates corresponding to each pixel point in the image selection area according to the original pixel coordinates and the depth image;

[0045] The observation origin of the image acquisition device when acquiring the panoramic image in this embodiment is obtained, and the observation origin is used as the coordinate origin of the spherical coordinates; for each pixel point in the image selection area, the three-dimensional ray vector corresponding to each pixel point in the image selection area is calculated based on the original pixel coordinates of the pixel point and the distance between the pixel point and the coordinate origin of the spherical coordinates, and the three-dimensional ray vector is converted into units to obtain the three-dimensional coordinates.

[0046] In a specific implementation, first, the original pixel coordinates of each pixel point in the selected area are converted into a three-dimensional unit vector based on the origin position of the spherical coordinates, the three-dimensional unit vector is adjusted to be synchronized with the orientation of the panoramic image, and then the three-dimensional unit vector is normalized so that its value is a floating point number between 0 and 1, and sampling is performed in the depth map to obtain the depth distance corresponding to each pixel point; the three-dimensional coordinates of each pixel point in the image selection area are calculated based on the depth distance and the normalized three-dimensional unit vector.

[0047] S104, identifying and removing background pixels in the selected area of ​​the image, obtaining the three-dimensional coordinates corresponding to the target pixels, and forming a target three-dimensional coordinate set;

[0048] Since this embodiment uses a frame selection within a panoramic image to obtain an image selection, the selected image region may contain background pixels. Therefore, the 3D coordinates of each pixel within the image selection region are traversed, and the background pixels are determined based on the distance between each pixel and the observation origin. These background pixels are then removed, leaving the target pixels for 3D image generation. The target 3D coordinate set is then formed based on the 3D coordinates corresponding to these target pixels.

[0049] S105 , generating a bounding box based on the target three-dimensional coordinate set, constructing a three-dimensional image surface according to the bounding box position and the target three-dimensional coordinate set, and generating a three-dimensional image corresponding to the image selection area based on the three-dimensional image surface.

[0050] A bounding box is constructed according to the three-dimensional coordinates contained in the target three-dimensional coordinate set, and a corresponding three-dimensional image surface is constructed according to the center point position of the bounding box and the corresponding three-dimensional coordinate set, and a three-dimensional image corresponding to the image selection area is generated according to the three-dimensional image surface.

[0051] The embodiment of the present invention calculates the three-dimensional coordinates of each pixel point by integrating the image pixel coordinates and depth information of the panoramic image, thereby improving the accuracy of the three-dimensional coordinate calibration and making the generated three-dimensional image more accurate.

[0052] 2 , a second embodiment of a method for generating a 3D image based on a panoramic image according to an embodiment of the present invention includes:

[0053] S201, obtaining a selected area of ​​a panoramic image for generating a three-dimensional image, and extracting the original pixel coordinates of each pixel point in the selected area of ​​the image;

[0054] S202, obtaining a depth image corresponding to the selected area of ​​the image;

[0055] Step S201 and step S202 in this embodiment are substantially the same as step S101 and step S102 in the aforementioned embodiment, and therefore are not described again here.

[0056] S203, obtaining the observation origin corresponding to the panoramic image, and using the observation point as the origin of the three-dimensional ray;

[0057] S204: Obtain the focal length of the panoramic image, calculate the ray near point and ray far point corresponding to the original pixel coordinates of each pixel according to the original pixel coordinates and the focal length, and determine the ray direction corresponding to each pixel based on the ray near point and ray far point;

[0058] S205, generating a three-dimensional ray corresponding to each pixel point according to the three-dimensional ray origin and the ray direction;

[0059] In this step, taking a pixel within the selected area as an example, we first obtain the original pixel coordinates (x, y) of the current pixel and the corresponding imaging focal length f of the panoramic image. The pixel coordinates are normalized, and then the x value of the original pixel coordinate is mapped to the range [-1, 1], and the y value is flipped. Specifically, x1 = 2f × x - 1f, y1 = 1f - 2f × y. Based on x1 and y1, we obtain the ray near point near (x1, y1, 0.001f) and the ray far point far (x1, y1, 0.25f).

[0060] In this embodiment, before obtaining a selection of an image from a panoramic image for generating a three-dimensional image, and before obtaining the panoramic image through scanning by an image acquisition device, the panoramic scanning point is set as the observation origin. The view matrix M of the image acquisition device used when acquiring image information for the current panoramic image is obtained, the translation elements of the M matrix are set to 0, and the inverse matrix V of the view projection matrix is ​​obtained based on the M matrix. The position of the observation origin is set as the origin of the ray, and the direction vector D of the ray is calculated. Specifically, in this embodiment, D = V × far - V × near. The direction vector of the ray is then normalized, and the three-dimensional ray is obtained based on the ray origin and the ray direction.

[0061] S206, generating UV coordinates corresponding to each pixel point based on the three-dimensional ray, and obtaining a depth value corresponding to each pixel point in the depth image based on the UV coordinates;

[0062] Obtain the origin coordinates of the depth image, define the projection direction, and calculate the pitch angle and yaw angle of the three-dimensional ray according to the defined projection direction; convert the pitch angle and yaw angle into UV coordinate values ​​corresponding to the pixel points contained in the panoramic image. Specifically, calculate the u coordinate value according to the pitch angle, and calculate the v coordinate value according to the yaw angle to obtain the UV coordinates corresponding to each pixel point; in this step, the UV coordinate refers to the plane coordinate system established according to the depth map corresponding to the panoramic image, wherein the horizontal direction is U and the vertical direction is V. Any pixel on the depth map is located by this two-dimensional UV coordinate system of the plane.

[0063] Determine whether linear interpolation is required to obtain depth data; if so, use the UV coordinates to sample the depth values ​​of the four pixels adjacent to the pixel point in the depth image, and perform interpolation calculation to obtain the depth value corresponding to the pixel point; if not, use the UV coordinates to sample the depth value of the position corresponding to the pixel point in the depth image to obtain the depth value corresponding to each pixel point.

[0064] In a specific implementation, when a user sends a request to generate a 3D image to the server, they can choose whether to prioritize accuracy or speed when generating the image. Linear interpolation is used to obtain depth data when accuracy is desired, while linear interpolation is not used when speed is desired. In this step, the original pixel coordinates of the depth are first calculated by multiplying the UV coordinates by the image size. The data type is then converted to obtain the decimal fraction lost during the conversion process. Linear interpolation is then used to extract depth samples from the four surrounding pixels to obtain the depth value corresponding to each pixel.

[0065] S207, scaling the depth value to obtain the three-dimensional coordinates corresponding to each pixel point;

[0066] Specifically, the three-dimensional coordinates in this embodiment are calculated as the depth value in the three-dimensional ray direction of the observation point position; the ray vector corresponding to the three-dimensional ray is scaled according to the unit of the depth data obtained in this embodiment, and converted to the same as the original unit of each pixel point to obtain the final three-dimensional coordinates of each pixel point.

[0067] S208, identifying and removing background pixels in the selected area of ​​the image, obtaining the three-dimensional coordinates corresponding to the target pixel points, and forming a target three-dimensional coordinate set;

[0068] The three-dimensional coordinates of each pixel point in the selected area of ​​the image are traversed, and the observation distance from each three-dimensional coordinate to the observation origin is calculated; the three-dimensional coordinates are sorted according to the observation distance to obtain a three-dimensional coordinate sequence; the three-dimensional coordinate sequence is tested for spatial continuity according to a pre-set distance threshold to identify background pixels; the three-dimensional coordinates corresponding to the background pixels are deleted, and the remaining three-dimensional coordinates in the three-dimensional coordinate sequence are combined into a target three-dimensional coordinate set.

[0069] S209 , generating a bounding box based on the target three-dimensional coordinate set, constructing a three-dimensional image surface according to the bounding box position and the target three-dimensional coordinate set, and generating a three-dimensional image corresponding to the image selection area based on the three-dimensional image surface.

[0070] A bounding box is constructed based on the target three-dimensional coordinate set obtained after screening, and the center coordinates of the bounding box are obtained; the plane to be constructed is determined, and the three-dimensional coordinates of the pixels contained in the plane to be constructed are traversed, and the normal direction corresponding to the plane to be constructed is calculated using the covariance matrix characteristic matrix; the sight vector is determined according to the observation origin, and the normal direction is corrected based on the sight vector; a normal vector is generated according to the center coordinates of the bounding box and the normal direction; an auxiliary plane perpendicular to the normal vector is made through the center of the bounding box, and the maximum distance between the plane to be constructed and the auxiliary plane is calculated; the auxiliary plane is translated along the normal vector by the maximum distance to obtain a reference plane; the three-dimensional coordinates of the pixels contained on the plane to be constructed are traversed, and the three-dimensional coordinates of the pixels contained on the plane to be constructed are converted into the plane coordinate space corresponding to the reference plane, and the coordinate extreme values ​​contained in the plane coordinate space are calculated; the image surface size is determined according to the coordinate extreme values, and the corresponding three-dimensional image surface is generated according to the image surface size and the pixel depth.

[0071] The embodiments of the present invention calculate the three-dimensional coordinates of each pixel point by comprehensively combining the image pixel coordinates and depth information of the panoramic image, and calibrate the coordinates of each pixel in the panoramic image in three dimensions, thereby improving the accuracy of the three-dimensional coordinate calibration. The user can also choose to use or not use linear depth difference when obtaining the depth value through user settings, thereby reducing the impact of errors on the generated three-dimensional coordinates and improving the accuracy of subsequently generated three-dimensional images.

[0072] 3 , a third embodiment of a method for generating a 3D image based on a panoramic image according to an embodiment of the present invention includes:

[0073] S301, obtaining a selected area of ​​a panoramic image for generating a three-dimensional image, and extracting the original pixel coordinates of each pixel point in the selected area of ​​the image;

[0074] S302, obtaining a depth image corresponding to the selected area of ​​the image;

[0075] S303, calculating the three-dimensional coordinates corresponding to each pixel point in the image selection area according to the original pixel coordinates and the depth image;

[0076] S304, identifying and removing background pixels in the selected area of ​​the image, obtaining the three-dimensional coordinates corresponding to the target pixel points, and forming a target three-dimensional coordinate set;

[0077] In this embodiment, the specific contents of steps S301-S304 are substantially the same as those of steps S201-S208 in the aforementioned embodiment, and thus will not be described in detail here.

[0078] S305, generating a bounding box based on the target three-dimensional coordinate set, and obtaining the coordinates of the center of the bounding box;

[0079] A bounding box is constructed based on the target three-dimensional coordinate set obtained after screening, and the center coordinates of the bounding box are obtained. The bounding box described in this step is an algorithm for solving the optimal bounding space of a discrete point set. The basic idea is to use a geometric body with a slightly larger volume and simpler characteristics to approximate a complex geometric object. This geometric body is called the bounding box of the geometric object. Specifically, a cube-shaped bounding box is used in this embodiment.

[0080] S306, determining a plane to be constructed, traversing the three-dimensional coordinates of the pixels included in the plane to be constructed, and calculating the normal direction corresponding to the plane to be constructed using the covariance matrix characteristic matrix;

[0081] S307, determining a sight line vector according to the observation origin, and correcting the normal direction based on the sight line vector;

[0082] S308, generating a normal vector according to the center coordinates of the bounding box and the normal direction;

[0083] Taking the plane to be constructed as the plane facing the current observation origin as an example, traverse the target three-dimensional coordinates contained in the target three-dimensional coordinate set obtained above, calculate the covariance characteristic matrix of the current plane, and calculate the plane normal of the current plane based on the covariance characteristic matrix.

[0084] The coordinates of the center of the bounding box are subtracted from the coordinates of the observation origin to obtain the sight vector pointing from the center of the bounding box to the panoramic viewpoint; the direction of the plane normal is corrected according to the sight vector. Specifically, the correction method is to multiply the sight vector and the vector of the plane normal. If the result is negative, the direction of the plane normal is reversed.

[0085] S309, draw an auxiliary plane perpendicular to the normal vector through the center of the bounding box, and calculate the maximum distance between the plane to be constructed and the auxiliary plane;

[0086] S310, translating the auxiliary plane along the normal vector by a maximum distance to obtain a reference plane;

[0087] After obtaining the vector of the plane normal, make an auxiliary plane perpendicular to the normal vector through the center of the aforementioned bounding box, traverse the target three-dimensional coordinates, calculate the maximum distance between each pixel point and the current auxiliary plane, and after obtaining the maximum distance, translate the auxiliary plane along the positive direction of the normal vector by the maximum distance to obtain the reference plane to ensure that the auxiliary plane is not overlapped by the aforementioned pixel points.

[0088] S311, traversing the three-dimensional coordinates of the pixels on the plane to be constructed, converting the three-dimensional coordinates of the pixels on the plane to be constructed into a plane coordinate space corresponding to the reference plane, and calculating the extreme values ​​of the coordinates in the plane coordinate space;

[0089] According to the position of the auxiliary plane, the spatial transformation matrix and inverse matrix of the plane are determined, the target three-dimensional coordinates are traversed, each pixel point is converted to the plane coordinate space through the inverse matrix, and the extreme coordinate values ​​on the two axes in this plane coordinate space are calculated.

[0090] S312 , determining the image plane size according to the extreme values ​​of the coordinates, generating a corresponding three-dimensional image plane according to the image plane size and pixel depth, and generating a three-dimensional image corresponding to the image selection area based on the three-dimensional image plane.

[0091] The length and width of the image surface are determined according to the extreme values ​​of the coordinates. Based on the aforementioned pixel depth, a constructed three-dimensional image surface is generated. The above three-dimensional image surface is translated and scaled, and the three-dimensional image surface is rendered to generate a three-dimensional image corresponding to the image selection area.

[0092] The embodiments of the present invention calculate the three-dimensional coordinates of each pixel point by comprehensively combining the image pixel coordinates and depth information of the panoramic image, and calibrate the coordinates of each pixel in the panoramic image in three dimensions, thereby improving the accuracy of the three-dimensional coordinate calibration. The user can also choose to use or not use linear depth difference when obtaining the depth value through user settings, thereby reducing the impact of errors on the generated three-dimensional coordinates and improving the accuracy of the generated three-dimensional image.

[0093] The above describes a method for generating a 3D image based on a panoramic image according to an embodiment of the present invention. Now, a device for generating a 3D image based on a panoramic image according to an embodiment of the present invention will be described. Referring to FIG. 4 , one embodiment of the device for generating a 3D image based on a panoramic image according to an embodiment of the present invention includes:

[0094] The first acquisition module 401 is used to acquire a selected area of ​​a panoramic image for generating a 3D image, and extract the original pixel coordinates of each pixel point in the selected area of ​​the image;

[0095] The second acquisition module 402 is used to acquire a depth image corresponding to the selected area of ​​the image;

[0096] A coordinate calculation module 403 is configured to calculate the three-dimensional coordinates corresponding to each pixel point in the selected area of ​​the image according to the original pixel coordinates and the depth image;

[0097] The screening module 404 is used to identify and remove background pixels in the selected area of ​​the image, obtain the three-dimensional coordinates corresponding to the target pixels, and form a target three-dimensional coordinate set;

[0098] The plane construction module 405 is used to generate a bounding box based on the target three-dimensional coordinate set, construct a three-dimensional image surface corresponding to each bounding box surface of the bounding box, and generate a three-dimensional image corresponding to the image selection area based on the three-dimensional image surface.

[0099] The solution of an embodiment of the present invention obtains a picture selection in a panoramic picture for generating a three-dimensional image, and extracts the original pixel coordinates of each pixel point in the picture selection; obtains a depth image corresponding to the picture selection; calculates the three-dimensional coordinates corresponding to each pixel point in the picture selection based on the original pixel coordinates and the depth image; identifies and eliminates background pixels in the picture selection to obtain a target three-dimensional coordinate set corresponding to the target pixel point; generates a bounding box based on the target three-dimensional coordinate set, constructs a three-dimensional image surface, and generates a three-dimensional image corresponding to the picture selection based on the three-dimensional image surface. This method calculates the three-dimensional coordinates of each pixel point by combining the image pixel coordinates and depth information of the panoramic picture, thereby improving the accuracy of the three-dimensional coordinate calibration and making the generated three-dimensional image more accurate.

[0100] Please refer to Figure 5. In another embodiment of the present application, the coordinate calculation module 403 includes: a ray origin determination unit 4031, used to obtain the observation origin corresponding to the panoramic image, and use the observation point as the three-dimensional ray origin; a ray direction determination unit 4032, used to obtain the focal length value of the panoramic image, calculate the ray near point and the ray far point corresponding to the original pixel coordinates of each pixel point according to the original pixel coordinates and the focal length value, and determine the ray direction corresponding to each pixel point based on the ray near point and the ray far point; a three-dimensional ray generation unit 4033, used to generate a three-dimensional ray corresponding to each pixel point according to the three-dimensional ray origin and the ray direction; a depth value acquisition unit 4034, used to generate a UV coordinate corresponding to each pixel point based on the three-dimensional ray, and obtain a depth value corresponding to each pixel point in the depth image based on the UV coordinate; a ray scaling unit 4035, used to scale the depth value to obtain the three-dimensional coordinate corresponding to each pixel point.

[0101] In another embodiment of the present application, the screening module 404 includes: a distance calculation unit 4041, used to traverse the three-dimensional coordinates of each pixel point in the image selection area, and calculate the observation distance of each three-dimensional coordinate to the observation origin; a sorting unit 4042, used to sort each three-dimensional coordinate according to the observation distance to obtain a three-dimensional coordinate sequence; a background identification unit 4043, used to perform a spatial continuity test on the three-dimensional coordinate sequence according to a pre-set distance threshold to identify background pixel points; a coordinate elimination unit 4044, used to delete the three-dimensional coordinates corresponding to the background pixel points, and form the remaining three-dimensional coordinates in the three-dimensional coordinate sequence into a target three-dimensional coordinate set.

[0102] In another embodiment of the present application, the plane construction module 405 includes: a center coordinate calculation unit 4051, used to obtain the center coordinates of the bounding box of the bounding box; a reference plane generation unit 4052, used to determine the plane to be constructed and generate a reference plane based on the plane to be constructed; an extreme value calculation unit 4053, used to traverse the three-dimensional coordinates of the pixels contained in the plane to be constructed, convert the three-dimensional coordinates of the pixels contained in the plane to be constructed into the plane coordinate space corresponding to the reference plane, and calculate the coordinate extreme values ​​contained in the plane coordinate space; an image surface generation unit 4054, used to determine the image surface size according to the coordinate extreme values, and generate a corresponding three-dimensional image surface according to the image surface size and the pixel depth.

[0103] In another embodiment of the present application, the depth value acquisition unit 4034 is further specifically used to: calculate the pitch angle and yaw angle of the three-dimensional ray; calculate the u coordinate value according to the pitch angle, and calculate the v coordinate value according to the yaw angle to obtain the UV coordinate corresponding to each pixel point.

[0104] In another embodiment of the present application, the depth value acquisition unit 4034 is further specifically used to: determine whether linear interpolation is required to obtain depth data; if so, use the UV coordinates to sample the depth values ​​of the four pixels adjacent to the pixel point in the depth image, and perform interpolation calculations to obtain the depth value corresponding to the pixel point; if not, use the UV coordinates to sample the depth values ​​of the positions corresponding to the pixel point in the depth image to obtain the depth values ​​corresponding to each pixel point.

[0105] In another embodiment of the present application, the reference plane generation unit 4052 is further specifically used to: traverse the three-dimensional coordinates of the pixels contained in the plane to be constructed, and use the covariance matrix characteristic matrix to calculate the normal direction corresponding to the plane to be constructed; determine the line of sight vector according to the observation origin, and correct the normal direction based on the line of sight vector; generate a normal vector according to the center coordinates of the bounding box and the normal direction; make an auxiliary plane perpendicular to the normal vector through the center of the bounding box, and calculate the maximum distance between the plane to be constructed and the auxiliary plane; translate the auxiliary plane along the normal vector by the maximum distance to obtain a reference plane.

[0106] The solution of the embodiment of the present invention obtains a picture selection in a panoramic picture for generating a three-dimensional image, and extracts the original pixel coordinates of each pixel point in the picture selection; obtains a depth image corresponding to the picture selection; calculates the three-dimensional coordinates corresponding to each pixel point in the picture selection based on the original pixel coordinates and the depth image; identifies and eliminates background pixels in the picture selection to obtain a target three-dimensional coordinate set corresponding to the target pixel point; generates a bounding box based on the target three-dimensional coordinate set, constructs a three-dimensional image surface, and generates a three-dimensional image corresponding to the picture selection based on the three-dimensional image surface. This method calculates the three-dimensional coordinates of each pixel point by comprehensively combining the image pixel coordinates and depth information of the panoramic picture, calibrates the coordinates of each pixel in the panoramic picture, and improves the accuracy of the three-dimensional coordinate calibration. The user can also choose to use or not use linear depth difference when obtaining the depth value, thereby reducing the impact of errors on the generated three-dimensional coordinates and improving the accuracy of the generated three-dimensional image.

[0107] 4 and 5 above describe in detail the panoramic view-based three-dimensional image generation device in the embodiment of the present invention from the perspective of modular functional entities. The following describes in detail the panoramic view-based three-dimensional image generation device in the embodiment of the present invention from the perspective of hardware processing.

[0108] FIG6 is a schematic diagram of the structure of a panoramic-based 3D image generation device provided by an embodiment of the present invention. The panoramic-based 3D image generation device 600 may vary significantly depending on configuration or performance. The device may include one or more central processing units (CPUs) 610 (e.g., one or more processors), a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 633 or data 632. The memory 620 and storage media 630 may be either transient or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown), each of which may include a series of instructions for operating the panoramic-based 3D image generation device 600. Furthermore, the processor 610 may be configured to communicate with the storage medium 630 to execute the series of instructions stored in the storage medium 630 on the panoramic-based 3D image generation device 600.

[0109] The panoramic-view-based 3D image generation device 600 may further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the structure of the panoramic-view-based 3D image generation device shown in FIG6 does not limit the panoramic-view-based 3D image generation device, and the panoramic-view-based 3D image generation device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0110] The present invention also provides a three-dimensional image generation device based on a panoramic image, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the three-dimensional image generation method based on a panoramic image in the above-mentioned embodiments.

[0111] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to execute the steps of the method for generating a three-dimensional image based on a panoramic image.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0114] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0115] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for generating a three-dimensional image based on a panorama, It is characterized in that include: Acquire a picture selection area in the panoramic picture for generating a three-dimensional image, and extract the original pixel coordinates of each pixel point in the picture selection area; Obtaining a depth image corresponding to the selected area of ​​the image; Calculate the three-dimensional coordinates corresponding to each pixel point in the image selection area according to the original pixel coordinates and the depth image; Identify and remove background pixels in the selected area of ​​the image to obtain the three-dimensional coordinates corresponding to the target pixels to form a target three-dimensional coordinate set; A bounding box is generated based on the target three-dimensional coordinate set, a three-dimensional image surface is constructed according to the bounding box position and the target three-dimensional coordinate set, and a three-dimensional image corresponding to the image selection area is generated based on the three-dimensional image surface.

2. A method for generating a three-dimensional image based on a panoramic image according to claim 1, It is characterized in that The calculating the three-dimensional coordinates corresponding to each pixel point in the image selection area according to the original pixel coordinates and the depth image comprises: Obtaining the observation origin corresponding to the panoramic image, and taking the observation point as the origin of the three-dimensional ray; Obtaining a focal length value of the panoramic image, calculating a ray near point and a ray far point corresponding to the original pixel coordinates of each pixel point according to the original pixel coordinates and the focal length value, and determining a ray direction corresponding to each pixel point based on the ray near point and the ray far point; Generate a three-dimensional ray corresponding to each of the pixel points according to the three-dimensional ray origin and the ray direction; Generate UV coordinates corresponding to each of the pixel points based on the three-dimensional rays, and obtain depth values ​​corresponding to each of the pixel points in the depth image based on the UV coordinates; The depth value is scaled to obtain the three-dimensional coordinates corresponding to each pixel point.

3. The method for generating a three-dimensional image based on a panoramic image according to claim 2, It is characterized in that The identifying and removing background pixels in the selected area of ​​the image to obtain the three-dimensional coordinates corresponding to the target pixel points to form a target three-dimensional coordinate set includes: Traversing the three-dimensional coordinates of each pixel point in the selected area of ​​the image, and calculating the observation distance from each three-dimensional coordinate to the observation origin; Sorting the three-dimensional coordinates according to the observation distance to obtain a three-dimensional coordinate sequence; Performing a spatial continuity check on the three-dimensional coordinate sequence according to a preset distance threshold to identify background pixel points; The three-dimensional coordinates corresponding to the background pixel points are deleted, and the remaining three-dimensional coordinates in the three-dimensional coordinate sequence are combined into a target three-dimensional coordinate set.

4. The method for generating a three-dimensional image based on a panoramic image according to claim 3, It is characterized in that Generating a bounding box based on the target three-dimensional coordinate set and constructing a three-dimensional image surface according to the position of the bounding box and the target three-dimensional coordinate set comprises: Obtaining the center coordinates of the bounding box; Determine a plane to be constructed, and generate a reference plane based on the plane to be constructed; Traversing the three-dimensional coordinates of the pixels on the plane to be constructed, converting the three-dimensional coordinates of the pixels on the plane to be constructed into a plane coordinate space corresponding to the reference plane, and calculating the extreme values ​​of the coordinates contained in the plane coordinate space; The image plane size is determined according to the extreme values ​​of the coordinates, and the corresponding three-dimensional image plane is generated according to the image plane size and the pixel depth.

5. A method for generating a three-dimensional image based on a panoramic image according to claim 4, It is characterized in that Generating UV coordinates corresponding to each pixel point based on the three-dimensional ray includes: Calculating the pitch angle and yaw angle of the three-dimensional ray; The u coordinate value is calculated according to the pitch angle, and the v coordinate value is calculated according to the yaw angle, so as to obtain the UV coordinate corresponding to each pixel point.

6. A method for generating a three-dimensional image based on a panoramic image according to claim 5, It is characterized in that The acquiring the depth value corresponding to each pixel point in the depth image based on the UV coordinate comprises: Determine whether linear interpolation is required to obtain depth data; If yes, use the UV coordinates to sample the depth values ​​of the four pixels adjacent to the pixel in the depth image, and perform interpolation calculation to obtain the depth value corresponding to the pixel; If not, the depth value is sampled at the position corresponding to the pixel point in the depth image using the UV coordinate to obtain the depth value corresponding to each pixel point.

7. The method for generating a three-dimensional image based on a panoramic image according to claim 6, It is characterized in that The generating a reference plane based on the plane to be constructed comprises: Traversing the three-dimensional coordinates of the pixels included in the plane to be constructed, and using the covariance matrix feature matrix to calculate the normal direction corresponding to the plane to be constructed; Determine a sight line vector according to an observation origin, and correct the normal direction based on the sight line vector; Generate a normal vector according to the center coordinates of the bounding box and the normal direction; Make an auxiliary plane perpendicular to the normal vector through the center of the bounding box, and calculate the maximum distance between the plane to be constructed and the auxiliary plane; The auxiliary plane is translated along the normal vector by the maximum distance to obtain a reference plane.

8. A three-dimensional image generation device based on a panoramic image, It is characterized in that The three-dimensional image generating device based on the panoramic image comprises: A first acquisition module is used to acquire a picture selection area in the panoramic picture where a three-dimensional image is to be generated, and to extract the original pixel coordinates of each pixel point in the picture selection area; A second acquisition module is used to acquire a depth image corresponding to the selected area of ​​the picture; A coordinate calculation module, used to calculate the three-dimensional coordinates corresponding to each pixel point in the image selection area according to the original pixel coordinates and the depth image; A screening module is used to identify and remove background pixels in the selected area of ​​the image, obtain the three-dimensional coordinates corresponding to the target pixels, and form a target three-dimensional coordinate set; A plane construction module is used to generate a bounding box based on the target three-dimensional coordinate set, construct a three-dimensional image surface according to the bounding box position and the target three-dimensional coordinate set, and generate a three-dimensional image corresponding to the image selection area based on the three-dimensional image surface.

9. A three-dimensional image generation device based on a panorama, It is characterized in that The panoramic image-based three-dimensional image generation device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the panoramic image-based 3D image generation device to perform the steps of the panoramic image-based 3D image generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instructions are executed by a processor, the steps of the method for generating a three-dimensional image based on a panoramic image as described in any one of claims 1 to 7 are implemented.

Images