Method, apparatus and storage medium for constructing a scene space model - Patents.com

Abstract

The present disclosure provides a method, an apparatus and a storage medium for constructing a scene space model, the method includes the steps of obtaining first point cloud information collected by a mobile point cloud collecting device, obtaining depth image information collected by a fixed-point depth photographing device, determining a rotation matrix of a camera coordinate system of the fixed-point depth photographing device corresponding to a global coordinate system, generating a first panoramic image according to the depth image information, mapping the first panoramic image to a three-dimensional unit sphere, rotating the three-dimensional unit sphere according to the rotation matrix to generate a second panoramic image, and generating a scene space model according to the first point cloud information and the second panoramic image.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This disclosure claims priority to a Chinese patent disclosure proposed to the China Patent Office on January 12, 2022, bearing disclosure number CN202210033988.6 and entitled "Method, Apparatus and Storage Medium for Constructing a Scene Space Model," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, and more particularly to a method, apparatus, and storage medium for constructing a scene space model. [Background technology]

[0003] Existing indoor 3D reconstruction technologies usually involve setting up laser scanning equipment or depth photography equipment indoors, adopting a fixed-point photography and collection method, collecting color data and depth data through the laser scanning equipment or depth photography equipment, and splicing point clouds of different points to generate 3D models of indoor scenes and panoramic images corresponding to each 3D model. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for constructing a scene space model. [Means for solving the problem]

[0005] According to a first aspect of an embodiment of the present disclosure, a method for constructing a scene space model is provided, the method including: acquiring first point cloud information corresponding to a target scene collected by a mobile point cloud collection device; acquiring depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, the depth image information including second point cloud information and color information of a corresponding image; determining a rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, the global coordinate system being a coordinate system corresponding to the first point cloud information; generating a first panoramic image based on the depth image information and mapping the first panoramic image to a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image.

[0006] Optionally, the step of determining a rotation matrix of the camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information includes a step of splicing second point cloud information collected by at least one fixed-point depth photography device to generate third point cloud information corresponding to a panorama of the target scene, a step of determining a first fixed-point rotation matrix of the camera coordinate system of the fixed-point depth photography device corresponding to the third point cloud information, a step of determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, and a step of determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0007] Optionally, the step of determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information is a step of matching the first point cloud information and the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

[0008] Optionally, the step of determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix includes the steps of: using the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth photography device; and using a point cloud matching algorithm to calculate the rotation matrix with the third fixed-point rotation matrix as an initial value, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

[0009] Optionally, the step of mapping the first panoramic image to a three-dimensional unit sphere includes mapping the first panoramic image to the three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0010] Optionally, the step of rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image includes rotating the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image, and generating a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0011] Optionally, the step of generating a second panoramic image based on the new three-dimensional coordinates and color information of at least one pixel includes determining a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adding color information of the at least one pixel of the first panoramic image to the new position to generate the second panoramic image.

[0012] Optionally, the step of generating a scene space model based on the first point cloud information and the second panoramic image includes a step of performing a surface reconstruction process on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information, a step of generating a map of the mesh based on mesh position information of the mesh model and the second panoramic image, and a step of applying the map to the corresponding mesh to generate a three-dimensional space model of the scene.

[0013] Optionally, the surface reconstruction algorithm includes a Possion surface reconstruction algorithm and the mesh includes a triangular mesh and a quadrilateral mesh.

[0014] According to a second aspect of the embodiment of the present disclosure, there is provided an apparatus for constructing a scene space model, the apparatus comprising: a first information acquisition module for acquiring first point cloud information corresponding to a target scene collected by a mobile point cloud collection device; a second information acquisition module used for acquiring depth image information corresponding to a partial region of the target scene collected by a fixed-point depth shooting device, the depth image information including second point cloud information and color information of a corresponding image; a rotation matrix determination module used for determining a rotation matrix of a camera coordinate system of the fixed-point depth shooting device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, the global coordinate system being a coordinate system corresponding to the first point cloud information; a panoramic image mapping module for generating a first panoramic image based on the depth image information and mapping the first panoramic image onto a three-dimensional unit sphere; a panoramic image transformation module for rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and a scene model generation module for generating a scene space model based on the first point cloud information and the second panoramic image.

[0015] Optionally, the rotation matrix determination module is used to splice second point cloud information collected by at least one fixed-point depth imaging device to generate third point cloud information corresponding to a panorama of the target scene, determine a first fixed-point rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to the third point cloud information, determine a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, and determine the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0016] Optionally, the rotation matrix determination module is further used to match the first point cloud information and the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

[0017] Optionally, the rotation matrix determination module further uses the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth photography device, and uses a point cloud matching algorithm to calculate the rotation matrix using the third fixed-point rotation matrix as an initial value, where the point cloud matching algorithm includes an iterative closest point algorithm.

[0018] Optionally, the panoramic image mapping module is specifically used to map the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0019] Optionally, the panoramic image transformation module is specifically used to rotate the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image, and generate a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0020] Optionally, the panoramic image transformation module further determines a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adds color information of the at least one pixel of the first panoramic image to the new position, which is used to generate the second panoramic image.

[0021] Optionally, the scene model generation module is specifically used to perform surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm, generate a mesh model corresponding to the first point cloud information, generate a map of the mesh based on mesh position information of the mesh model and the second panoramic image, and assign the map to the corresponding mesh, thereby generating a three-dimensional space model of the scene.

[0022] Optionally, the surface reconstruction algorithm includes a Possion surface reconstruction algorithm and the mesh includes a triangular mesh and a quadrilateral mesh.

[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device comprising: a memory for storing a computer program product; and a processor for executing a computer program product stored in the memory, the computer program product, when executed, implementing a method for constructing a scene space model as described in any of the above embodiments.

[0024] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium storing computer program instructions which, when executed by a processor, implements a method for constructing a scene space model as described in any of the above embodiments.

[0025] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product including computer program instructions, the computer program instructions implementing, when executed by a processor, a method for constructing a scene space model as described in any of the above embodiments. Effect of the Invention

[0026] Based on the method, apparatus, and storage medium for constructing a scene space model provided by the above embodiments of the present disclosure, a scene space model can be generated by integrating the features of a fixed-point depth shooting equipment and a mobile point cloud collection equipment, thereby reducing the splicing error of different points during the shooting process, and effectively solving the problem that accumulated errors are easily introduced in the existing scene space model construction method, resulting in serious distortion of the scene space model, improving the shooting efficiency, lowering the requirements for the shooting equipment, and improving the modeling efficiency and accuracy of the scene space model, so as to provide the user with a better VR display experience and improve the user's experience.

[0027] The technical solutions of the embodiments of the present disclosure are further described in detail below through drawings and examples. [Brief description of the drawings]

[0028] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] The present disclosure can be more clearly understood from the following detailed description, taken in conjunction with the drawings, in which:

[0030] [Figure 1] 1 is a flowchart of one embodiment of a method for constructing a scene space model of the present disclosure. [Diagram 2] 4 is a flowchart for determining a rotation matrix in one embodiment of a method for constructing a scene space model of the present disclosure. [Diagram 3] 11 is a flowchart illustrating a method for constructing a scene space model according to another embodiment of the present disclosure for determining a rotation matrix based on a first fixed-point rotation matrix and a second fixed-point rotation matrix. [Figure 4] 1 is a flowchart of generating a scene space model in one embodiment of a method for constructing a scene space model of the present disclosure. [Diagram 5]FIG. 2 is a structural schematic diagram of one embodiment of a method for constructing a scene space model of the present disclosure. [Figure 6] FIG. 1 is a structural diagram of an embodiment of an electronic device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the relative arrangement of the components and steps, formulas, values, etc. described in these embodiments do not limit the scope of the present disclosure unless otherwise specified.

[0032] At the same time, it should be understood that for ease of explanation, the dimensions of each part shown in the drawings are not drawn to scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to be construed as limiting the present disclosure and its application or uses.

[0034] Techniques, methods and equipment known to those skilled in the art may not be described in detail, but where appropriate, said techniques, methods and equipment should be considered as part of the description.

[0035] It should be noted that because like symbols and letters represent like items in the following accompanying drawings, once an item is defined in one accompanying drawing, there is no need to discuss it further in subsequent drawings.

[0036] Embodiments of the present disclosure may be applied to a computer system / server that may operate in conjunction with many other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronic products, networked personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments that include any of the above systems.

[0037] A computer system / server may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, target programs, components, logic, data structures, etc. for performing particular tasks or implementing particular abstract data types. The computer system / server may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in storage media, including storage devices, of local or remote computing systems.

[0038] Overview of the Application In the process of implementing the present disclosure, the inventors have found that for scenes with large area, cumulative errors are easily introduced when constructing a scene space model, resulting in serious distortion in the scene space model spliced ​​with point clouds collected at different points, and therefore a new scene space model construction scheme is needed.

[0039] The method for constructing a scene space model provided by the present disclosure includes obtaining first point cloud information corresponding to a panorama of a target scene collected by a mobile point cloud collection device, obtaining depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device based on the first point cloud information and the second point cloud information, determining a rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, generating a first panoramic image based on the depth image information, mapping the first panoramic image to a three-dimensional unit sphere, rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image, and generating a scene space model based on the first point cloud information and the second panoramic image, which can effectively reduce the distortion phenomenon of the scene space model, improve the photography efficiency, lower the requirements for the photography device, and improve the modeling efficiency and accuracy of the scene space model, so as to provide a better VR exhibition experience for users and improve the user experience.

[0040] Exemplary Methods In this disclosure, step numbers such as "Step 1", "Step 2", "S101", "S102", etc. are only intended to distinguish different steps and do not represent the order between different steps, and the order of steps having different numbers may be adjusted when performed.

[0041] FIG. 1 is a flow chart of one embodiment of a method for constructing a scene space model of the present disclosure, the method shown in FIG. 1 includes the following steps:

[0042] S101, obtaining first point cloud information corresponding to a panorama of a target scene, the first point cloud information being collected by a mobile point cloud collecting device;

[0043] In some alternative embodiments, the target scene may be a large-scale scene, such as a stadium, a museum, etc. The mobile point cloud collection devices may include a variety of devices, such as a handheld laser scanner, and the mobile point cloud collection devices may collect three-dimensional point cloud information.

[0044] In a target scene such as a stadium or a museum, a user holds a mobile point cloud collection device in his / her hand and walks through the target scene at a normal walking speed to collect panoramic point cloud information, i.e., first point cloud information, within the target scene using the mobile point cloud collection device. The information collected by the mobile point cloud collection device does not have high-resolution color data, so it is not possible to provide high-resolution color photos.

[0045] S102, obtaining depth image information corresponding to a partial region of a target scene, the depth image information being collected by a fixed-point depth photographing device.

[0046] Here, the depth image information includes the second point cloud information and color information of the corresponding image.

[0047] In some alternative embodiments, the fixed-point depth photographing device can include various photographing devices, such as depth cameras. At least one photographing point is provided in a target scene, such as a stadium, a museum, etc., and a depth photographing device is attached to the at least one photographing point as a fixed-point depth photographing device. The at least one fixed-point depth photographing device collects depth image information, including second point cloud information within the target area and color information of the image. The fixed-point depth photographing device can collect high-resolution color data, thereby providing high-resolution color photos.

[0048] S103, determining a rotation matrix of a camera coordinate system of a fixed-point depth photographing device corresponding to the global coordinate system according to the first point cloud information and the second point cloud information.

[0049] Here, the global coordinate system is a coordinate system corresponding to the first point cloud information, and the camera coordinate system of the fixed-point depth imaging device is a coordinate system corresponding to the point cloud data collected by the fixed-point depth imaging device.

[0050] In some alternative embodiments, the first point cloud information is panorama point cloud information corresponding to a panorama of a target scene, and the second point cloud information is point cloud information of a partial area in the target scene. By setting a coordinate system corresponding to the first point cloud information to a global coordinate system and determining a rotation matrix of the fixed-point depth photographing device in the global coordinate system, orientation information of the fixed-point depth photographing device in the panorama model generated by the first point cloud information can be obtained.

[0051] S104, generating a first panoramic image based on the depth image information, and mapping the first panoramic image onto a three-dimensional unit sphere.

[0052] In some alternative embodiments, any one or more existing methods can be used to splice depth image information collected by at least one fixed-point depth imaging device into a first panoramic image and map the first panoramic image onto a three-dimensional unit sphere.

[0053] S105, rotating the three-dimensional unit sphere according to the rotation matrix to generate a second panoramic image.

[0054] S106, generating a scene space model based on the first point cloud information and the second panoramic image.

[0055] In some alternative embodiments, the first point cloud information is used to generate a global space model, and image color information in the depth image information is obtained. The second point cloud information is matched with the first point cloud information to obtain the three-dimensional position information of the fixed-point depth photographing device in the panorama, so that the image color information is mapped to the corresponding position in the global model, and a scene space model is generated.

[0056] The method for constructing a scene space model in the above embodiment can integrate the features of a fixed-point depth shooting device and a mobile point cloud collection device to generate a scene space model, thereby providing users with a better virtual reality (VR) exhibition experience.

[0057] FIG. 2 is a flow chart for determining a rotation matrix in one embodiment of a method for constructing a scene space model of the present disclosure. The method shown in FIG. 2 includes the following steps:

[0058] S201, splicing second point cloud information collected by at least one fixed-point depth imaging device to generate third point cloud information corresponding to a panorama of a target scene.

[0059] In some alternative embodiments, any one or more existing splicing methods, such as an existing Iterative Closest Points (ICP) algorithm, can be used to splice at least a portion (e.g., all) of the second point cloud information (region point cloud information) to generate a third point cloud information (panorama point cloud information) corresponding to a panorama of the target scene.

[0060] S202, a first fixed-point rotation matrix of a camera coordinate system of a fixed-point type depth photographing device corresponding to the third point cloud information is determined.

[0061] In some alternative embodiments, when splicing by an iterative closest point algorithm, a first fixed-point rotation matrix of a fixed-point depth imaging device corresponding to the third point cloud information can be determined.

[0062] S203, determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information.

[0063] In some optional embodiments, the first point cloud information and the third point cloud information are matched to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, such as by manual positioning or point cloud feature extraction, or by a point cloud matching algorithm, including but not limited to an iterative closest point algorithm.

[0064] S204, determining a rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0065] FIG. 3 is a flowchart of determining a rotation matrix based on a first fixed-point rotation matrix and a second fixed-point rotation matrix in another embodiment of a method for constructing a scene space model of the present disclosure, and the method shown in FIG. 3 includes the following steps:

[0066] S301, a product of a first fixed-point rotation matrix and a second fixed-point rotation matrix is ​​used as a third fixed-point rotation matrix corresponding to a fixed-point depth imaging device.

[0067] S302, using a point cloud matching algorithm, calculate a rotation matrix using the third fixed point rotation matrix as an initial value.

[0068] Here, the point cloud matching algorithm includes an iterative closest point algorithm and the like.

[0069] In one embodiment, a user holds a mobile point cloud collection device in his / her hand and walks in the target scene at a normal walking speed, and collects a first point cloud information (denoted as GlobalCloud) of the entire shooting scene by the mobile point cloud collection device. A fixed-point depth shooting device at at least one shooting point collects depth image information including a second point cloud information (denoted as subLocalCloud) of a partial area in the target scene and color information of the image. Since the depth image data collected by the fixed-point depth shooting device is not finally used to generate a scene space model, multiple people can use the fixed-point depth shooting device in parallel to shoot, which can improve the shooting efficiency, and the area shot by people does not exceed a certain threshold, which can effectively reduce the splicing error of different points in the shooting process.

[0070] The coordinate system corresponding to the GlobalCloud is a global coordinate system, and the rotation matrix of each LocalCloud is positioned in the GlobalCloud; that is, the subLocalCloud (second point cloud information) photographed at each shooting point is spliced ​​to generate a LocalCloud, thereby obtaining the rotation matrix M0 (first fixed point rotation matrix) of the subLocalCloud relative to the LocalCloud collected at each individual shooting point.

[0071] Obtain the rotation matrix M1 (second fixed-point rotation matrix) of the LocalCloud relative to the GlobalCloud through manual positioning or point cloud feature extraction, and the iterative nearest neighbor algorithm, etc. Obtain the rotation matrix M3 (third fixed-point rotation matrix) = M1 * MO of the fixed-point depth imaging device at each single imaging point relative to the global coordinate system.

[0072] The rotation matrix M3 can be used as the initial value of a point cloud construction algorithm, such as the initial value of an iterative closest point algorithm, to calculate the rotation matrix M4 of a fixed-point depth shooting device at each shooting point in the GlobalCloud, and determine the rotation matrix of each LocalCloud in the GlobalCloud based on the rotation matrix M4.

[0073] For example, the rotation matrix M4 is given as follows:

number

number

[0074] FIG. 4 is a flowchart of generating a scene space model in one embodiment of the method for constructing a scene space model of the present disclosure. The method shown in FIG. 4 includes the following steps:

[0075] S401, mapping a first panoramic image to a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0076] S402, rotate the three-dimensional unit sphere according to the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image.

[0077] S403, generating a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0078] In some alternative embodiments, a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere is determined based on the new three-dimensional coordinates, and color information of the at least one pixel of the first panoramic image is added to the new position to generate a second panoramic image.

[0079] S404: performing a surface reconstruction process on the first point cloud information based on a surface reconstruction algorithm, and generating a mesh model corresponding to the first point cloud information.

[0080] The mesh model may be a polygonal mesh model, such as a triangular mesh model or a quadrilateral mesh model.

[0081] S405, generating a mesh map based on the mesh position information of the mesh model and the second panoramic image.

[0082] S406, the map is applied to a corresponding mesh to generate a 3D space model of the scene. The surface reconstruction algorithm includes, but is not limited to, the existing Position surface reconstruction algorithm, and the mesh includes, but is not limited to, a triangular mesh, a quadrilateral mesh, and the like.

[0083] In some alternative embodiments, the position and camera orientation of the fixed-point depth photographing device in the GlobalCloud can be determined by obtaining a rotation matrix M4, in which the R matrix (rotation matrix) is the camera orientation, and the three components of T are the three-dimensional positions x, y, and z of the fixed-point depth photographing device (subLocalCloud) in the GlobalCloud.

[0084] A first panoramic image is generated according to the depth image information collected by at least one fixed-point depth photographing device, and the first panoramic image is rotated, that is, the first panoramic image is projected onto a unit sphere, and the unit sphere is rotated using the R matrix, and reprojected onto the panorama to obtain the visual angle of the panoramic image in the global mode, that is, the second panoramic image. The three-dimensional coordinate point3 of each pixel of the first panoramic image on the unit sphere can be obtained in various ways, for example, the relevant code is as follows: / / 2d->3d voidPano2Cloud(intx_,inty_,PointTapoint3,introws,intcols) { floatX,y,z; double phi,theta; phi=(MLPI*y_) / rows; phi-=(M_PI / 2); y = sin(phi); theta=(2*M_PI*x_) / cols; if(theta>=1.5*M_PI){ x=sin(theta)*cos(phi); z=-Sqrt(l-x*x-y*y); } if(theta>=M_PI*0.5&&theta<=1.5*M_PI){ theta-=M_PI; ×=sin(theta)*cos(phi); / / z=sgrt(1-x*×-Y*y); }else{ theta = -theta; ×=sin(theta)*cos(phi); z=-sqrt(1-×*xy*y); } point3.x=x; point3.y=y; point3.z=z; }

[0085] Using the rotation matrix R, obtain new coordinates R*point3 of at least one pixel of the first panoramic image after rotation. Using the inverse operation of the operation method for projecting the first panoramic image onto the unit sphere, obtain the position of at least one pixel (e.g., each pixel) of the first panoramic image in the new panoramic image, and then obtain a new second panoramic image by writing the color RGB information of the at least one pixel of the first panoramic image at the original position to the new position.

[0086] A Poisson surface reconstruction algorithm is used to generate a mesh model of the GlobalCloud point cloud, and combined with the second panoramic image to generate a map of at least one of the mesh models, thereby completing the model reconstruction of the scene and generating a scene space model.

[0087] The method for constructing a scene space model in the above embodiment can effectively reduce the distortion phenomenon of the scene space model, improve the shooting efficiency, reduce the requirements for shooting equipment, improve the modeling efficiency and accuracy of the scene space model, provide users with a better virtual display exhibition experience, and improve the user experience.

[0088] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by instructing relevant hardware with a program, and the program may be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are executed, and the storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

[0089] FIG. 5 is a structural diagram of an embodiment of the apparatus of the present disclosure. The apparatus of this embodiment can be used to realize at least one embodiment of the method of the present disclosure. As shown in FIG. 5, the present disclosure provides an apparatus for constructing a scene space model. The apparatus includes a first information acquisition module 51, a second information acquisition module 52, a rotation matrix determination module 53, a panoramic image mapping module 54, a panoramic image transformation module 55 and a scene model generation module 56.

[0090] The first information acquisition module 51 acquires first point cloud information corresponding to a panorama of a target scene collected by a mobile point cloud collection device. The second information acquisition module 52 acquires depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, where the depth image information includes the second point cloud information and color information of the corresponding image.

[0091] The rotation matrix determination module 53 determines a rotation matrix of the camera coordinate system of the fixed-point depth photographing device corresponding to the global coordinate system based on the first point cloud information and the second point cloud information, and the global coordinate system is the coordinate system corresponding to the first point cloud information. The panoramic image mapping module 54 generates a first panoramic image based on the depth image information and maps the first panoramic image to a three-dimensional unit sphere. The panoramic image conversion module 55 rotates the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image. The scene model generation module 56 generates a scene space model based on the first point cloud information and the second panoramic image.

[0092] In some alternative embodiments, the rotation matrix determination module 53 splices the second point cloud information collected by at least one fixed-point depth photographing device to generate a third point cloud information corresponding to a panorama of the target scene. The rotation matrix determination module 53 determines a first fixed-point rotation matrix of the fixed-point depth photographing device corresponding to the third point cloud information. The rotation matrix determination module 53 determines a second fixed-point rotation matrix between the first point cloud information and the second depth photographing information, and determines a rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0093] For example, the rotation matrix determination module 53 matches the first point cloud information and the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, where the point cloud matching algorithm includes, but is not limited to, an iterative closest point algorithm, etc.

[0094] The rotation matrix determination module 53 uses the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device. The rotation matrix determination module 53 uses a point cloud matching algorithm to calculate a rotation matrix using the third fixed-point rotation matrix as an initial value.

[0095] In some alternative embodiments, the panoramic image mapping module 54 maps the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates. The panoramic image transformation module 55 rotates the three-dimensional unit sphere based on a rotation matrix to obtain new three-dimensional coordinates of the at least one pixel of the first panoramic image. The panoramic image transformation module 55 generates a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0096] For example, the panoramic image transformation module 55 determines a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adds color information of the at least one pixel of the first panoramic image to the new position to generate a second panoramic image.

[0097] The scene model generation module 56 performs surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm, and generates a mesh model corresponding to the first point cloud information. The scene model generation module 56 generates a map of the mesh based on the position information of the mesh of the mesh model and the second panoramic image, and provides the map to the corresponding mesh to generate a three-dimensional space model of the scene.

[0098] The business processing device in the above embodiment can effectively solve the problem that accumulated errors are easily introduced in the existing scene space model construction method, causing serious distortion in the scene space model, improve the shooting efficiency, reduce the requirements for shooting equipment, improve the modeling efficiency and accuracy of the scene space model, provide users with a better VR exhibition experience, and improve the user experience.

[0099] 6 is a structural diagram of an embodiment of an electronic device of the present disclosure. As shown in FIG. 6, an electronic device 61 includes one or more processors 611 and a memory 612.

[0100] The processor 611 may be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities and may control other components within the electronic device 61 to perform desired functions.

[0101] The memory 612 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, and flash memory. The computer-readable storage medium may store one or more computer program instructions, and the processor 611 may execute the program instructions to realize the method for constructing a scene space model in each embodiment of the present disclosure above and / or other desired functions. The computer-readable storage medium may also store various contents, such as an input signal, a signal component, and a noise component.

[0102] In one example, the electronic device 61 may further include an input device 613 and an output device 614, and these components are connected to each other via a bus system and / or other form of connection mechanism (not shown). The input device 613 may further include, for example, a keyboard, a mouse, etc. The output device 614 may output various information to the outside. The output device 614 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0103] Of course, for simplicity, only some of the components in the electronic device 61 related to the present disclosure are shown in Fig. 6, and components such as buses, input / output interfaces, etc. In addition, the electronic device 61 may further include any other suitable components according to specific application situations.

[0104] In addition to the methods and apparatus described above, an embodiment of the present disclosure may be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the method for constructing a scene space model according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0105] The computer program product may have program code written in any combination of one or more programming languages ​​for performing operations of the embodiments of the present disclosure, including object-oriented programming languages ​​such as Java, C++, and the like, and further including traditional procedural programming languages ​​such as "C" or similar programming languages. The program code may execute completely on the user's computing device, partially on the user's computing device, as a single separate software package, partially on the user's computing device and partially on a remote computing device, or completely on a remote computing device or server.

[0106] Additionally, an embodiment of the present disclosure may be a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform steps of a method for constructing a scene space model according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0107] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0108] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, it should be noted that the advantages, advantages, and effects mentioned in the present disclosure are only examples and are not limited thereto, and these advantages, advantages, and effects are not considered to be necessary for each embodiment of the present disclosure. In addition, the specific details of the above disclosure are only for illustrating examples and facilitating understanding, but are not limited thereto, and the above details are not intended to restrict the present disclosure to be realized using the above specific details.

[0109] In the method, device, electronic device, and storage medium for constructing a scene space model in the above embodiments, the features of a fixed-point depth shooting device and a mobile point cloud collection device can be integrated to generate a scene space model, thereby reducing the splicing error of different points during the shooting process, and effectively solving the problem that accumulated errors are easily introduced in the existing scene space model construction method, resulting in serious distortion of the scene space model, thereby improving the shooting efficiency, lowering the requirements for the shooting device, and improving the modeling efficiency and accuracy of the scene space model, so as to provide users with a better VR exhibition experience and improve the user experience.

[0110] At least one embodiment in this specification is described step by step, and the contents mainly described in each embodiment are the differences from other embodiments, and the similar parts between the embodiments can be mutually referred to. The system embodiment basically corresponds to the method embodiment, so it is described briefly, and the related contents can be referred to a part of the description of the method embodiment.

[0111] The method and apparatus of the present disclosure can be implemented in many ways. For example, the method and apparatus of the present disclosure can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration purposes only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specified. In some embodiments, the present disclosure can also be implemented as a program recorded on a recording medium, which program includes machine-readable instructions for implementing the method according to the present disclosure. Thus, the present disclosure also includes a storage medium having a program stored thereon for executing the method according to the present disclosure.

[0112] The description of the present disclosure has been provided for purposes of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better explain the principles and practical applications of the present disclosure, and to enable those skilled in the art to understand the present disclosure and to design various embodiments with various modifications suitable for particular applications.

Claims

1. 1. A method for constructing a scene space model, comprising: acquiring first point cloud information corresponding to a target scene, the first point cloud information being collected by a mobile point cloud collection device; acquiring depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photographing device, the depth image information including second point cloud information and color information of the corresponding image; determining a rotation matrix of a camera coordinate system of the fixed-point depth photographing device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, the global coordinate system being a coordinate system corresponding to the first point cloud information; generating a first panoramic image based on the depth image information and mapping the first panoramic image onto a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; generating a scene space model based on the first point cloud information and the second panoramic image.

2. The step of determining a rotation matrix of a camera coordinate system of the fixed-point depth photographing device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information includes: splicing second point cloud information collected by at least one fixed-point depth imaging device to generate a third point cloud information corresponding to a panorama of the target scene; determining a first fixed-point rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to the third point cloud information; determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

3. The step of determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information further comprises: Matching the first point cloud information with the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; 3. The method of claim 2, wherein the point cloud matching algorithm comprises an iterative closest point algorithm.

4. The step of determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix comprises: using the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device; 3. The method of claim 2, further comprising: calculating the rotation matrix using a point cloud matching algorithm with the third fixed-point rotation matrix as an initial value, the point cloud matching algorithm including an iterative closest point algorithm.

5. The step of mapping the first panoramic image to a three-dimensional unit sphere comprises:

2. The method of claim 1, further comprising: mapping the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

6. The step of rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image comprises: rotating the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image; and generating a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

7. generating a second panoramic image based on the new three-dimensional coordinates and color information of at least one pixel, determining a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates; and adding color information of at least one pixel of the first panoramic image to the new location to generate the second panoramic image.

8. The step of generating a scene space model based on the first point cloud information and the second panoramic image includes: performing a surface reconstruction process on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information; generating a map of the mesh based on position information of the mesh of the mesh model and the second panoramic image; and providing said maps to corresponding said meshes to generate a three-dimensional spatial model of said scene.

9. 9. The method of claim 8, wherein the surface reconstruction algorithm comprises a Poisson surface reconstruction algorithm, and the mesh comprises a triangular mesh and a quadrilateral mesh.

10. 1. An apparatus for constructing a scene space model, comprising: A first information acquisition module for acquiring first point cloud information corresponding to the target scene, the first point cloud information being collected by a mobile point cloud collection device; A second information acquisition module is used to acquire depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, the depth image information including second point cloud information and color information of the corresponding image; a rotation matrix determination module, which is used to determine a rotation matrix of a camera coordinate system of the fixed-point depth photographing device corresponding to a global coordinate system according to the first point cloud information and the second point cloud information, the global coordinate system being a coordinate system corresponding to the first point cloud information; a panoramic image mapping module for generating a first panoramic image not based on the depth image information and for mapping the first panoramic image onto a three-dimensional unit sphere; a panoramic image transformation module for rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; a scene model generation module for generating a scene space model based on the first point cloud information and the second panoramic image.

11. 11. The apparatus of claim 10, wherein the rotation matrix determination module is used to splice second point cloud information collected by at least one fixed-point depth photography device to generate third point cloud information corresponding to a panorama of the target scene, determine a first fixed-point rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to the third point cloud information, determine a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, and determine the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

12. 12. The apparatus of claim 11, wherein the rotation matrix determination module is further used to match the first point cloud information and the third point cloud information to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information based on a preset point cloud matching algorithm, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

13. 12. The apparatus of claim 11, wherein the rotation matrix determination module further uses the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device, and uses a point cloud matching algorithm to calculate the rotation matrix with the third fixed-point rotation matrix as an initial value, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

14. The apparatus of claim 10, characterized in that the panoramic image mapping module is specifically used for mapping the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

15. The apparatus of claim 10, wherein the panoramic image transformation module is specifically used to rotate the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image, and generate a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

16. 16. The apparatus of claim 15, wherein the panoramic image transformation module is further used to determine a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, add color information of the at least one pixel of the first panoramic image to the new position, and generate the second panoramic image.

17. The device according to any one of claims 10 to 16, characterized in that the scene model generation module is used to perform surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm, generate a mesh model corresponding to the first point cloud information, generate a map of the mesh based on mesh position information of the mesh model and the second panoramic image, and provide the map to the corresponding mesh, thereby generating a three-dimensional space model of the scene.

18. 18. The apparatus of claim 17, wherein the surface reconstruction algorithm comprises a Poisson surface reconstruction algorithm, and the mesh comprises a triangular mesh and a quadrilateral mesh.

19. A computer readable storage medium having computer program instructions stored thereon, comprising: A computer readable storage medium comprising computer program instructions which, when executed by a processor, implement the method of any one of claims 1 to 9.

20. An electronic device, a memory for storing a computer program product; 10. An electronic device comprising: a processor for executing a computer program product stored in said memory, said processor being adapted to implement a method according to any one of claims 1 to 9 when said computer program product is executed.

21. 1. A computer program product comprising computer program instructions, A computer program product comprising computer program instructions which, when executed by a processor, implements a method according to any one of claims 1 to 9.

Images