Method, apparatus and storage medium for constructing a scene space model

Abstract

The present disclosure provides a method, an apparatus and a storage medium for constructing a scene space model, the method includes the steps of obtaining first point cloud information collected by a mobile point cloud collecting device, obtaining depth image information collected by a fixed-point depth photographing device, determining a rotation matrix of a camera coordinate system of the fixed-point depth photographing device corresponding to a global coordinate system, generating a first panoramic image according to the depth image information, mapping the first panoramic image to a three-dimensional unit sphere, rotating the three-dimensional unit sphere according to the rotation matrix to generate a second panoramic image, and generating a scene space model according to the first point cloud information and the second panoramic image.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This disclosure claims priority to a Chinese patent disclosure submitted to the China Patent Office on January 12, 2022, bearing disclosure number CN202210033988.6 and entitled "Method, apparatus and storage medium for constructing a scene space model," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, and more particularly to a method, apparatus, and storage medium for constructing a scene space model. [Background technology]

[0003] Existing indoor 3D reconstruction technologies typically install laser scanning equipment or depth photography equipment indoors, adopt a fixed-point photography and collection method, collect color data and depth data using the laser scanning equipment or depth photography equipment, and splice point clouds of different points to generate 3D models of indoor scenes and panoramic images corresponding to each 3D model. Summary of the Invention [Problem to be solved by the invention]

[0004] Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for constructing a scene space model. [Means for solving the problem]

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for constructing a scene space model, the method including: acquiring first point cloud information corresponding to a target scene collected by a mobile point cloud collection device; acquiring depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, wherein the depth image information includes second point cloud information and color information of the corresponding image; determining a rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, wherein the global coordinate system is a coordinate system corresponding to the first point cloud information; generating a first panoramic image based on the depth image information and mapping the first panoramic image to a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image.

[0006] Optionally, the step of determining a rotation matrix of the camera coordinate system of the fixed-point depth imaging device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information includes the steps of splicing second point cloud information collected by at least one fixed-point depth imaging device to generate third point cloud information corresponding to a panorama of the target scene; determining a first fixed-point rotation matrix of the camera coordinate system of the fixed-point depth imaging device corresponding to the third point cloud information; determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; and determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0007] Optionally, the step of determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information is a step of matching the first point cloud information and the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

[0008] Optionally, the step of determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix includes the steps of: using the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device; and using a point cloud matching algorithm to calculate the rotation matrix using the third fixed-point rotation matrix as an initial value, wherein the point cloud matching algorithm includes an iterative nearest neighbor algorithm.

[0009] Optionally, mapping the first panoramic image to a three-dimensional unit sphere includes mapping the first panoramic image to a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0010] Optionally, the step of rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image includes the steps of rotating the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image, and generating a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0011] Optionally, the step of generating a second panoramic image based on the new three-dimensional coordinates and color information of at least one pixel includes determining a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adding color information of the at least one pixel of the first panoramic image to the new position to generate the second panoramic image.

[0012] Optionally, the step of generating a scene space model based on the first point cloud information and the second panoramic image includes a step of performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information, a step of generating a map of the mesh based on position information of a mesh of the mesh model and the second panoramic image, and a step of applying the map to the corresponding mesh to generate a three-dimensional space model of the scene.

[0013] Optionally, the surface reconstruction algorithm includes a Poisson surface reconstruction algorithm, and the mesh includes a triangular mesh and a quadrilateral mesh.

[0014] According to a second aspect of the present disclosure, there is provided an apparatus for constructing a scene space model, the apparatus comprising: a first information acquisition module for acquiring first point cloud information corresponding to a target scene collected by a mobile point cloud collection device; a second information acquisition module used to acquire depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, the depth image information including second point cloud information and color information of the corresponding image; a rotation matrix determination module used to determine a rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, the global coordinate system being a coordinate system corresponding to the first point cloud information; a panoramic image mapping module for generating a first panoramic image based on the depth image information and mapping the first panoramic image to a three-dimensional unit sphere; a panoramic image transformation module for rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and a scene model generation module for generating a scene space model based on the first point cloud information and the second panoramic image.

[0015] Optionally, the rotation matrix determination module is used to splice second point cloud information collected by at least one fixed-point depth imaging device to generate third point cloud information corresponding to a panorama of the target scene, determine a first fixed-point rotation matrix of the camera coordinate system of the fixed-point depth imaging device corresponding to the third point cloud information, determine a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, and determine the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0016] Optionally, the rotation matrix determination module is further used to match the first point cloud information with the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, wherein the point cloud matching algorithm includes an iterative closest point algorithm.

[0017] Optionally, the rotation matrix determination module further uses the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device, and uses a point cloud matching algorithm to calculate the rotation matrix using the third fixed-point rotation matrix as an initial value, where the point cloud matching algorithm includes an iterative nearest neighbor algorithm.

[0018] Optionally, the panoramic image mapping module is specifically used to map the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0019] Optionally, the panoramic image transformation module is specifically used to rotate the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image, and generate a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0020] Optionally, the panoramic image transformation module further determines a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adds color information of the at least one pixel of the first panoramic image to the new position, which is used to generate the second panoramic image.

[0021] Optionally, the scene model generation module is specifically used to perform surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm, generate a mesh model corresponding to the first point cloud information, generate a map of the mesh based on mesh position information of the mesh model and the second panoramic image, and assign the map to the corresponding mesh, thereby generating a three-dimensional space model of the scene.

[0022] Optionally, the surface reconstruction algorithm includes a Poisson surface reconstruction algorithm, and the mesh includes a triangular mesh and a quadrilateral mesh.

[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device comprising: a memory for storing a computer program product; and a processor for executing the computer program product stored in the memory, and when the computer program product is executed, for implementing the method for constructing a scene space model as described in any of the above embodiments.

[0024] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium storing computer program instructions that, when executed by a processor, implements a method for constructing a scene space model as set forth in any of the above embodiments.

[0025] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product including computer program instructions, the computer program instructions, when executed by a processor, implementing the method for constructing a scene space model according to any of the above embodiments. [Effects of the Invention]

[0026] Based on the method, apparatus, and storage medium for constructing a scene space model provided by the above embodiments of the present disclosure, a scene space model can be generated by integrating the features of a fixed-point depth photography device and a mobile point cloud collection device, thereby reducing the splicing error of different points during the photography process, and effectively solving the problem that accumulated errors are easily introduced in existing scene space model construction methods, resulting in serious distortion of the scene space model, thereby improving photography efficiency, lowering the requirements for photography equipment, and improving the modeling efficiency and accuracy of the scene space model, thereby providing users with a better VR display experience and improving the user experience.

[0027] The technical solutions of the embodiments of the present disclosure are further described in detail below through drawings and examples. [Brief explanation of the drawings]

[0028] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] The present disclosure can be more clearly understood from the following detailed description, taken in conjunction with the drawings, in which:

[0030] [Figure 1] 1 is a flowchart of one embodiment of a method for constructing a scene space model of the present disclosure. [Figure 2] 1 is a flowchart illustrating determining a rotation matrix in one embodiment of a method for constructing a scene space model of the present disclosure. [Figure 3] 10 is a flowchart illustrating a method for constructing a scene space model according to another embodiment of the present disclosure for determining a rotation matrix based on a first fixed-point rotation matrix and a second fixed-point rotation matrix. [Figure 4] 1 is a flowchart illustrating a method for generating a scene space model according to one embodiment of the present disclosure. [Figure 5]FIG. 2 is a structural schematic diagram of one embodiment of a method for constructing a scene space model of the present disclosure. [Figure 6] 1 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the relative arrangement of components and steps, formulas, numerical values, etc. described in these embodiments do not limit the scope of the present disclosure unless otherwise specified.

[0032] At the same time, it should be understood that for ease of explanation, the dimensions of the various parts shown in the drawings have not been drawn to scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to be construed as limiting the present disclosure and its application or uses.

[0034] Techniques, methods and equipment known to those skilled in the art may not be described in detail, but where appropriate, said techniques, methods and equipment should be considered part of the description.

[0035] It should be noted that similar symbols and letters represent similar items in the following accompanying drawings, so that once an item is defined in one accompanying drawing, there is no need to discuss it further in subsequent drawings.

[0036] Embodiments of the present disclosure may be applied to a computer system / server that can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics products, networked personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments that include any of the above systems.

[0037] A computer system / server may be described in the general context of computer system-executable instructions (e.g., program modules) executed by a computer system. Typically, program modules include routines, programs, target programs, components, logic, data structures, etc. for performing particular tasks or implementing particular abstract data types. The computer system / server may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in storage media, including storage devices, of local or remote computing systems.

[0038] Summary of the application In the process of realizing the present disclosure, the inventors found that for scenes with large area, cumulative errors are easily introduced when constructing a scene space model, resulting in serious distortion in the scene space model spliced ​​from point clouds collected at different points; therefore, a new scene space model construction scheme is needed.

[0039] The method for constructing a scene space model provided by the present disclosure includes: obtaining first point cloud information corresponding to a panorama of a target scene collected by a mobile point cloud collection device; obtaining depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device based on the first point cloud information and the second point cloud information; determining a rotation matrix of a camera coordinate system of the fixed-point depth photography device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information; generating a first panoramic image based on the depth image information; mapping the first panoramic image to a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image, thereby effectively reducing the distortion phenomenon of the scene space model, improving photography efficiency, lowering requirements for photography equipment, and improving the modeling efficiency and accuracy of the scene space model, thereby providing users with a better VR exhibition experience and improving the user experience.

[0040] Exemplary Methods In the present disclosure, step numbers such as "Step 1," "Step 2," "S101," and "S102" are only used to distinguish different steps and do not represent the order between different steps, and the order of steps with different numbers may be adjusted when performed.

[0041] FIG. 1 is a flowchart of one embodiment of a method for constructing a scene space model of the present disclosure, and the method shown in FIG. 1 includes the following steps:

[0042] S101, obtaining first point cloud information corresponding to a panorama of a target scene collected by a mobile point cloud collecting device;

[0043] In some alternative embodiments, the target scene may be a large-scale scene, such as a stadium, a museum, etc. The mobile point cloud collection device may include a variety of devices, such as a handheld laser scanner, and the mobile point cloud collection device may collect three-dimensional point cloud information.

[0044] In a target scene such as a stadium or a museum, a user holds a mobile point cloud collection device and walks through the target scene at a normal walking speed to collect panoramic point cloud information, i.e., first point cloud information, within the target scene using the mobile point cloud collection device. However, the information collected by the mobile point cloud collection device does not contain high-resolution color data, making it impossible to provide high-resolution color photographs.

[0045] S102, obtaining depth image information corresponding to a partial region of a target scene collected by a fixed-point depth photographing device;

[0046] Here, the depth image information includes the second point cloud information and color information of the corresponding image.

[0047] In some alternative embodiments, the fixed-point depth photographing device may include various photographing devices, such as a depth camera. At least one photographing point is provided in a target scene, such as a stadium or a museum, and a depth photographing device is attached to the at least one photographing point as a fixed-point depth photographing device. The at least one fixed-point depth photographing device collects depth image information including second point cloud information within the target area and image color information. The fixed-point depth photographing device can collect high-resolution color data, thereby providing high-resolution color photographs.

[0048] S103: determining a rotation matrix of a camera coordinate system of a fixed-point depth photographing device corresponding to the global coordinate system according to the first point cloud information and the second point cloud information;

[0049] Here, the global coordinate system is a coordinate system corresponding to the first point cloud information, and the camera coordinate system of the fixed-point depth imaging device is a coordinate system corresponding to the point cloud data collected by the fixed-point depth imaging device.

[0050] In some alternative embodiments, the first point cloud information is panorama point cloud information corresponding to a panorama of the target scene, and the second point cloud information is point cloud information of a partial region within the target scene. By setting a coordinate system corresponding to the first point cloud information to a global coordinate system and determining a rotation matrix of the fixed-point depth imaging device in the global coordinate system, orientation information of the fixed-point depth imaging device in the panorama model generated by the first point cloud information can be obtained.

[0051] S104, generating a first panoramic image based on the depth image information, and mapping the first panoramic image onto a three-dimensional unit sphere.

[0052] In some alternative embodiments, any one or more existing methods can be used to splice depth image information collected by at least one fixed-point depth imaging device into a first panoramic image and map the first panoramic image onto a three-dimensional unit sphere.

[0053] S105, rotating the three-dimensional unit sphere according to the rotation matrix to generate a second panoramic image.

[0054] S106: generating a scene space model based on the first point cloud information and the second panoramic image;

[0055] In some alternative embodiments, the first point cloud information is used to generate a global space model to obtain color information of the image in the depth image information, and the second point cloud information is matched with the first point cloud information to obtain three-dimensional position information of the fixed-point depth photographing device in the panorama, thereby mapping the color information of the image to a corresponding position in the global model to generate a scene space model.

[0056] The method for constructing a scene space model in the above embodiment can integrate the features of a fixed-point depth photography device and a mobile point cloud collection device to generate a scene space model, thereby providing users with a better virtual reality (VR) exhibition experience.

[0057] FIG. 2 is a flowchart for determining a rotation matrix in one embodiment of a method for constructing a scene space model of the present disclosure. The method shown in FIG. 2 includes the following steps:

[0058] S201, splicing second point cloud information collected by at least one fixed-point depth imaging device to generate third point cloud information corresponding to a panorama of a target scene.

[0059] In some alternative embodiments, any one or more existing splicing methods, such as an existing Iterative Closest Points (ICP) algorithm, can be used to splice at least a portion (e.g., all) of the second point cloud information (region point cloud information) to generate third point cloud information (panorama point cloud information) corresponding to a panorama of the target scene.

[0060] S202: determining a first fixed-point rotation matrix of a camera coordinate system of a fixed-point depth photographing device corresponding to the third point cloud information;

[0061] In some alternative embodiments, when splicing by an iterative closest point algorithm, a first fixed-point rotation matrix of the fixed-point depth imaging device corresponding to the third point cloud information can be determined.

[0062] S203: determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information;

[0063] In some alternative embodiments, the first point cloud information and the third point cloud information are matched to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information by manual positioning or point cloud feature extraction, or a point cloud matching algorithm, such as, but not limited to, an iterative closest point algorithm.

[0064] S204, determining a rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0065] FIG. 3 is a flowchart illustrating another embodiment of a method for constructing a scene space model according to the present disclosure, in which a rotation matrix is ​​determined based on a first fixed-point rotation matrix and a second fixed-point rotation matrix. The method illustrated in FIG. 3 includes the following steps:

[0066] S301, the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix is ​​used as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device.

[0067] S302, using a point cloud matching algorithm, calculate a rotation matrix using the third fixed point rotation matrix as an initial value.

[0068] Here, the point cloud matching algorithm includes an iterative closest point algorithm and the like.

[0069] In one embodiment, a user holds a mobile point cloud collection device and walks through a target scene at a normal walking speed, collecting first point cloud information (referred to as GlobalCloud) of the entire scene. A fixed-point depth photography device at at least one photography point collects second point cloud information (referred to as subLocalCloud) of a subregion within the target scene and depth image information including image color information. Because the depth image data collected by the fixed-point depth photography device is not ultimately used to generate a scene space model, multiple people can use the fixed-point depth photography device in parallel to take photos, improving photography efficiency. Since the area photographed by multiple people does not exceed a certain threshold, splicing errors between different points during the photography process can be effectively reduced.

[0070] The coordinate system corresponding to the GlobalCloud is a global coordinate system, and the rotation matrix of each LocalCloud is positioned in the GlobalCloud. That is, by splicing the subLocalCloud (second point cloud information) photographed at each shooting point and generating a LocalCloud, the rotation matrix M0 (first fixed point rotation matrix) of the subLocalCloud relative to the LocalCloud collected at each individual shooting point can be obtained.

[0071] Obtain the rotation matrix M1 (second fixed point rotation matrix) of the LocalCloud relative to the GlobalCloud through manual positioning or point cloud feature extraction, and the iterative nearest neighbor algorithm, etc. Obtain the rotation matrix M3 (third fixed point rotation matrix) = M1 * MO of the fixed-point depth camera at each individual shooting point relative to the global coordinate system.

[0072] The rotation matrix M3 can be used as the initial value of a point cloud construction algorithm, for example, the initial value of an iterative closest point algorithm, to calculate the rotation matrix M4 of the fixed-point depth shooting device at each shooting point in the GlobalCloud, and the rotation matrix of each LocalCloud in the GlobalCloud can be determined based on the rotation matrix M4.

[0073] For example, the rotation matrix M4 is given as follows:

number

number

[0074] FIG. 4 is a flowchart of generating a scene space model in one embodiment of the method for constructing a scene space model of the present disclosure, and the method shown in FIG. 4 includes the following steps:

[0075] S401, mapping a first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

[0076] S402, rotate the three-dimensional unit sphere according to the rotation matrix to obtain new three-dimensional coordinates of at least one pixel of the first panoramic image.

[0077] S403, generating a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0078] In some alternative embodiments, a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere is determined based on the new three-dimensional coordinates, and color information of the at least one pixel of the first panoramic image is added to the new position to generate a second panoramic image.

[0079] S404: Performing surface reconstruction processing on the first point cloud information based on the surface reconstruction algorithm, and generating a mesh model corresponding to the first point cloud information.

[0080] The mesh model may be a polygonal mesh model such as a triangular mesh model or a quadrilateral mesh model.

[0081] S405: generating a mesh map based on the mesh position information of the mesh model and the second panoramic image;

[0082] S406: The map is applied to a corresponding mesh to generate a 3D space model of the scene. The surface reconstruction algorithm includes, but is not limited to, the existing Position surface reconstruction algorithm, and the mesh includes, but is not limited to, a triangular mesh, a quadrilateral mesh, and the like.

[0083] In some alternative embodiments, the position of the fixed-point depth imaging device and the orientation of the camera in the GlobalCloud can be determined by obtaining a rotation matrix M4, in which the R matrix (rotation matrix) is the orientation of the camera, and the three components of T are the three-dimensional positions x, y, and z of the fixed-point depth imaging device (subLocalCloud) in the GlobalCloud.

[0084] A first panoramic image is generated based on depth image information collected by at least one fixed-point depth photographing device, and the first panoramic image is rotated (i.e., projected onto a unit sphere), and the unit sphere is rotated using the R matrix and re-projected onto the panorama to obtain the panoramic image's viewing angle in global mode (i.e., a second panoramic image). The three-dimensional coordinate point3 of each pixel of the first panoramic image on the unit sphere can be obtained in various ways, for example, the relevant code is as follows: / / 2d->3d voidPano2Cloud(intx_,inty_,PointTapoint3,introws,intcols) { floatX,y,z; double phi,theta; phi=(MLPI*y_) / rows; phi-=(M_PI / 2); y=sin(phi); theta=(2*M_PI*x_) / cols; if(theta>=1.5*M_PI){ x=sin(theta)*cos(phi); z=-Sqrt(l-×*×-y*y); } if(theta>=M_PI*0.5&&theta<=1.5*M_PI){ theta-=M_PI; ×=sin(theta)*cos(phi); / / z=sgrt(1-x*×-Y*y); }else{ theta=-theta; ×=sin(theta)*cos(phi); z=-sqrt(1-×*xy*y); } point3.x=x; point3.y=y; point3.z=z; }

[0085] A new coordinate R*point3 of at least one pixel of the first panoramic image after rotation is obtained using the rotation matrix R. A position of at least one pixel (e.g., each pixel) of the first panoramic image in the new panoramic image is obtained using the inverse operation of the operation method for projecting the first panoramic image onto the unit sphere, and then a new second panoramic image is obtained by writing the color RGB information of at least one pixel of the first panoramic image at its original position to the new position.

[0086] A Poisson surface reconstruction algorithm is used to generate a mesh model of the GlobalCloud point cloud, which is then combined with the second panoramic image to generate a map of at least one mesh of the mesh model, thereby completing the model reconstruction of the scene and generating a scene space model.

[0087] The method for constructing a scene space model in the above embodiment can effectively reduce the distortion phenomenon of the scene space model, improve the shooting efficiency, reduce the requirements for shooting equipment, improve the modeling efficiency and accuracy of the scene space model, provide users with a better virtual display exhibition experience, and improve the user experience.

[0088] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by instructing relevant hardware with a program, and the program may be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are executed. The storage medium includes various media that can store program code, such as a ROM, a RAM, a magnetic disk, or an optical disk.

[0089] 5 is a structural diagram of an embodiment of the apparatus of the present disclosure. The apparatus of this embodiment can be used to realize at least one embodiment of the method of the present disclosure. As shown in FIG. 5, the present disclosure provides an apparatus for constructing a scene space model. The apparatus includes a first information acquisition module 51, a second information acquisition module 52, a rotation matrix determination module 53, a panoramic image mapping module 54, a panoramic image transformation module 55, and a scene model generation module 56.

[0090] The first information acquisition module 51 acquires first point cloud information corresponding to a panorama of a target scene collected by a mobile point cloud collection device. The second information acquisition module 52 acquires depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, where the depth image information includes second point cloud information and color information of the corresponding image.

[0091] The rotation matrix determination module 53 determines a rotation matrix of the camera coordinate system of the fixed-point depth imaging device corresponding to the global coordinate system based on the first point cloud information and the second point cloud information, where the global coordinate system is the coordinate system corresponding to the first point cloud information. The panoramic image mapping module 54 generates a first panoramic image based on the depth image information and maps the first panoramic image to a three-dimensional unit sphere. The panoramic image transformation module 55 rotates the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image. The scene model generation module 56 generates a scene space model based on the first point cloud information and the second panoramic image.

[0092] In some alternative embodiments, the rotation matrix determination module 53 splices second point cloud information collected by at least one fixed-point depth camera to generate third point cloud information corresponding to a panorama of the target scene. The rotation matrix determination module 53 determines a first fixed-point rotation matrix of the fixed-point depth camera corresponding to the third point cloud information. The rotation matrix determination module 53 determines a second fixed-point rotation matrix between the first point cloud information and the second depth camera, and determines the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

[0093] For example, the rotation matrix determination module 53 matches the first point cloud information and the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information, where the point cloud matching algorithm includes, but is not limited to, an iterative closest point algorithm.

[0094] The rotation matrix determination module 53 uses the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device. The rotation matrix determination module 53 uses a point cloud matching algorithm to calculate the rotation matrix using the third fixed-point rotation matrix as an initial value.

[0095] In some alternative embodiments, the panoramic image mapping module 54 maps the first panoramic image onto a three-dimensional unit sphere by transforming at least one pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates. The panoramic image transformation module 55 rotates the three-dimensional unit sphere based on a rotation matrix to obtain new three-dimensional coordinates for the at least one pixel of the first panoramic image. The panoramic image transformation module 55 generates a second panoramic image based on the new three-dimensional coordinates and color information of the at least one pixel.

[0096] For example, the panoramic image transformation module 55 determines a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates, and adds color information of the at least one pixel of the first panoramic image to the new position to generate a second panoramic image.

[0097] The scene model generation module 56 performs surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information. The scene model generation module 56 generates a mesh map based on the mesh position information of the mesh model and the second panoramic image, and assigns the map to the corresponding mesh to generate a three-dimensional space model of the scene.

[0098] The business processing device in the above embodiment can effectively solve the problem that the existing scene space model construction method easily introduces accumulated errors, causing serious distortion in the scene space model, improves the shooting efficiency, reduces the requirements for shooting equipment, improves the modeling efficiency and accuracy of the scene space model, provides users with a better VR exhibition experience, and improves the user experience.

[0099] 6 is a structural diagram of an embodiment of an electronic device of the present disclosure. As shown in FIG. 6, an electronic device 61 includes one or more processors 611 and a memory 612.

[0100] The processor 611 may be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and may control other components within the electronic device 61 to perform desired functions.

[0101] The memory 612 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, and flash memory. One or more computer program instructions may be stored in the computer-readable storage medium, and the processor 611 may execute the program instructions to implement the method for constructing a scene space model in each embodiment of the present disclosure and / or other desired functions. The computer-readable storage medium may also store various contents, such as an input signal, a signal component, and a noise component.

[0102] In one example, the electronic device 61 may further include an input device 613 and an output device 614, and these components may be connected to each other via a bus system and / or other form of connection mechanism (not shown). The input device 613 may further include, for example, a keyboard, a mouse, etc. The output device 614 may output various information to the outside. The output device 614 may include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0103] Of course, for simplicity, only some of the components in the electronic device 61 that are relevant to the present disclosure are shown in Fig. 6, and components such as buses, input / output interfaces, etc. In addition, the electronic device 61 may further include any other appropriate components depending on the specific application.

[0104] In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method for constructing a scene space model according to various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0105] The computer program product may have program code written in any combination of one or more programming languages ​​for carrying out operations of embodiments of the present disclosure, including object-oriented programming languages ​​such as Java, C++, etc., and further including conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may execute entirely on a user's computing device, partially on a user's computing device, as a separate software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or a server.

[0106] Additionally, an embodiment of the present disclosure may be a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform steps of a method for constructing a scene space model according to various embodiments of the present disclosure, as described in the "Example Method" section above of this specification.

[0107] The computer-readable storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a readable signal medium or a readable storage medium. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0108] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, it should be noted that the benefits, advantages, and effects mentioned in the present disclosure are merely examples and are not limited thereto, and these benefits, advantages, and effects are not considered to be necessary for each embodiment of the present disclosure. Furthermore, the specific details of the above disclosure are merely for the purpose of illustrating examples and facilitating understanding, but are not limited thereto, and the above details do not limit the present disclosure to be realized using the above specific details.

[0109] The method, device, electronic device, and storage medium for constructing a scene space model in the above embodiments can generate a scene space model by integrating the features of a fixed-point depth photography device and a mobile point cloud collection device, thereby reducing the splicing error of different points during the photography process, and effectively solving the problem that accumulated errors are easily introduced in existing scene space model construction methods, resulting in serious distortion of the scene space model, thereby improving photography efficiency, reducing the requirements for photography equipment, and improving the modeling efficiency and accuracy of the scene space model, thereby providing users with a better VR exhibition experience and improving the user experience.

[0110] At least one embodiment in this specification will be described step by step, and the main content described in each embodiment is the difference from other embodiments, and similar parts between the embodiments can be referred to each other. Since the system embodiment basically corresponds to the method embodiment, it will be described briefly, and for related content, it is only necessary to refer to a part of the description of the method embodiment.

[0111] The methods and apparatuses of the present disclosure can be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise specified. In some embodiments, the present disclosure can also be implemented as a program recorded on a recording medium, which program includes machine-readable instructions for implementing the methods of the present disclosure. Thus, the present disclosure also includes a storage medium having a program stored thereon for executing the methods of the present disclosure.

[0112] The description of the present disclosure has been provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described in order to better explain the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and to design various embodiments with various modifications suitable for particular applications.

Claims

1. 1. A method for constructing a scene space model, comprising: acquiring first point cloud information corresponding to a target scene collected by a mobile point cloud collection device; acquiring depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, the depth image information including second point cloud information and color information of the corresponding image; determining a rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, wherein the global coordinate system is a coordinate system corresponding to the first point cloud information; generating a first panoramic image based on the depth image information and mapping the first panoramic image to a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; generating a scene space model based on the first point cloud information and the second panoramic image.

2. determining a rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, splicing the second point cloud information collected by each fixed-point depth imaging device to generate a third point cloud information corresponding to a panorama of the target scene; determining a first fixed-point rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to the third point cloud information; determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.

3. The step of determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information includes: and matching the first point cloud information with the third point cloud information based on a preset point cloud matching algorithm to obtain a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; The method of claim 2 , wherein the point cloud matching algorithm includes an Iterative Closest Points (ICP) algorithm.

4. determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix, using the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as a third fixed-point rotation matrix corresponding to the fixed-point depth imaging device; 4. The method of claim 3, further comprising: calculating the rotation matrix using the point cloud matching algorithm and using the third fixed-point rotation matrix as an initial value.

5. The step of mapping the first panoramic image to a three-dimensional unit sphere comprises:

2. The method of claim 1, further comprising: mapping the first panoramic image onto a three-dimensional unit sphere by transforming each pixel of the first panoramic image from two-dimensional coordinates to three-dimensional coordinates.

6. the step of rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image comprises: rotating the three-dimensional unit sphere based on the rotation matrix to obtain new three-dimensional coordinates for each pixel of the first panoramic image; and generating a second panoramic image based on the new three-dimensional coordinates and color information for each pixel.

7. generating a second panoramic image based on the new three-dimensional coordinates and color information of each pixel, determining a new position of each pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates; and adding color information of each pixel of the first panoramic image to the new location to generate the second panoramic image.

8. The step of generating a scene space model based on the first point cloud information and the second panoramic image includes: performing a surface reconstruction process on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information; generating a map of the mesh based on position information of the mesh of the mesh model and the second panoramic image; and applying the map to the corresponding mesh to generate a three-dimensional model of the scene space model.

9. 1. An apparatus for constructing a scene space model, comprising: a first information acquisition module for acquiring first point cloud information corresponding to the target scene, the first point cloud information being collected by a mobile point cloud collection device; a second information acquisition module used to acquire depth image information corresponding to a partial region of the target scene collected by a fixed-point depth photography device, the depth image information including second point cloud information and color information of the corresponding image; a rotation matrix determination module, which is used to determine a rotation matrix of a camera coordinate system of the fixed-point depth imaging device corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, wherein the global coordinate system is a coordinate system corresponding to the first point cloud information; a panoramic image mapping module for generating a first panoramic image not based on the depth image information and for mapping the first panoramic image onto a three-dimensional unit sphere; a panoramic image transformation module for rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; a scene model generation module for generating a scene space model based on the first point cloud information and the second panoramic image.

10. A computer-readable storage medium storing a computer program, A computer-readable storage medium, characterized in that the computer program implements the method according to any one of claims 1 to 8 when the computer program is executed.

Images