Three-dimensional model construction method, device and computer-readable storage medium

Abstract

The present invention provides a method, device and computer-readable storage medium for constructing a three-dimensional model. The method for constructing a three-dimensional model provided by the present invention comprises: obtaining N panoramic images of the same space, wherein the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, and N is an integer greater than 1; generating a three-dimensional model using each panoramic image to obtain N three-dimensional models; aligning the N three-dimensional models using a target object in the space; detecting isomorphic voxels and heterogeneous voxels in the aligned N three-dimensional models; correcting the three-dimensional coordinates and color values ​​of the heterogeneous voxels, and enhancing the color values ​​of the isomorphic voxels. The technical solution of the present invention can improve the quality of the constructed three-dimensional model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional model construction method, device and computer-readable storage medium. Background Art

[0002] With the rapid development of display technology and image processing technology, the demand for displaying three-dimensional space images of scenes (three-dimensional space) using display devices is growing. A three-dimensional model construction method of a three-dimensional space can be used to obtain a three-dimensional model of the three-dimensional space based on a two-dimensional panoramic image of the three-dimensional space, and in the display stage, a three-dimensional space image of the scene is rendered based on the obtained three-dimensional model of the three-dimensional space.

[0003] In the related art, when restoring a three-dimensional model from a panoramic image, voxels closer to the panoramic camera often have higher visual quality. Conversely, the farther away from the camera, the worse the visual quality of the voxels. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a three-dimensional model construction method, device and computer-readable storage medium, which can improve the quality of the constructed three-dimensional model.

[0005] According to one aspect of an embodiment of the present invention, a three-dimensional model construction method is provided, comprising:

[0006] Acquire N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0007] Generate a 3D model using each panoramic image to obtain N 3D models;

[0008] Aligning the N three-dimensional models using the target object in the space;

[0009] Detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0010] The three-dimensional coordinates and color values ​​of the isomorphic voxels are corrected, and the color values ​​of the isomorphic voxels are enhanced.

[0011] In addition, according to at least one embodiment of the present invention, generating a three-dimensional model using each panoramic image includes:

[0012] Detecting two-dimensional coordinates of corner points of the wall in the panoramic image, and calculating three-dimensional coordinates of the corner points of the wall using the two-dimensional coordinates;

[0013] Determine the three-dimensional coordinates of a panoramic camera that shoots the panoramic image using the three-dimensional coordinates of the corner points of the wall, wherein the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model;

[0014] The three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system are calculated using the three-dimensional coordinates of the corner points of the wall, and the color value of each voxel on the three-dimensional model is determined using the panoramic image.

[0015] In addition, according to at least one embodiment of the present invention, aligning the N three-dimensional models using the target object in the space includes:

[0016] Detecting the same target object in the N panoramic images;

[0017] Calculating the two-dimensional coordinates of the target object in each panoramic image;

[0018] Calculating the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, selecting a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, and calculating the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates;

[0019] The three-dimensional coordinates of the voxels of the other three-dimensional model are calculated based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and the other three-dimensional model is aligned with the reference three-dimensional model.

[0020] In addition, according to at least one embodiment of the present invention, detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models includes:

[0021] Selecting a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determining a second voxel corresponding to the first voxel in other three-dimensional models, wherein the first voxel and the second voxel are the same physical point in the space;

[0022] The three-dimensional distance between the first voxel and each second voxel is calculated respectively. If any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; if all the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels.

[0023] In addition, according to at least one embodiment of the present invention, the preset threshold is 5 voxels.

[0024] Furthermore, according to at least one embodiment of the present invention, the three-dimensional distance between at least part of the first voxels of the reference model and the coordinate origin of the benchmark three-dimensional model is smaller than the three-dimensional distance between the corresponding second voxels and the coordinate origin of the benchmark three-dimensional model.

[0025] In addition, according to at least one embodiment of the present invention, the correcting the three-dimensional coordinates and color values ​​of the isomorphic voxels and enhancing the color values ​​of the isomorphic voxels comprises:

[0026] The first voxel includes a first isomeric voxel that is an isomeric voxel, calculating a three-dimensional distance between a second voxel corresponding to the first isomeric voxel and a panoramic camera corresponding to the reference model, and replacing the three-dimensional coordinates and color value of the first isomeric voxel with the three-dimensional coordinates and color value of the second voxel having the largest three-dimensional distance;

[0027] The first voxel includes a first isomorphous voxel belonging to isomorphous voxels, a three-dimensional distance between the first isomorphous voxel and all panoramic cameras is calculated, and a color value of the first isomorphous voxel is determined by using the three-dimensional distance between the first isomorphous voxel and all panoramic cameras.

[0028] In addition, according to at least one embodiment of the present invention, N=2, the N panoramic images are respectively taken by panoramic camera A and panoramic camera B, the 3D model restored by the panoramic image taken by panoramic camera A is used as a reference model, and the color value C of the first isomorphic voxel in the reference model is determined by the following formula:

[0029]

[0030] where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C B is the color value of the second voxel in the 3D model restored from the panoramic image captured by the panoramic camera B.

[0031] According to another aspect of an embodiment of the present invention, there is provided a three-dimensional model construction device, comprising:

[0032] An acquisition unit, configured to acquire N panoramic images of the same space, wherein the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0033] A 3D model generating unit, used to generate a 3D model using each panoramic image, to obtain N 3D models;

[0034] an alignment unit, configured to align the N three-dimensional models using a target object in the space;

[0035] A detection unit, used for detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0036] The processing unit is used to correct the three-dimensional coordinates and color values ​​of the heterogeneous voxels and enhance the color values ​​of the homogeneous voxels.

[0037] In addition, according to at least one embodiment of the present invention, the three-dimensional model generation unit is specifically used to detect the two-dimensional coordinates of the wall corner points in the panoramic image, and calculate the three-dimensional coordinates of the wall corner points using the two-dimensional coordinates; determine the three-dimensional coordinates of the panoramic camera that took the panoramic image using the three-dimensional coordinates of the wall corner points, and the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model; calculate the three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system using the three-dimensional coordinates of the wall corner points, and determine the color value of each voxel on the three-dimensional model using the panoramic image.

[0038] In addition, according to at least one embodiment of the present invention, the alignment unit is specifically used to detect the same target object in the N panoramic images; calculate the two-dimensional coordinates of the target object in each panoramic image; use the two-dimensional coordinates of the target object to calculate the three-dimensional coordinates of the target object in the three-dimensional model, select a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model are the reference three-dimensional coordinates, calculate the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates; calculate the three-dimensional coordinates of voxels of the other three-dimensional models based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and align the other three-dimensional models with the reference three-dimensional model.

[0039] In addition, according to at least one embodiment of the present invention, the detection unit is specifically used to select a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determine a second voxel corresponding to the first voxel in other three-dimensional models, and the first voxel and the second voxel are the same physical point in the space; calculate the three-dimensional distance between the first voxel and each second voxel respectively, if any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are heterogeneous voxels; if all of the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are homogeneous voxels.

[0040] Furthermore, according to at least one embodiment of the present invention, the first voxel includes a first isomorphic voxel that is an isomorphic voxel, and the first voxel includes a first isomorphic voxel that is an isomorphic voxel;

[0041] The processing unit is specifically used to calculate the three-dimensional distance between the second voxel corresponding to the first heterogeneous voxel and the panoramic camera corresponding to the reference model, and replace the three-dimensional coordinates and color value of the first heterogeneous voxel with the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance; calculate the three-dimensional distance between the first homogeneous voxel and all panoramic cameras, and determine the color value of the first homogeneous voxel using the three-dimensional distance between the first homogeneous voxel and all panoramic cameras.

[0042] An embodiment of the present invention also provides a three-dimensional model construction device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the three-dimensional model construction method described above when executed by the processor.

[0043] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the three-dimensional model construction method described above are implemented.

[0044] Compared with the prior art, the three-dimensional model construction method, device and computer-readable storage medium provided in the embodiments of the present invention obtain multiple panoramic images of the same space, generate a three-dimensional model using the multiple panoramic images, align the multiple three-dimensional models, detect isomorphic voxels and heterogeneous voxels in the aligned three-dimensional model, correct the three-dimensional coordinates and color values ​​of the heterogeneous voxels, and enhance the color values ​​of the homogeneous voxels, so as to obtain a three-dimensional model with high visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 A schematic diagram of restoring a three-dimensional model using a panoramic image;

[0047] Figure 2 A schematic diagram of a process of constructing a three-dimensional model according to an embodiment of the present invention;

[0048] Figure 3The two panoramic images are obtained by photographing the same space using panoramic cameras set at two different positions in the embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a process of generating a three-dimensional model using a panoramic image according to an embodiment of the present invention;

[0050] Figure 5 For the embodiment of the present invention, Figure 3 The two panoramic images shown are restored into a schematic diagram of a three-dimensional model;

[0051] Figure 6 A schematic diagram of a process for aligning a three-dimensional model according to an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of determining a target object according to an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of two three-dimensional models aligned according to an embodiment of the present invention;

[0054] Fig. 9 A schematic diagram of a process for detecting isomeric voxels and isomeric voxels according to an embodiment of the present invention;

[0055] Fig.10 It is a schematic diagram of the effect of correcting the three-dimensional coordinates and color values ​​of isomeric voxels according to an embodiment of the present invention;

[0056] Fig.11 It is a schematic diagram of the effect of correcting the three-dimensional coordinates and color values ​​of isomorphic voxels and isomorphic voxels according to an embodiment of the present invention;

[0057] Fig.12 A schematic diagram of the structure of a three-dimensional model building device according to an embodiment of the present invention;

[0058] Fig.13 This is another structural schematic diagram of the three-dimensional model building device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present invention. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for clarity and brevity, the description of known functions and structures is omitted.

[0060] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0061] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0062] When using a panoramic image to restore a 3D model, voxels closer to the panoramic camera that took the panoramic image tend to have higher visual quality. On the contrary, the farther away from the panoramic camera, the worse the visual quality of the voxels. There are two main reasons: First, the voxels are closer to the panoramic camera. Figure 1 For example, the right picture in the figure is taken by panoramic camera B. The distance between K2 area and panoramic camera B is far, and the visual quality of voxels in K2 area is poor. This is because the voxels in K2 area are often generated by interpolation. Figure 1 For example, the left picture in the figure is taken by panoramic camera A. Because the corner points of the wall far away from panoramic camera A are blocked, the panoramic image taken by panoramic camera A can only restore part of the 3D model, resulting in poor visual quality of the voxels in the blocked K1 area. In summary, the visual quality of the 3D model is poor.

[0063] Embodiments of the present invention provide a three-dimensional model construction method, device and computer-readable storage medium, which can improve the quality of the constructed three-dimensional model.

[0064] Embodiment 1

[0065] The embodiment of the present invention provides a three-dimensional model construction method, such as Figure 2 As shown, including:

[0066] Step 101: Acquire N panoramic images of the same space, wherein the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0067] In this embodiment, one or two or more panoramic cameras may be provided. If one panoramic camera is provided, the panoramic camera may be used to photograph the same space at different times and positions to obtain multiple panoramic images. If two or more panoramic cameras are provided, the two or more panoramic cameras are located at different positions of the same space, and the two or more panoramic cameras may be used to photograph multiple panoramic images of the space. The multiple panoramic images may improve the visual quality of the generated three-dimensional model of the space. The more panoramic cameras are provided, the higher the visual quality of the ultimately generated three-dimensional model will be, but at the same time, the cost and the complexity of the calculation will increase. In some embodiments, the value of N may be 2, and the panoramic cameras provided at two different positions may be used to photograph the following three-dimensional model: Figure 3 The two panoramic images shown in FIG. 1 correspond to the same space; or a panoramic camera can be used to capture images at different positions as shown in FIG. Figure 3 The two panoramic images shown correspond to the same space.

[0068] Step 102: Generate a 3D model using each panoramic image to obtain N 3D models;

[0069] like Figure 4 As shown, step 102 includes the following steps:

[0070] Step 1021: Detecting the two-dimensional coordinates of the corner points of the wall in the panoramic image, and calculating the three-dimensional coordinates of the corner points of the wall using the two-dimensional coordinates;

[0071] Specifically, the deep learning algorithm can be used to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Of course, this embodiment is not limited to the use of the deep learning algorithm to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Other algorithms, deep learning algorithms, can also be used to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Then, the two-dimensional coordinates of the corner points of the wall are used to calculate the three-dimensional coordinates of the corner points of the wall. Figure 5 As shown, the wall corner point is the intersection of three adjacent walls in space. The coordinates of the wall corner point can be used to determine the three-dimensional coordinates of each physical point in space.

[0072] Step 1022: using the three-dimensional coordinates of the corner points of the wall, determining the three-dimensional coordinates of a panoramic camera that shoots the panoramic image, wherein the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model;

[0073] like Figure 5As shown, the three-dimensional coordinates of the panoramic camera that took the panoramic image are determined using the three-dimensional coordinates of the corner points of the wall, that is, the three-dimensional coordinates of point O in the figure. Point O is the origin of the three-dimensional coordinate system of the three-dimensional model restored using the panoramic image. In the three-dimensional coordinate system, the coordinates of O are (0,0,0). The three-dimensional coordinate system includes an X-axis, a Y-axis, and a Z-axis. The three-dimensional coordinates of any voxel in the three-dimensional model are (x, y, z).

[0074] Step 1023: Calculate the three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system using the three-dimensional coordinates of the corner points of the wall, and determine the color value of each voxel on the three-dimensional model using the panoramic image.

[0075] Voxel is the abbreviation of volume pixel. A volume containing voxels can be expressed by stereo rendering or extracting polygonal isosurfaces with given threshold contours. Voxel is the smallest unit of digital data in three-dimensional space segmentation and is used in three-dimensional imaging, scientific data, medical imaging and other fields.

[0076] After the above steps 1021-1023, a three-dimensional model corresponding to the panoramic image can be obtained. Each panoramic image corresponds to a three-dimensional model and the three-dimensional coordinates of the panoramic camera. In the three-dimensional model, the origin of the three-dimensional coordinate system is the three-dimensional coordinates of the corresponding panoramic camera.

[0077] Will Figure 3 After the two panoramic images shown in are restored to a three-dimensional model, we can get Figure 5 The two three-dimensional models shown in Figure 3 The panoramic image on the left corresponds to Figure 5 The 3D model on the left. Figure 3 The panoramic image on the right corresponds to Figure 5 The 3D model on the right. Figure 5 The two three-dimensional models shown correspond to the same space.

[0078] Step 103: aligning the N three-dimensional models using the target object in the space;

[0079] like Figure 6 As shown, step 103 includes the following steps:

[0080] Step 1031: Detecting the same target object in the N panoramic images;

[0081] Objects with significant features in the space can be used as target objects, such as doors in the space.

[0082] Step 1032: Calculate the two-dimensional coordinates of the target object in each panoramic image;

[0083] In a specific example, Figure 3 The two-dimensional coordinates of the target object are calculated respectively in the two panoramic images shown.

[0084] Step 1033: Calculate the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, select a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, use the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, and calculate the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates;

[0085] like Figure 7 As shown, the center point of one of the doors is taken as the target object, the three-dimensional coordinates of the target object in the three-dimensional model on the left are (X1, Y1, Z1), and the three-dimensional coordinates of the target object in the three-dimensional model on the right are (X1', Y1', Z1'). For example, the three-dimensional model on the right can be taken as the reference three-dimensional model, and the three-dimensional displacement d1 of the target object in the three-dimensional model on the left relative to the target object in the three-dimensional model on the right is calculated by the three-dimensional coordinates (X1, Y1, Z1) and (X1', Y1', Z1'); or the center point of another door is taken as the target object, the three-dimensional coordinates of the target object in the three-dimensional model on the left are (X2, Y2, Z2), and the three-dimensional coordinates of the target object in the three-dimensional model on the right are (X2', Y2', Z2'). For example, the three-dimensional model on the right can be taken as the reference three-dimensional model, and the three-dimensional displacement d2 of the target object in the three-dimensional model on the left relative to the target object in the three-dimensional model on the right is calculated by the three-dimensional coordinates (X2, Y2, Z2) and (X2', Y2', Z2').

[0086] Step 1034: Calculate the three-dimensional coordinates of the voxels of the other three-dimensional model based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and align the other three-dimensional model with the reference three-dimensional model.

[0087] The N aligned 3D models are aligned to the same 3D coordinate system, and the 3D coordinates of any voxel in the 3D model in the 3D coordinate system are (Xa, Ya, Za) or (Xb, Yb, Zb). Figure 7 The two 3D models shown are aligned as Figure 8 As shown, the coordinate origins of the two three-dimensional models are both the coordinate origins of the reference three-dimensional model.

[0088] Step 104: Detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0089] like Fig. 9 As shown, step 104 includes the following steps:

[0090] Step 1041: selecting a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determining a second voxel corresponding to the first voxel in other three-dimensional models, wherein the first voxel and the second voxel are the same physical point in the space;

[0091] Among them, the reference model can be the same three-dimensional model as the benchmark three-dimensional model, or a different three-dimensional model. In order to facilitate distinction, the voxel in the reference model is defined as a first voxel. For each first voxel of the reference model, a second voxel corresponding to the first voxel is determined in other three-dimensional models (i.e., a three-dimensional model other than the reference model). The first voxel and the corresponding second voxel are the same physical point in the space, that is, they are the same point in the physical world.

[0092] When selecting a reference model, the following criterion may be used: the three-dimensional distance between at least part of the first voxels of the reference model and the coordinate origin of the reference three-dimensional model is less than the three-dimensional distance between the corresponding second voxels and the coordinate origin of the reference three-dimensional model. A three-dimensional model that meets the above criterion is selected as a reference model, so that other three-dimensional models can have a stronger sense of space than the reference model, and then the voxels of the reference model can be used to correct the voxels of the reference model to obtain the final three-dimensional model.

[0093] Step 1042: Calculate the three-dimensional distance between the first voxel and each second voxel respectively. If any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are heterogeneous voxels; if all of the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are homogeneous voxels.

[0094] In some embodiments, the preset threshold is 5 voxels. Of course, the value of the preset threshold is not limited to 5 voxels, and can also be set to other values ​​as needed.

[0095] Step 105: Correct the three-dimensional coordinates and color values ​​of the isomorphic voxels, and enhance the color values ​​of the isomorphic voxels.

[0096] In this embodiment, different operations are performed on heterogeneous voxels and homogeneous voxels. The area where the heterogeneous voxels are located is a region where other three-dimensional models are quite different from the reference model, and is also a region with relatively poor visual quality. Therefore, it is necessary to correct the three-dimensional coordinates and color values ​​of the voxels at the same time; the area where the homogeneous voxels are located is a region where other three-dimensional models are quite different from the reference model, and is also a region with relatively good visual quality. Only the color value needs to be enhanced.

[0097] In some embodiments, the first voxel includes a first isomeric voxel that is an isomeric voxel, and the three-dimensional distance between the second voxel corresponding to the first isomeric voxel and the panoramic camera corresponding to the reference model is calculated, and the three-dimensional coordinates and color value of the first isomeric voxel are replaced by the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance. If there are only two three-dimensional models, the three-dimensional coordinates and color value of the second voxel of the three-dimensional model other than the reference model are replaced by the three-dimensional coordinates and color value of the first isomeric voxel; if there are more than three three-dimensional models, the three-dimensional distances of the other three-dimensional models other than the reference model are calculated respectively, the largest three-dimensional distance is selected, and the three-dimensional coordinates and color value of the first isomeric voxel are replaced by the three-dimensional coordinates and color value of the second voxel corresponding to the largest three-dimensional distance. Fig.10 As shown in Figure 2, after the three-dimensional coordinates and color values ​​of the isomeric voxels are corrected, Fig.10 The portion of the 3D model shown on the left is converted to Fig.10 From the 3D model shown on the right, it can be seen that the visual quality of the 3D model is greatly improved.

[0098] For the first isomeric voxel, the corresponding second voxel has a larger three-dimensional distance from the panoramic camera corresponding to the reference model, that is, the three-dimensional model to which the second voxel belongs has a stronger spatial stereoscopic sense in this area (i.e., the area where the first isomeric voxel is located), and can better reflect the concave and convex changes in the space of this area compared to the reference model. Therefore, replacing the three-dimensional coordinates and color values ​​of the first isomeric voxel with the three-dimensional coordinates and color values ​​of the second voxel can improve the visual quality of the three-dimensional model in this area.

[0099] In some embodiments, the first voxel includes a first isomorphous voxel belonging to isomorphous voxels, a three-dimensional distance between the first isomorphous voxel and all panoramic cameras is calculated, and a color value of the first isomorphous voxel is determined using the three-dimensional distance between the first isomorphous voxel and all panoramic cameras.

[0100] In a specific example, the N panoramic images are taken by panoramic camera A and panoramic camera B respectively, and the three-dimensional model restored from the panoramic image taken by panoramic camera A is used as a reference model. The color value C of the first isomorphic voxel in the reference model is determined by the following formula:

[0101]

[0102] where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C Bis the color value of the second voxel in the 3D model restored from the panoramic image taken by the panoramic camera B.

[0103] like Fig.11 As shown in Figure 2, after the three-dimensional coordinates and color values ​​of the isomorphic voxels and the isomorphic voxels are corrected, Fig.11 The two 3D models shown on the left are converted to Fig.11 The 3D model shown on the right. Fig.11 As shown, the area with poor visual quality of the 3D model in the upper left corner can be replaced by the voxels of the corresponding area of ​​the 3D model in the lower left corner, thereby obtaining Fig.11 The three-dimensional model shown on the right shows that Fig.11 The two 3D models shown on the left, Fig.11 The visual quality of the 3D model shown on the right is greatly improved.

[0104] In this embodiment, multiple panoramic images of the same space are acquired, a three-dimensional model is generated using the multiple panoramic images, the multiple three-dimensional models are aligned, isomorphic voxels and heterogeneous voxels are detected in the aligned three-dimensional model, the three-dimensional coordinates and color values ​​of the heterogeneous voxels are corrected, and the color values ​​of the isomorphic voxels are enhanced, so that a three-dimensional model with high visual quality can be obtained.

[0105] Embodiment 2

[0106] The embodiment of the present invention also provides a three-dimensional model construction device, such as Fig.12 As shown, including:

[0107] An acquisition unit 21 is used to acquire N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0108] In this embodiment, one or two or more panoramic cameras may be provided. If one panoramic camera is provided, the panoramic camera may be used to photograph the same space at different times and positions to obtain multiple panoramic images. If two or more panoramic cameras are provided, the two or more panoramic cameras are located at different positions of the same space, and the two or more panoramic cameras may be used to photograph multiple panoramic images of the space. The multiple panoramic images may improve the visual quality of the generated three-dimensional model of the space. The more panoramic cameras are provided, the higher the visual quality of the ultimately generated three-dimensional model will be, but at the same time, the cost and the complexity of the calculation will increase. In some embodiments, the value of N may be 2, and the panoramic cameras provided at two different positions may be used to photograph the following three-dimensional model: Figure 3 The two panoramic images shown in FIG. 1 correspond to the same space; or a panoramic camera can be used to capture the two panoramic images at different positions. Figure 3The two panoramic images shown correspond to the same space.

[0109] A three-dimensional model generating unit 22, configured to generate a three-dimensional model using each panoramic image, to obtain N three-dimensional models;

[0110] The three-dimensional model generation unit 22 is specifically used to detect the two-dimensional coordinates of the wall corner points in the panoramic image, and calculate the three-dimensional coordinates of the wall corner points using the two-dimensional coordinates; determine the three-dimensional coordinates of the panoramic camera that took the panoramic image using the three-dimensional coordinates of the wall corner points, and the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model; calculate the three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system using the three-dimensional coordinates of the wall corner points, and determine the color value of each voxel on the three-dimensional model using the panoramic image.

[0111] Specifically, the deep learning algorithm can be used to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Of course, this embodiment is not limited to the use of the deep learning algorithm to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Other algorithms, deep learning algorithms, can also be used to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image. Then, the two-dimensional coordinates of the corner points of the wall are used to calculate the three-dimensional coordinates of the corner points of the wall. Figure 5 As shown in , the corner point of the wall is the intersection of three adjacent walls in space. The coordinates of the corner point of the wall can be used to determine the three-dimensional coordinates of each physical point in space. Figure 5 As shown, the three-dimensional coordinates of the panoramic camera that took the panoramic image are determined using the three-dimensional coordinates of the corner points of the wall, that is, the three-dimensional coordinates of point O in the figure. Point O is the origin of the three-dimensional coordinate system of the three-dimensional model restored using the panoramic image. In the three-dimensional coordinate system, the coordinates of O are (0,0,0). The three-dimensional coordinate system includes an X-axis, a Y-axis, and a Z-axis. The three-dimensional coordinates of any voxel in the three-dimensional model are (x, y, z).

[0112] After the 3D model generation unit 22 generates the 3D model, each panoramic image corresponds to a 3D model and the 3D coordinates of the panoramic camera. In the 3D model, the origin of the 3D coordinate system is the 3D coordinates of the corresponding panoramic camera.

[0113] An alignment unit 23, configured to align the N three-dimensional models using the target object in the space;

[0114] The alignment unit 23 is specifically used to detect the same target object in the N panoramic images; calculate the two-dimensional coordinates of the target object in each panoramic image; use the two-dimensional coordinates of the target object to calculate the three-dimensional coordinates of the target object in the three-dimensional model, select a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model are the reference three-dimensional coordinates, calculate the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates; calculate the three-dimensional coordinates of the voxels of the other three-dimensional models based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and align the other three-dimensional models with the reference three-dimensional model.

[0115] Specifically, an object with significant features in the space can be used as a target object, such as a door in the space. Figure 7 As shown, the center point of one of the doors is taken as the target object, the three-dimensional coordinates of the target object in the three-dimensional model on the left are (X1, Y1, Z1), and the three-dimensional coordinates of the target object in the three-dimensional model on the right are (X1', Y1', Z1'). For example, the three-dimensional model on the right can be taken as the reference three-dimensional model, and the three-dimensional displacement d1 of the target object in the three-dimensional model on the left relative to the target object in the three-dimensional model on the right is calculated by the three-dimensional coordinates (X1, Y1, Z1) and (X1', Y1', Z1'); or the center point of another door is taken as the target object, the three-dimensional coordinates of the target object in the three-dimensional model on the left are (X2, Y2, Z2), and the three-dimensional coordinates of the target object in the three-dimensional model on the right are (X2', Y2', Z2'). For example, the three-dimensional model on the right can be taken as the reference three-dimensional model, and the three-dimensional displacement d2 of the target object in the three-dimensional model on the left relative to the target object in the three-dimensional model on the right is calculated by the three-dimensional coordinates (X2, Y2, Z2) and (X2', Y2', Z2'). The three-dimensional coordinates of the voxels of the other three-dimensional models are calculated based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and the other three-dimensional models are aligned with the reference three-dimensional model. The aligned N three-dimensional models are aligned to the same three-dimensional coordinate system, and the three-dimensional coordinates of any voxel in the three-dimensional model in the three-dimensional coordinate system are (Xa, Ya, Za). In a specific example, Figure 7 The two 3D models shown are aligned as Figure 8 As shown, the coordinate origins of the two three-dimensional models are both the coordinate origins of the reference three-dimensional model.

[0116] A detection unit 24, configured to detect isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0117] The detection unit 24 is specifically used to select a reference model from the N aligned three-dimensional models, and for each first voxel of the reference model, determine a second voxel corresponding to the first voxel in other three-dimensional models, and the first voxel and the second voxel are the same physical point in the space; calculate the three-dimensional distance between the first voxel and each second voxel respectively, if any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are heterogeneous voxels; if all the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are homogeneous voxels.

[0118] Specifically, the reference model can be the same three-dimensional model as the benchmark three-dimensional model, or a different three-dimensional model. To facilitate distinction, the voxel in the reference model is defined as a first voxel. For each first voxel of the reference model, a second voxel corresponding to the first voxel is determined in other three-dimensional models (i.e., a three-dimensional model other than the reference model). The first voxel and the corresponding second voxel are the same physical point in the space, that is, they are the same point in the physical world.

[0119] When selecting a reference model, the following criterion may be used: the three-dimensional distance between at least part of the first voxels of the reference model and the coordinate origin of the reference three-dimensional model is less than the three-dimensional distance between the corresponding second voxels and the coordinate origin of the reference three-dimensional model. A three-dimensional model that meets the above criterion is selected as a reference model, so that other three-dimensional models can have a stronger sense of space than the reference model, and then the voxels of the reference model can be used to correct the voxels of the reference model to obtain the final three-dimensional model.

[0120] In some embodiments, the preset threshold is 5 voxels. Of course, the value of the preset threshold is not limited to 5 voxels, and can also be set to other values ​​as needed.

[0121] The processing unit 25 is used to correct the three-dimensional coordinates and color values ​​of the heterogeneous voxels and enhance the color values ​​of the homogeneous voxels.

[0122] The processing unit 25 is specifically used to calculate the three-dimensional distance between the second voxel corresponding to the first heterogeneous voxel and the panoramic camera corresponding to the reference model, and replace the three-dimensional coordinates and color value of the first heterogeneous voxel with the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance; calculate the three-dimensional distance between the first homogeneous voxel and all panoramic cameras, and determine the color value of the first homogeneous voxel using the three-dimensional distance between the first homogeneous voxel and all panoramic cameras.

[0123] In this embodiment, the processing unit 25 performs different operations on heterogeneous voxels and homogeneous voxels. The area where the heterogeneous voxels are located is the area where other three-dimensional models are quite different from the reference model, and is also the area with relatively poor visual quality. Therefore, it is necessary to correct the three-dimensional coordinates and color values ​​of the voxels at the same time; the area where the homogeneous voxels are located is the area where other three-dimensional models are slightly different from the reference model, and is also the area with relatively good visual quality. Only the color value needs to be enhanced.

[0124] In some embodiments, the first voxel includes a first isomeric voxel that is an isomeric voxel, and the processing unit 25 calculates the three-dimensional distance between the second voxel corresponding to the first isomeric voxel and the panoramic camera corresponding to the reference model, and replaces the three-dimensional coordinates and color value of the first isomeric voxel with the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance. If there are only two three-dimensional models, the three-dimensional coordinates and color value of the second voxel of the three-dimensional model other than the reference model are used to replace the three-dimensional coordinates and color value of the first isomeric voxel; if there are more than three three-dimensional models, the three-dimensional distances of the other three-dimensional models other than the reference model are calculated respectively, the largest three-dimensional distance is selected, and the three-dimensional coordinates and color value of the second voxel corresponding to the largest three-dimensional distance are used to replace the three-dimensional coordinates and color value of the first isomeric voxel. Fig.10 As shown in Figure 2, after the three-dimensional coordinates and color values ​​of the isomeric voxels are corrected, Fig.10 The portion of the 3D model shown on the left is converted to Fig.10 From the 3D model shown on the right, it can be seen that the visual quality of the 3D model is greatly improved.

[0125] For the first isomeric voxel, the corresponding second voxel has a larger three-dimensional distance from the panoramic camera corresponding to the reference model, that is, the three-dimensional model to which the second voxel belongs has a stronger spatial stereoscopic sense in this area (i.e., the area where the first isomeric voxel is located), and can better reflect the concave and convex changes in the space of this area compared to the reference model. Therefore, replacing the three-dimensional coordinates and color values ​​of the first isomeric voxel with the three-dimensional coordinates and color values ​​of the second voxel can improve the visual quality of the three-dimensional model in this area.

[0126] In some embodiments, the first voxel includes a first isomorphous voxel belonging to isomorphous voxels, and the processing unit 25 calculates the three-dimensional distance between the first isomorphous voxel and all panoramic cameras, and determines the color value of the first isomorphous voxel using the three-dimensional distance between the first isomorphous voxel and all panoramic cameras.

[0127] In a specific example, the N panoramic images are taken by panoramic camera A and panoramic camera B respectively, and the three-dimensional model restored from the panoramic image taken by panoramic camera A is used as a reference model. The color value C of the first isomorphic voxel in the reference model is determined by the following formula:

[0128]

[0129] where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C B is the color value of the second voxel in the 3D model restored from the panoramic image captured by the panoramic camera B.

[0130] like Fig.11 As shown in Figure 2, after the three-dimensional coordinates and color values ​​of the isomorphic voxels and the isomorphic voxels are corrected, Fig.11 The two 3D models shown on the left are converted to Fig.11 The 3D model shown on the right. Fig.11 As shown, the area with poor visual quality of the 3D model in the upper left corner can be replaced by the voxels of the corresponding area of ​​the 3D model in the lower left corner, thereby obtaining Fig.11 The three-dimensional model shown on the right shows that Fig.11 The two 3D models shown on the left, Fig.11 The visual quality of the 3D model shown on the right is greatly improved.

[0131] In this embodiment, multiple panoramic images of the same space are acquired, a three-dimensional model is generated using the multiple panoramic images, the multiple three-dimensional models are aligned, isomorphic voxels and heterogeneous voxels are detected in the aligned three-dimensional model, the three-dimensional coordinates and color values ​​of the heterogeneous voxels are corrected, and the color values ​​of the isomorphic voxels are enhanced, so that a three-dimensional model with high visual quality can be obtained.

[0132] Embodiment 3

[0133] The embodiment of the present invention also provides a three-dimensional model construction device 30, such as Fig.13 As shown, including:

[0134] processor 32; and

[0135] a memory 34 in which computer program instructions are stored,

[0136] When the computer program instructions are executed by the processor, the processor 32 is caused to perform the following steps:

[0137] Acquire N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0138] Generate a 3D model using each panoramic image to obtain N 3D models;

[0139] Aligning the N three-dimensional models using the target object in the space;

[0140] Detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0141] The three-dimensional coordinates and color values ​​of the isomorphic voxels are corrected, and the color values ​​of the isomorphic voxels are enhanced.

[0142] Furthermore, if Fig.13 As shown, the three-dimensional model building device 30 also includes a network interface 31 , an input device 33 , a hard disk 35 and a display device 36 .

[0143] The above-mentioned interfaces and devices can be interconnected through a bus architecture. The bus architecture can include any number of interconnected buses and bridges. Specifically, various circuits of one or more central processing units (CPUs) represented by processor 32 and one or more memories represented by memory 34 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art, so they are not described in detail herein.

[0144] The network interface 31 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, such as a panoramic image, etc., and save it in the hard disk 35.

[0145] The input device 33 can receive various instructions input by the operator and send them to the processor 32 for execution. The input device 33 can include a keyboard or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0146] The display device 36 can display the results obtained by the processor 32 executing the instructions.

[0147] The memory 34 is used to store programs and data necessary for the operation of the operating system, as well as data such as intermediate results during the calculation process of the processor 32.

[0148] It is understood that the memory 34 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. The memory 34 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0149] In some embodiments, the memory 34 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 341 and application programs 342 .

[0150] The operating system 341 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 342 includes various application programs, such as a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 342.

[0151] The processor 32, when calling and executing the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, obtains N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, and N is an integer greater than 1; generates a three-dimensional model using each panoramic image to obtain N three-dimensional models; aligns the N three-dimensional models using the target object in the space; detects isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models; corrects the three-dimensional coordinates and color values ​​of the heteromorphic voxels, and enhances the color values ​​of the isomorphic voxels.

[0152] Furthermore, when the processor 32 calls and executes the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, it detects the two-dimensional coordinates of the wall corner points in the panoramic image, and calculates the three-dimensional coordinates of the wall corner points using the two-dimensional coordinates; determines the three-dimensional coordinates of the panoramic camera that took the panoramic image using the three-dimensional coordinates of the wall corner points, and the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model; calculates the three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system using the three-dimensional coordinates of the wall corner points, and determines the color value of each voxel on the three-dimensional model using the panoramic image.

[0153] Furthermore, the processor 32, when calling and executing the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, detects the same target object in the N panoramic images; calculates the two-dimensional coordinates of the target object in each panoramic image; calculates the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, selects one of the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, calculates the three-dimensional displacement of the three-dimensional coordinates of the target object in the other three-dimensional models relative to the reference three-dimensional coordinates; calculates the three-dimensional coordinates of the voxels of the other three-dimensional models based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and aligns the other three-dimensional models with the reference three-dimensional model.

[0154] Furthermore, when the processor 32 calls and executes the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, it selects a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determines a second voxel corresponding to the first voxel in other three-dimensional models, and the first voxel and the second voxel are the same physical point in the space; calculates the three-dimensional distance between the first voxel and each second voxel respectively, and if any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are heterogeneous voxels; if all of the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are homogeneous voxels.

[0155] Further, the first voxel includes a first isomorphic voxel that is an isomorphic voxel, and the first voxel includes a first isomorphic voxel that is an isomorphic voxel. The processor 32, when calling and executing the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, calculates the three-dimensional distance between the second voxel corresponding to the first isomorphic voxel and the panoramic camera corresponding to the reference model, and replaces the three-dimensional coordinates and color value of the first isomorphic voxel with the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance respectively; calculates the three-dimensional distance between the first isomorphic voxel and all panoramic cameras, and determines the color value of the first isomorphic voxel using the three-dimensional distance between the first isomorphic voxel and all panoramic cameras.

[0156] Further, N=2, the N panoramic images are respectively taken by panoramic camera A and panoramic camera B, and the three-dimensional model restored by the panoramic image taken by panoramic camera A is used as the reference model. When the processor 32 calls and executes the application and data stored in the memory 34, specifically, the program or instruction stored in the application 342, the color value C of the first isomorphic voxel in the reference model is determined by the following formula:

[0157]

[0158] where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C B is the color value of the second voxel in the 3D model restored from the panoramic image taken by the panoramic camera B.

[0159] The method disclosed in the above embodiment of the present invention can be applied to the processor 32, or implemented by the processor 32. The processor 32 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 32 or the instruction in the form of software. The above processor 32 can be a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 34, and the processor 32 reads the information in the memory 34 and completes the steps of the above method in combination with its hardware.

[0160] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0161] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0162] Embodiment 4

[0163] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following steps:

[0164] Acquire N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1;

[0165] Generate a 3D model using each panoramic image to obtain N 3D models;

[0166] Aligning the N three-dimensional models using the target object in the space;

[0167] Detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models;

[0168] The three-dimensional coordinates and color values ​​of the isomorphic voxels are corrected, and the color values ​​of the isomorphic voxels are enhanced.

[0169] Furthermore, when the computer program is executed by a processor, the processor is caused to perform the following steps:

[0170] Detecting two-dimensional coordinates of corner points of the wall in the panoramic image, and calculating three-dimensional coordinates of the corner points of the wall using the two-dimensional coordinates;

[0171] Determine the three-dimensional coordinates of a panoramic camera that shoots the panoramic image using the three-dimensional coordinates of the corner points of the wall, wherein the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model;

[0172] The three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system are calculated using the three-dimensional coordinates of the corner points of the wall, and the color value of each voxel on the three-dimensional model is determined using the panoramic image.

[0173] Furthermore, when the computer program is executed by a processor, the processor is caused to perform the following steps:

[0174] Detecting the same target object in the N panoramic images;

[0175] Calculating the two-dimensional coordinates of the target object in each panoramic image;

[0176] Calculating the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, selecting a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, and calculating the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates;

[0177] The three-dimensional coordinates of the voxels of the other three-dimensional model are calculated based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and the other three-dimensional model is aligned with the reference three-dimensional model.

[0178] Furthermore, when the computer program is executed by a processor, the processor is caused to perform the following steps:

[0179] Selecting a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determining a second voxel corresponding to the first voxel in other three-dimensional models, wherein the first voxel and the second voxel are the same physical point in the space;

[0180] The three-dimensional distance between the first voxel and each second voxel is calculated respectively. If any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; if all the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels.

[0181] Further, the first voxel includes a first isomorphic voxel that is an isomorphic voxel, and the first voxel includes a first isomorphic voxel that is an isomorphic voxel. When the computer program is executed by a processor, the processor performs the following steps:

[0182] Calculating a three-dimensional distance between a second voxel corresponding to the first isomeric voxel and a panoramic camera corresponding to the reference model, and replacing the three-dimensional coordinates and color values ​​of the first isomeric voxel with the three-dimensional coordinates and color values ​​of the second voxel having the largest three-dimensional distance;

[0183] The three-dimensional distance between the first isomorphic voxel and all panoramic cameras is calculated, and the color value of the first isomorphic voxel is determined by using the three-dimensional distance between the first isomorphic voxel and all panoramic cameras.

[0184] Furthermore, the N panoramic images are respectively taken by panoramic camera A and panoramic camera B, and the three-dimensional model restored from the panoramic image taken by panoramic camera A is used as a reference model. When the computer program is executed by the processor, the processor performs the following steps:

[0185] The color value C of the first isomorphic voxel in the reference model is determined using the following formula:

[0186]

[0187] where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C B is the color value of the second voxel in the 3D model restored from the panoramic image taken by the panoramic camera B.

[0188] The above are preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A three-dimensional model construction method, characterized in that: include: Acquire N panoramic images of the same space, where the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1; Generate a 3D model using each panoramic image to obtain N 3D models; Aligning the N three-dimensional models using the target object in the space; Detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models; Correcting the three-dimensional coordinates and color values ​​of the isomorphic voxels, and enhancing the color values ​​of the isomorphic voxels; wherein a reference model is selected from the aligned N three-dimensional models, and for each first voxel of the reference model, a second voxel corresponding to the first voxel is determined in other three-dimensional models, and the first voxel and the second voxel are the same physical point in the space; Calculating the three-dimensional distance between the first voxel and each second voxel respectively, if any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; if all the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; The correcting of the three-dimensional coordinates and color values ​​of the isomorphic voxels and the enhancement of the color values ​​of the isomorphic voxels include: The first voxel includes a first isomeric voxel that is an isomeric voxel, calculating a three-dimensional distance between a second voxel corresponding to the first isomeric voxel and a panoramic camera corresponding to the reference model, and replacing the three-dimensional coordinates and color value of the first isomeric voxel with the three-dimensional coordinates and color value of the second voxel having the largest three-dimensional distance; The first voxel includes a first isomorphous voxel belonging to isomorphous voxels, a three-dimensional distance between the first isomorphous voxel and all panoramic cameras is calculated, and a color value of the first isomorphous voxel is determined by using the three-dimensional distance between the first isomorphous voxel and all panoramic cameras.

2. The three-dimensional model construction method according to claim 1, characterized in that: The generating a three-dimensional model using each panoramic image comprises: Detecting two-dimensional coordinates of corner points of the wall in the panoramic image, and calculating three-dimensional coordinates of the corner points of the wall using the two-dimensional coordinates; Determine the three-dimensional coordinates of a panoramic camera that shoots the panoramic image using the three-dimensional coordinates of the corner points of the wall, wherein the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model; The three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system are calculated using the three-dimensional coordinates of the corner points of the wall, and the color value of each voxel on the three-dimensional model is determined using the panoramic image.

3. The three-dimensional model construction method according to claim 2, characterized in that: The aligning the N three-dimensional models by using the target object in the space comprises: Detecting the same target object in the N panoramic images; Calculating the two-dimensional coordinates of the target object in each panoramic image; Calculating the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, selecting a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, and calculating the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates; The three-dimensional coordinates of the voxels of the other three-dimensional model are calculated based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and the other three-dimensional model is aligned with the reference three-dimensional model.

4. The three-dimensional model construction method according to claim 1, characterized in that: The preset threshold is 5 voxels.

5. The three-dimensional model construction method according to claim 3, characterized in that: The three-dimensional distance between at least part of the first voxels of the reference model and the coordinate origin of the benchmark three-dimensional model is smaller than the three-dimensional distance between the corresponding second voxels and the coordinate origin of the benchmark three-dimensional model.

6. The three-dimensional model construction method according to claim 5, characterized in that: N=2, the N panoramic images are taken by panoramic camera A and panoramic camera B respectively, the 3D model restored by the panoramic image taken by panoramic camera A is used as the reference model, and the color value C of the first isomorphic voxel in the reference model is determined by the following formula: where d A is the 3D distance between the first homogeneous voxel and the panoramic camera A, d B is the 3D distance between the first homogeneous voxel and the panoramic camera B, α is the preset weighting coefficient, C A is the original color value of the first isomorphic voxel, C B is the color value of the second voxel in the 3D model restored from the panoramic image captured by the panoramic camera B.

7. A three-dimensional model construction device, characterized in that: include: An acquisition unit, configured to acquire N panoramic images of the same space, wherein the N panoramic images are obtained by photographing the space at N different positions using a panoramic camera, where N is an integer greater than 1; A 3D model generating unit, used to generate a 3D model using each panoramic image, to obtain N 3D models; an alignment unit, configured to align the N three-dimensional models using a target object in the space; A detection unit, used for detecting isomorphic voxels and heteromorphic voxels in the aligned N three-dimensional models; A processing unit, configured to correct the three-dimensional coordinates and color values ​​of the isomorphic voxels and enhance the color values ​​of the isomorphic voxels; The detection unit is specifically used to select a reference model from the aligned N three-dimensional models, and for each first voxel of the reference model, determine a second voxel corresponding to the first voxel in other three-dimensional models, wherein the first voxel and the second voxel are the same physical point in the space; respectively calculate the three-dimensional distance between the first voxel and each second voxel, and if any of the three-dimensional distances is greater than a preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; if all the three-dimensional distances are not greater than the preset threshold, the first voxel and the corresponding second voxel are isomorphic voxels; The first voxel includes a first isomorphic voxel that is an isomorphic voxel, and the first voxel includes a first isomorphic voxel that is an isomorphic voxel; The processing unit is specifically used to calculate the three-dimensional distance between the second voxel corresponding to the first heterogeneous voxel and the panoramic camera corresponding to the reference model, and replace the three-dimensional coordinates and color value of the first heterogeneous voxel with the three-dimensional coordinates and color value of the second voxel with the largest three-dimensional distance; calculate the three-dimensional distance between the first homogeneous voxel and all panoramic cameras, and determine the color value of the first homogeneous voxel using the three-dimensional distance between the first homogeneous voxel and all panoramic cameras.

8. The three-dimensional model building device according to claim 7, characterized in that: The three-dimensional model generation unit is specifically used to detect the two-dimensional coordinates of the corner points of the wall in the panoramic image, and calculate the three-dimensional coordinates of the corner points of the wall using the two-dimensional coordinates; determine the three-dimensional coordinates of the panoramic camera that shoots the panoramic image using the three-dimensional coordinates of the corner points of the wall, and the three-dimensional coordinates of the panoramic camera are the coordinate origin of the three-dimensional coordinate system of the three-dimensional model; The three-dimensional coordinates of each voxel on the three-dimensional model in the three-dimensional coordinate system are calculated using the three-dimensional coordinates of the corner points of the wall, and the color value of each voxel on the three-dimensional model is determined using the panoramic image.

9. The three-dimensional model building device according to claim 7, characterized in that: The alignment unit is specifically used to detect the same target object in the N panoramic images; calculate the two-dimensional coordinates of the target object in each panoramic image; calculate the three-dimensional coordinates of the target object in the three-dimensional model using the two-dimensional coordinates of the target object, select a three-dimensional model from the N three-dimensional models as a reference three-dimensional model, the three-dimensional coordinates of the target object in the reference three-dimensional model as the reference three-dimensional coordinates, and calculate the three-dimensional displacement of the three-dimensional coordinates of the target object in other three-dimensional models relative to the reference three-dimensional coordinates; The three-dimensional coordinates of the voxels of the other three-dimensional model are calculated based on the three-dimensional displacement and the coordinate origin of the reference three-dimensional model, and the other three-dimensional model is aligned with the reference three-dimensional model.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional model construction method according to any one of claims 1 to 6 are implemented.

Images