Live three-dimensional scene fusion method and device, electronic equipment and storage medium

Abstract

This application relates to the field of geospatial information technology and discloses a scene fusion method for real-world 3D scenes, comprising: acquiring first photographic images captured by multiple first photographic devices and first view frustums corresponding to the multiple first photographic devices; determining the texture coordinates corresponding to each first view frustum based on each first photographic image; constructing first candidate 3D geometric surfaces for each first photographic image, and determining the texture coordinates of the first target view frustum as the texture coordinates of the first candidate 3D geometric surfaces; determining a first 3D geometric surface to be rendered from each of the first candidate 3D geometric surfaces; and rendering the first target image onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, thereby obtaining a first fused scene. This facilitates mapping the texture of photographic images onto 3D geometric surfaces. This application also discloses a scene fusion device, electronic device, and storage medium for real-world 3D scenes.

Description

Technical Field

[0001] This application relates to the field of geospatial information technology, such as a method and apparatus for scene fusion in real-world 3D, electronic devices, and storage media. Background Technology

[0002] Currently, surveying and mapping geospatial information technology is developing rapidly. Among them, 3D reality, as spatiotemporal information that can realistically, three-dimensionally, and temporally reflect human production, living, and ecological spaces, has become an important new type of national infrastructure. Integrating photographic information with 3D reality can achieve a near-realistic virtual reality monitoring experience, thereby breaking down barriers between distributed and discrete videos, improving the completeness of monitoring video information, and effectively enhancing urban management. However, the process of mapping the texture of photographic images onto 3D geometric surfaces during the integration of photographic information and 3D reality is complex and challenging. Summary of the Invention

[0003] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for scene fusion in real-world 3D, so as to map the texture of a photographic image onto a 3D geometric surface.

[0005] In some embodiments, a scene fusion method for real-world 3D includes: acquiring first photographic images captured by multiple first photographic devices and first view frustums corresponding to the multiple first photographic devices; determining texture coordinates corresponding to each first view frustum based on each first photographic image; constructing first candidate 3D geometric surfaces for each first photographic image, and determining the texture coordinates of a first target view frustum as the texture coordinates of the first candidate 3D geometric surface; the first target view frustum being the first view frustum corresponding to the first photographic image used to construct the first candidate 3D geometric surface; determining a first 3D geometric surface to be rendered from each of the first candidate 3D geometric surfaces; rendering a first target image onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, thereby obtaining a first fused scene; the first target image being the first photographic image used to construct the first 3D geometric surface to be rendered.

[0006] In some embodiments, the first viewing frustum corresponding to the first photographic device is obtained by: obtaining the first interior orientation element and the first exterior orientation element of the first photographic device; the first interior orientation element includes the principal point and principal distance of the first photographic device; the first exterior orientation element includes the spatial position of the photographic center corresponding to the first photographic device and the attitude parameters of the first photographic device; and constructing the first viewing frustum based on the first interior orientation element and the first exterior orientation element.

[0007] In some embodiments, determining the texture coordinates corresponding to the first view frustum based on the first photographic image includes: determining the texture coordinates corresponding to the first view frustum based on the geometric relationship between the first photographic image and the first view frustum corresponding to the first photographic image.

[0008] In some embodiments, constructing a first candidate three-dimensional geometric surface for a first photographic image includes: dividing the first photographic image into multiple grids; constructing a first ray with the photographic center corresponding to the first photographic image as the starting point and the vertices of the grids of the first photographic image as another point; intersecting the first ray with the three-dimensional scene to obtain the coordinates of the vertices of the grids of the first photographic image in the real-world three-dimensional scene; and fitting the vertices of each grid of the first photographic image according to a polynomial interpolation fitting method to obtain the first candidate three-dimensional geometric surface.

[0009] In some embodiments, determining a first three-dimensional geometric surface to be rendered from each of the first candidate three-dimensional geometric surfaces includes: obtaining a first overlap range common to each of the first candidate three-dimensional geometric surfaces and a first texture distortion degree corresponding to each of the first candidate three-dimensional geometric surfaces; and determining the first three-dimensional geometric surface to be rendered based on the first overlap range and the first texture distortion degree.

[0010] In some embodiments, rendering a first target image onto a first three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered includes: rendering the first target image onto the first three-dimensional geometric surface to be rendered using a texture-to-texture method according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered.

[0011] In some embodiments, after rendering the first target image onto the first three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered and obtaining the first fused scene, the method further includes: obtaining one or more second fused scenes; and stitching the first fused scene and the second fused scene together.

[0012] In some embodiments, a scene fusion apparatus for real-world 3D includes: an acquisition module configured to acquire first photographic images captured by a plurality of first photographic devices and first view frustums corresponding to the plurality of first photographic devices; a texture coordinate determination module configured to determine texture coordinates corresponding to each first view frustum based on each first photographic image; a construction module configured to construct first candidate 3D geometric surfaces for each first photographic image and determine the texture coordinates of a first target view frustum as the texture coordinates of the first candidate 3D geometric surfaces; the first target view frustum is the first view frustum corresponding to the first photographic image used to construct the first candidate 3D geometric surfaces; a 3D geometric surface to be rendered module configured to determine a first 3D geometric surface to be rendered from each of the first candidate 3D geometric surfaces; and a scene fusion module configured to render a first target image onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, thereby obtaining a first fused scene; the first target image is the first photographic image used to construct the first 3D geometric surface to be rendered.

[0013] In some embodiments, the electronic device includes a processor and a memory storing program instructions, the processor being configured to execute the above-described scene fusion method for real-world 3D when executing the program instructions.

[0014] In some embodiments, the storage medium stores program instructions that, when executed, perform the scene fusion method for real-world 3D scenes described above.

[0015] The scene fusion method, apparatus, electronic device, and storage medium for real-world 3D scenes provided in this disclosure can achieve the following technical effects: First photographic images captured by multiple first photographic devices and first view frustums corresponding to each of the first photographic devices are acquired. Texture coordinates corresponding to each first view frustum are determined based on each first photographic image. First candidate 3D geometric surfaces are constructed for each first photographic image, and the texture coordinates of the first target view frustum are determined as the texture coordinates of the first candidate 3D geometric surfaces. The first target view frustum is the first view frustum corresponding to the first photographic image used to construct the first candidate 3D geometric surfaces. A first 3D geometric surface to be rendered is determined from each of the first candidate 3D geometric surfaces. The first target image is rendered onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, thus obtaining a first fused scene. The first target image is the first photographic image used to construct the first 3D geometric surface to be rendered. In this way, by using the texture coordinates corresponding to the first view frustum as the texture coordinates of the first candidate 3D geometric surface, the texture relationship between the photographic image and the first candidate 3D geometric surface can be easily established. This facilitates mapping the texture of the photographic image onto the first 3D geometric surface to be rendered.

[0016] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of a scene fusion method for real-world 3D provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram illustrating the geometric relationship between a first photographic image and a first viewing cone, provided in an embodiment of this disclosure. Figure 3 This is a schematic diagram of the shooting range of a photographic device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of rendering a first photographic image onto a three-dimensional geometric surface according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a scene fusion device for real-world 3D provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0018] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0019] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0020] Unless otherwise stated, the term "multiple" means two or more.

[0021] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0022] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0023] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0024] Currently, the fusion of photographic information with 3D reality often faces challenges such as large data computation, difficulties in camera calibration, and challenges in accurate texture mapping. The fusion method described in this application enables the dynamic combination of static 3D reality and dynamic multi-channel photographic information, providing a good solution for the expression and utilization of diverse and dynamic urban information.

[0025] Combination Figure 1 As shown, this disclosure provides a scene fusion method for real-world 3D scenes, including: Step S101: The electronic device acquires first photographic images captured by multiple first photographic devices and first viewing cones corresponding to the multiple first photographic devices.

[0026] In step S102, the electronic device determines the texture coordinates corresponding to each first view frustum based on each first photographic image.

[0027] In step S103, the electronic device constructs a first candidate three-dimensional geometric surface for each first photographic image and determines the texture coordinates of the first target viewing frustum as the texture coordinates of the first candidate three-dimensional geometric surface; the first target viewing frustum is the first viewing frustum corresponding to the first photographic image for which the first candidate three-dimensional geometric surface is constructed.

[0028] In step S104, the electronic device determines the first three-dimensional geometric surface to be rendered from each of the first candidate three-dimensional geometric surfaces.

[0029] In step S105, the electronic device renders the first target image onto the first three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, thereby obtaining the first fused scene; the first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered.

[0030] The scene fusion method for real-world 3D provided in this disclosure involves acquiring first photographic images captured by multiple first photographic devices and first view frustums corresponding to each of the first photographic devices. Texture coordinates corresponding to each first view frustum are determined based on each first photographic image. First candidate 3D geometric surfaces are constructed for each first photographic image, and the texture coordinates of the first target view frustum are determined as the texture coordinates of the first candidate 3D geometric surfaces. The first target view frustum is the first view frustum corresponding to the first photographic image used to construct the first candidate 3D geometric surfaces. A first 3D geometric surface to be rendered is determined from each of the first candidate 3D geometric surfaces. The first target image is rendered onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, thus obtaining a first fused scene. The first target image is the first photographic image used to construct the first 3D geometric surface to be rendered. In this way, by using the texture coordinates corresponding to the first view frustum as the texture coordinates of the first candidate 3D geometric surface, the texture relationship between the photographic image and the first candidate 3D geometric surface can be easily established. This facilitates mapping the texture of the photographic image onto the first 3D geometric surface to be rendered.

[0031] In some embodiments, multiple first camera devices are installed on both sides of the road for road monitoring.

[0032] In some embodiments, the first target image is a first photographic image for constructing a first three-dimensional geometric surface to be rendered, that is, the first target image is a first photographic image for constructing a first alternative three-dimensional geometric surface corresponding to the first three-dimensional geometric surface to be rendered.

[0033] In some embodiments, the first viewing cone corresponding to the first photographic device that captures the first photographic image is used as the first viewing cone corresponding to the first photographic image.

[0034] Optionally, the first viewing frustum corresponding to the first imaging device is obtained by acquiring the first interior orientation elements and the first exterior orientation elements of the first imaging device. The first interior orientation elements include the principal point and principal distance of the first imaging device. The first exterior orientation elements include the spatial position of the imaging center corresponding to the first imaging device and the attitude parameters of the first imaging device. The first viewing frustum is constructed based on the first interior orientation elements and the first exterior orientation elements. The first interior orientation elements are determined by the calibration of the first imaging device. The imaging center is also called the optical center; the spatial position of the imaging center is its coordinates X, Y, and Z in the object space. The attitude parameters are three parameters describing the attitude: pitch angle θ, yaw angle ψ, and roll angle Φ. Thus, the first viewing frustum corresponding to the first imaging device can be accurately constructed using the first interior orientation elements and the first exterior orientation elements.

[0035] In some embodiments, the first viewing cone is a cone that describes the visible range of the first photographic device in a real scene. The first viewing cone consists of four side surfaces and one base surface.

[0036] Optionally, determining the texture coordinates corresponding to the first view frustum based on the first photographic image includes: determining the texture coordinates corresponding to the first view frustum based on the geometric relationship between the first photographic image and the first view frustum corresponding to the first photographic image. Thus, since there is a geometric correspondence between the first photographic image and the first view frustum, the texture coordinates corresponding to the first view frustum can be accurately determined through the geometric relationship between the first photographic image and the first view frustum corresponding to the first photographic image.

[0037] Furthermore, based on the geometric relationship between the first photographic image and the corresponding first viewing frustum, the texture coordinates corresponding to the first viewing frustum are determined, including: dividing the bottom surface of the first viewing frustum corresponding to the first photographic image into multiple grids; dividing the first photographic image into multiple grids; using the vertices of the grids in the first photographic image as first image plane points; using the vertices of the grids in the first viewing frustum as first grid vertices; and, based on the geometric relationship between the first photographic image and the first viewing frustum, assigning a one-to-one correspondence between the first grid vertices and the first image plane points, and using the first image plane points corresponding to the first grid vertices as the texture coordinates corresponding to the first grid vertices. Thus, due to the imaging principle of the first photographic device, the first photographic image captured by the first photographic device needs to be rotated 180 degrees clockwise for the scene direction presented in the first photographic image to be the same as the scene direction reflected by the bottom surface of the viewing frustum. Therefore, only by assigning a one-to-one correspondence between the first grid vertices and the first image plane points based on the geometric relationship between the first photographic image and the first viewing frustum can the texture coordinates corresponding to the first viewing frustum be accurately determined.

[0038] Optionally, based on the geometric relationship between the first photographic image and the first viewing cone, a one-to-one correspondence is established between the vertices of the first grid and the points on the first image plane, including: rotating the captured first photographic image 180 degrees clockwise; dividing the rotated first photographic image into n×n grids; dividing the bottom surface of the first viewing cone corresponding to the first photographic image into n×n grids; obtaining the row and column numbers corresponding to each grid in the first photographic image and the first viewing cone, respectively; establishing a correspondence between grids when the number of rows and columns of the grids are the same; and establishing a one-to-one correspondence between the grids of the first photographic image and the grids of the first viewing cone when a correspondence exists. Where n is a positive integer. In some embodiments, the grid F in the first photographic image is 5 rows and 6 columns; a correspondence is established between the grid F in the first photographic image and the grid G ​​located in the 5 rows and 6 columns of the first viewing cone. The first image plane point of grid F is used as the texture coordinate corresponding to the first square vertex of grid G.

[0039] Optionally, the correspondence between the first image plane points of the grid of the first photographic image and the first grid vertices of the grid of the first frustum includes: corresponding the first image plane point at the upper left corner of the grid of the first photographic image to the first grid vertex at the upper left corner of the grid of the first frustum; corresponding the first image plane point at the upper right corner of the grid of the first photographic image to the first grid vertex at the upper right corner of the grid of the first frustum; corresponding the first image plane point at the lower left corner of the grid of the first photographic image to the first grid vertex at the lower left corner of the grid of the first frustum; and corresponding the first image plane point at the lower right corner of the grid of the first photographic image to the first grid vertex at the lower right corner of the grid of the first frustum.

[0040] In some embodiments, combined with Figure 2 As shown, the base of the first viewing cone corresponding to the first photographic image is divided into an n×n grid. The unrotated first photographic image is also divided into an n×n grid. Based on the geometric relationship between the first photographic image and the first viewing cone, the vertices of the grid are mapped one-to-one with points on the image plane. For example, the lower left corner of the first photographic image is taken as the origin. The lower left corner of the base of the first viewing cone is also taken as the origin. The grid in the first column, row 6 of the first photographic image is mapped to the grid in the 9th column, row 1 of the base of the first viewing cone.

[0041] Optionally, constructing a first candidate 3D geometric surface for the first photographic image includes: dividing the first photographic image into multiple grids; constructing a first ray with the photographic center corresponding to the first photographic image as the starting point and the vertices of the grids of the first photographic image as another point; intersecting the first ray with the 3D scene to obtain the coordinates of the vertices of the grids of the first photographic image in the real-world 3D scene; and fitting the vertices of each grid of the first photographic image using a polynomial interpolation fitting method to obtain the first candidate 3D geometric surface. The first candidate 3D geometric surface in the real-world 3D scene, constructed by sequentially fitting using polynomial interpolation, has no texture information on its surface.

[0042] Optionally, determining the first 3D geometric surface to be rendered from each of the first candidate 3D geometric surfaces includes: obtaining a common first overlap range of each of the first candidate 3D geometric surfaces and a first texture distortion degree corresponding to each of the first candidate 3D geometric surfaces. The first 3D geometric surface to be rendered is determined based on the first overlap range and the first texture distortion degree. In this way, by using the first overlap range and the first texture distortion degree, the 3D geometric surface with the best projection effect can be selected for rendering.

[0043] In some embodiments, obtaining a common first overlap range for each of the first candidate three-dimensional geometric surfaces includes: performing overlay analysis in geographic data processing on each of the first candidate three-dimensional geometric surfaces to obtain a common first overlap range for each of the first candidate three-dimensional geometric surfaces. The first overlap range is the range of surfaces that overlap with each other among the multiple first candidate three-dimensional geometric surfaces. Thus, since the camera coverage areas of different first imaging devices are different, there may be overlap between the coverage areas of different first imaging devices. Combined with... Figure 3 As shown, for example, the shooting range of the first camera device A is the camera coverage area A, and the shooting range of the first camera device B is the camera coverage area B. Through intersection analysis, the overlapping range between camera coverage area A and camera coverage area B can be determined. Since there is overlap between the shooting ranges of the first camera devices, there is also overlap between the first candidate 3D geometric surfaces constructed from each first photographic image. Obtaining the common first overlap range of each first candidate 3D geometric surface facilitates the selection of a better first candidate 3D geometric surface for rendering.

[0044] Optionally, determining the first three-dimensional geometric surface to be rendered based on the first overlap range and the first texture distortion degree includes: determining the first candidate three-dimensional geometric surface with the lowest first texture distortion degree as the first target three-dimensional geometric surface; and determining the portion of the first target three-dimensional geometric surface belonging to the first overlap range as the first three-dimensional geometric surface to be rendered.

[0045] Optionally, rendering the first target image onto the first three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered includes: rendering the first target image onto the first three-dimensional geometric surface to be rendered using a texture-to-texture method according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered. Thus, combined with... Figure 4 As shown, the first photographic image is rendered onto a 3D geometric surface. This 3D geometric surface is the actual 3D surface of the scene. It allows for easy mapping of the texture of the photographic image onto the first 3D geometric surface to be rendered.

[0046] Optionally, after rendering the first target image onto the first three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, and obtaining the first fused scene, the method further includes: obtaining one or more second fused scenes; and stitching the first fused scene and the second fused scene together.

[0047] Further, obtaining the second fused scene includes: acquiring second photographic images captured by multiple second photographic devices and second view frustums corresponding to each of the multiple second photographic devices. Determining the texture coordinates corresponding to each second view frustum based on each second photographic image. Constructing second candidate 3D geometric surfaces for each second photographic image, and determining the texture coordinates of the second target view frustum as the texture coordinates of the second candidate 3D geometric surfaces. The second target view frustum is the second view frustum corresponding to the second photographic image used to construct the second candidate 3D geometric surfaces. Determining a second 3D geometric surface to be rendered from each of the second candidate 3D geometric surfaces. Rendering the second target image onto the second 3D geometric surface to be rendered based on the texture coordinates corresponding to the second 3D geometric surface to be rendered, thus obtaining the second fused scene. The second target image is the second photographic image used to construct the second 3D geometric surface to be rendered.

[0048] Furthermore, multiple second camera devices are installed on both sides of the road to monitor the road. In some embodiments, the road monitored by the second camera devices is different from the road monitored by the first camera device.

[0049] In some embodiments, the second target image is a second photographic image for constructing the second three-dimensional geometric surface to be rendered, that is, the second target image is a second photographic image for constructing the second alternative three-dimensional geometric surface corresponding to the second three-dimensional geometric surface to be rendered.

[0050] In some embodiments, the second viewing cone corresponding to the second photographic device that captures the second photographic image is used as the second viewing cone corresponding to the second photographic image.

[0051] Optionally, the second viewing frustum corresponding to the second imaging device is obtained by: obtaining the second interior orientation elements and the second exterior orientation elements of the second imaging device. The second interior orientation elements include the principal point and principal distance of the second imaging device. The second exterior orientation elements include the spatial position of the imaging center corresponding to the second imaging device and the pose parameters of the second imaging device. The second viewing frustum is constructed based on the second interior orientation elements and the second exterior orientation elements. The second interior orientation elements are determined by the calibration of the second imaging device. The second viewing frustum is a cone that describes the visible range of the second imaging device in the real scene. The second viewing frustum consists of four sides and one base.

[0052] Further, determining the texture coordinates corresponding to the second view frustum based on the second photographic image includes: determining the texture coordinates corresponding to the second view frustum based on the geometric relationship between the second photographic image and the second view frustum corresponding to the second photographic image.

[0053] Furthermore, based on the geometric relationship between the second photographic image and the corresponding second view frustum, the texture coordinates corresponding to the second view frustum are determined, including: dividing the bottom surface of the second view frustum corresponding to the second photographic image into multiple grids; dividing the second photographic image into multiple grids; using the vertices of the grids in the second photographic image as second image plane points; using the vertices of the grids in the second view frustum as second grid vertices; and, based on the geometric relationship between the second photographic image and the second view frustum, establishing a one-to-one correspondence between the second grid vertices and the second image plane points, and using the second image plane points corresponding to the second grid vertices as the texture coordinates corresponding to the second grid vertices.

[0054] Furthermore, based on the geometric relationship between the second photographic image and the second viewing cone, a one-to-one correspondence is established between the vertices of the second grid and the points on the second image plane. This includes: rotating the captured second photographic image 180 degrees clockwise; dividing the rotated second photographic image into v×v grids; dividing the base of the second viewing cone corresponding to the second photographic image into v×v grids; obtaining the row and column numbers corresponding to each grid in the second photographic image and the second viewing cone; establishing a correspondence between grids when the number of rows and columns is the same; and, when a correspondence exists between the grids of the second photographic image and the grids of the second viewing cone, establishing a one-to-one correspondence between the second image plane points of the grids in the second photographic image and the vertices of the second grids in the second viewing cone. Here, v is a positive integer.

[0055] Furthermore, constructing a second candidate three-dimensional geometric surface for the second photographic image includes: dividing the second photographic image into multiple grids; constructing a second ray with the photographic center corresponding to the second photographic image as the starting point and the vertices of the grids of the second photographic image as another point; intersecting the second ray with the three-dimensional scene to obtain the coordinates of the vertices of the grids of the second photographic image in the real-world three-dimensional scene; and fitting the vertices of each grid of the second photographic image using a polynomial interpolation fitting method to obtain the second candidate three-dimensional geometric surface.

[0056] Further, determining the second three-dimensional geometric surface to be rendered from each of the second candidate three-dimensional geometric surfaces includes: obtaining the common second overlap range of each of the second candidate three-dimensional geometric surfaces and the second texture distortion degree corresponding to each of the second candidate three-dimensional geometric surfaces; and determining the second three-dimensional geometric surface to be rendered based on the second overlap range and the second texture distortion degree.

[0057] In some embodiments, obtaining the common second overlap range of each second candidate three-dimensional geometric surface includes: performing overlay analysis in geographic data processing on each second candidate three-dimensional geometric surface to obtain the common second overlap range of each second candidate three-dimensional geometric surface. The second overlap range is the range of surfaces that overlap with each other among the multiple second candidate three-dimensional geometric surfaces.

[0058] Furthermore, the second three-dimensional geometric surface to be rendered is determined based on the second overlap range and the second texture distortion degree, including: determining the second candidate three-dimensional geometric surface with the lowest second texture distortion degree as the second target three-dimensional geometric surface; and determining the portion of the second target three-dimensional geometric surface that belongs to the second overlap range as the second three-dimensional geometric surface to be rendered.

[0059] Furthermore, rendering the second target image onto the second three-dimensional geometric surface to be rendered according to the texture coordinates corresponding to the second three-dimensional geometric surface to be rendered includes: rendering the second target image onto the second three-dimensional geometric surface to be rendered using a texture-to-texture method according to the texture coordinates corresponding to the second three-dimensional geometric surface to be rendered.

[0060] Combination Figure 5 As shown, this embodiment of the present disclosure provides a scene fusion device 1 for real-world 3D, including: an acquisition module 2, a texture coordinate determination module 3, a construction module 4, a 3D geometry surface determination module 5, and a scene fusion module 6. The acquisition module 2 is configured to acquire first photographic images captured by multiple first photographic devices and first view frustums corresponding to the multiple first photographic devices. The texture coordinate determination module 3 is configured to determine the texture coordinates corresponding to each first view frustum based on each first photographic image. The construction module 4 is configured to construct first candidate 3D geometric surfaces for each first photographic image, and determine the texture coordinates of a first target view frustum as the texture coordinates of the first candidate 3D geometric surfaces; the first target view frustum is the first view frustum corresponding to the first photographic image from which the first candidate 3D geometric surfaces are constructed. The first 3D geometric surface determination module 5 is configured to determine the first 3D geometric surface to be rendered from each of the first candidate 3D geometric surfaces. Scene fusion module 6 is configured to render the first target image onto the first three-dimensional geometric surface to be rendered based on the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, thereby obtaining the first fused scene; the first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered.

[0061] Using the display apparatus provided in this disclosure, an acquisition module acquires first photographic images captured by multiple first photographic devices and first viewing frustums corresponding to each of the first photographic devices. A texture coordinate determination module determines the texture coordinates corresponding to each first viewing frustum based on each first photographic image. A construction module constructs first candidate three-dimensional geometric surfaces for each first photographic image and determines the texture coordinates of the first target viewing frustum as the texture coordinates of the first candidate three-dimensional geometric surface. The first target viewing frustum is the first viewing frustum corresponding to the first photographic image used to construct the first candidate three-dimensional geometric surface. A first three-dimensional geometric surface to be rendered determination module determines the first three-dimensional geometric surface to be rendered from each of the first candidate three-dimensional geometric surfaces. A scene fusion module renders the first target image onto the first three-dimensional geometric surface to be rendered based on the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, obtaining a first fused scene. The first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered. In this way, by using the texture coordinates corresponding to the first viewing frustum as the texture coordinates of the first candidate three-dimensional geometric surface, the texture relationship between the photographic image and the first candidate three-dimensional geometric surface can be easily established. This facilitates mapping the texture of the photographic image onto the first three-dimensional geometric surface to be rendered.

[0062] Combination Figure 6 As shown, this embodiment of the disclosure provides an electronic device 7, including a processor 8 and a memory 9. Optionally, the device may further include a communication interface 10 and a bus 11. The processor 8, communication interface 10, and memory 9 can communicate with each other via the bus 11. The communication interface 10 can be used for information transmission. The processor 8 can call logical instructions in the memory 9 to execute the scene fusion method for real-world 3D described in the above embodiment.

[0063] Using the electronic device provided in this disclosure, first photographic images captured by multiple first photographic devices and first view frustums corresponding to each of the first photographic devices are acquired. Texture coordinates corresponding to each first view frustum are determined based on each first photographic image. First candidate three-dimensional geometric surfaces are constructed for each first photographic image, and the texture coordinates of the first target view frustum are determined as the texture coordinates of the first candidate three-dimensional geometric surfaces. The first target view frustum is the first view frustum corresponding to the first photographic image used to construct the first candidate three-dimensional geometric surfaces. A first three-dimensional geometric surface to be rendered is determined from each of the first candidate three-dimensional geometric surfaces. The first target image is rendered onto the first three-dimensional geometric surface to be rendered based on the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, obtaining a first fused scene. The first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered. Thus, by using the texture coordinates corresponding to the first view frustum as the texture coordinates of the first candidate three-dimensional geometric surface, the texture relationship between the photographic image and the first candidate three-dimensional geometric surface can be easily established. This facilitates mapping the texture of the photographic image onto the first three-dimensional geometric surface to be rendered.

[0064] Furthermore, the logical instructions in the aforementioned memory 9 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0065] The memory 9, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 8 executes the program instructions / modules stored in the memory 9 to perform functional applications and data processing, that is, to implement the scene fusion method for real-world 3D in the above embodiments.

[0066] The memory 9 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 9 may include high-speed random access memory and may also include non-volatile memory.

[0067] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the above-described scene fusion method for real-world 3D.

[0068] This disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the above-described scene fusion method for real-world 3D.

[0069] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0070] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0071] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0072] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A scene fusion method for real-world 3D scenes, characterized in that, include: Acquire first photographic images captured by multiple first photographic devices and first viewing cones corresponding to the multiple first photographic devices respectively; Based on the geometric relationship between each first photographic image and the first view frustum corresponding to the first photographic image, the texture coordinates corresponding to each first view frustum are determined respectively; For each first photographic image, a first candidate three-dimensional geometric surface is constructed, and the texture coordinates of the first target view frustum are determined as the texture coordinates of the first candidate three-dimensional geometric surface; the first target view frustum is the first view frustum corresponding to the first photographic image for which the first candidate three-dimensional geometric surface is constructed; The first 3D geometric surface to be rendered is determined from each of the first candidate 3D geometric surfaces; The first target image is rendered onto the first three-dimensional geometric surface to be rendered based on the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, thereby obtaining the first fused scene; The first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered.

2. The method according to claim 1, characterized in that, The first viewing cone corresponding to the first imaging device is obtained through the following method: Acquire the first interior orientation elements and the first exterior orientation elements of the first imaging device; the first interior orientation elements include the principal image point and principal image distance of the first imaging device; the first exterior orientation elements include the spatial position of the imaging center corresponding to the first imaging device and the attitude parameters of the first imaging device; Construct the first view frustum based on the first inner orientation element and the first outer orientation element.

3. The method according to claim 1, characterized in that, Constructing a first candidate 3D geometry from the first photographic image, including: The first photographic image is divided into multiple grids; A first ray is constructed with the center of the first photographic image as the starting point and the vertices of the grid of the first photographic image as another point. Intersect the first ray with the 3D scene to obtain the coordinates of the vertices of the grid of the first photographic image in the real 3D scene; The vertices of the grid of the first photographic image are fitted using a polynomial interpolation method to obtain the first candidate three-dimensional geometric surface.

4. The method according to claim 1, characterized in that, The first 3D geometric surface to be rendered is determined from each of the first candidate 3D geometric surfaces, including: Obtain the first overlap range common to each of the first candidate three-dimensional geometric surfaces and the first texture distortion degree corresponding to each of the first candidate three-dimensional geometric surfaces; The first 3D geometry to be rendered is determined based on the first overlap range and the first texture distortion degree.

5. The method according to claim 1, characterized in that, Rendering the first target image onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered includes: According to the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, the first target image is rendered onto the first three-dimensional geometric surface to be rendered using the method of rendering to texture.

6. The method according to any one of claims 1 to 5, characterized in that, After rendering the first target image onto the first 3D geometric surface to be rendered based on the texture coordinates corresponding to the first 3D geometric surface to be rendered, and obtaining the first blended scene, the process further includes: Acquire one or more second fusion scenes; The first and second merged scenes are spliced ​​together.

7. A scene fusion device for real-world 3D scenes, characterized in that, include: The acquisition module is configured to acquire first photographic images captured by multiple first photographic devices and first viewing cones corresponding to the multiple first photographic devices respectively; The texture coordinate determination module is configured to determine the texture coordinates corresponding to each first view frustum based on the geometric relationship between each first photographic image and the first view frustum corresponding to the first photographic image. The construction module is configured to construct a first candidate three-dimensional geometric surface for each first photographic image, and to determine the texture coordinates of the first target view frustum as the texture coordinates of the first candidate three-dimensional geometric surface; the first target view frustum is the first view frustum corresponding to the first photographic image for which the first candidate three-dimensional geometric surface is constructed; The module for determining the three-dimensional geometry to be rendered is configured to determine the first three-dimensional geometry to be rendered from each of the first candidate three-dimensional geometry surfaces. The scene fusion module is configured to render the first target image onto the first three-dimensional geometric surface to be rendered based on the texture coordinates corresponding to the first three-dimensional geometric surface to be rendered, thereby obtaining the first fused scene; the first target image is the first photographic image used to construct the first three-dimensional geometric surface to be rendered.

8. An electronic device comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when executing the program instructions, perform the scene fusion method for real-world 3D as described in any one of claims 1 to 6.

9. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the scene fusion method for real-world 3D as described in any one of claims 1 to 6.

Images