Three-dimensional scene reconstruction method, device, computer equipment and storage medium

Abstract

The embodiments of the present application belong to the field of image processing technology in artificial intelligence, and relate to a three-dimensional scene reconstruction method, device, computer equipment and storage medium. The present application obtains sampling space points by ray tracing method, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted by a lightweight network, and the three-dimensional scene of the target scene is reconstructed by the predicted color value, which can realize the scene (approximately 300×300m) within 1.5 hours. 2 ) and ensures high quality of the three-dimensional scene.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional scene reconstruction method, apparatus, computer equipment, and storage medium. Background Art

[0002] Large-scale scene reconstruction from multi-view images is a classic problem in computer vision. Traditional approaches often rely on structure-from-motion (SFM) methods to estimate camera pose and apply dense multi-view stereo to reconstruct a 3D model of the scene. Since 2020, a new neural scene representation, Neural Radiance Fields (NeRF), has revolutionized the problem of object (scene) reconstruction and novel view synthesis. It has garnered widespread attention for its realistic rendering quality.

[0003] There is a 3D scene reconstruction method, inspired by Mega-NeRF, which uses images collected by drones to reconstruct large-scale 3D scenes (such as cities, campuses, suburbs, etc.).

[0004] However, the applicant found that the traditional three-dimensional scene reconstruction method simply uses the network to learn the attributes in the three-dimensional scene (such as the color and density of a certain spatial point). The process of searching for three-dimensional spatial points through the network and then obtaining the attributes is very time-consuming and requires a lot of computing power (for example, it takes a day to train on a machine with 8 GPUs). As a result, such methods cannot be applied on low-cost, low-computing power devices and require a long training time. It can be seen that the traditional three-dimensional scene reconstruction method has the problems of high cost, high equipment requirements and low efficiency. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to propose a three-dimensional scene reconstruction method, apparatus, computer equipment and storage medium to solve the problems of high cost, high equipment requirements and low efficiency of traditional three-dimensional scene reconstruction methods.

[0006] In order to solve the above technical problems, the present application provides a method for reconstructing a three-dimensional scene, which adopts the following technical solutions:

[0007] receiving a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene;

[0008] Performing a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points;

[0009] Performing spatial parameterization on the sampling space points to obtain spatial point parameters;

[0010] Performing a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data;

[0011] Performing a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data;

[0012] Performing a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data;

[0013] Inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the spatial point color;

[0014] A target three-dimensional scene corresponding to the target scene is constructed according to the colors of the spatial points.

[0015] Furthermore, the plane feature data includes vertical plane data. After the step of performing a plane feature extraction operation on the spatial point parameters according to the high-resolution plane to obtain the plane feature data, the following steps are further included:

[0016] A scaling operation is performed on the vertical plane data.

[0017] Furthermore, the step of inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the color of the spatial point specifically includes the following steps:

[0018] The attribute prediction operation is performed on the mixed feature data according to a supervised loss function, wherein the supervised loss function is expressed as:

[0019]

[0020] Wherein, R represents the ray set of sampling points in the mixed feature data; Represents the color of the spatial point; C(r) represents the true value of the pixel color.

[0021] Furthermore, the step of inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the color of the spatial point specifically includes the following steps:

[0022] Inputting the mixed feature data into the lightweight network to obtain a predicted color value;

[0023] The predicted color values ​​are accumulated according to a volume rendering method to obtain the spatial point color.

[0024] Furthermore, the volume rendering method is expressed as:

[0025]

[0026] Among them, δ i Indicates the interval distance between adjacent sampling points; T i Indicates the cumulative transparency of the calculation, the T i Expressed as:

[0027]

[0028] In order to solve the above technical problems, the present application also provides a 3D scene reconstruction device, which adopts the following technical solution:

[0029] a request receiving module, configured to receive a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene;

[0030] A spatial point sampling module, configured to perform a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points;

[0031] A spatial parameterization module, configured to perform spatial parameterization on the sampling spatial points to obtain spatial point parameters;

[0032] A hash feature extraction module is used to perform a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data;

[0033] A plane feature extraction module is used to perform a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data;

[0034] A feature mixing module, configured to perform a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data;

[0035] A color prediction module is used to input the mixed feature data into a lightweight network to perform a color prediction operation to obtain the spatial point color;

[0036] A construction module is used to construct a target three-dimensional scene corresponding to the target scene according to the color of the spatial point.

[0037] Furthermore, the device further comprises:

[0038] The scaling operation module is used to perform a scaling operation on the vertical plane data.

[0039] Furthermore, the color prediction module includes:

[0040] A data input submodule, configured to input the mixed feature data into the lightweight network to obtain a predicted color value;

[0041] The accumulation operation submodule is used to perform an accumulation operation on the predicted color value according to a volume rendering method to obtain the color of the spatial point.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] The system comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the three-dimensional scene reconstruction method as described above when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0045] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the three-dimensional scene reconstruction method described above are implemented.

[0046] The present application provides a three-dimensional scene reconstruction method, comprising: receiving a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene; performing a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points; performing spatial parameterization on the sampling spatial points to obtain spatial point parameters; performing a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data; performing a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data; performing a feature blending operation on the hash feature data and the plane feature data to obtain blended feature data; inputting the blended feature data into a lightweight network to perform a color prediction operation to obtain spatial point colors; and constructing a target three-dimensional scene corresponding to the target scene according to the spatial point colors. Compared with the existing technology, this application obtains sampling space points through ray tracing, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted through a lightweight network, and the three-dimensional scene of the target scene is reconstructed by the predicted color value, which can realize the scene (approximately 300×300m) in 1.5 hours. 2 ) and ensures high quality of the three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0049] Figure 2 This is a flowchart of the implementation of the three-dimensional scene reconstruction method provided in Example 1 of the present application;

[0050] Figure 3 Schematic diagram of the structure of the three-dimensional scene reconstruction device provided in Example 2 of the present application;

[0051] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0053] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0055] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0056] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0057] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0058] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0059] It should be noted that the three-dimensional scene reconstruction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the three-dimensional scene reconstruction device is generally set in the server / terminal device.

[0060] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0061] Example 1

[0062] Continue to refer Figure 2 , shows an implementation flow chart of the three-dimensional scene reconstruction method provided in Example 1 of the present application. For the sake of convenience, only the part related to the present application is shown.

[0063] The above-mentioned three-dimensional scene reconstruction method includes the following steps: step S201, step S202, step S203, step S204, step S205, step S206, step S207 and step S208.

[0064] In step S201, a scene reconstruction request sent by a user terminal is received, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene.

[0065] In the embodiments of the present application, the user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not used to limit the present invention.

[0066] In step S202, a spatial point sampling operation is performed on the large-scale image according to a ray tracing method to obtain sampling spatial points.

[0067] Using the ray tracing method, a ray is emitted from the image pixel position, and then a three-dimensional space point is sampled on the ray to obtain the coordinates and direction of the space point.

[0068] In the embodiment of the present application, for sampling points, the space is limited and divided into foreground and background. The present invention adopts two sampling methods for the two areas respectively:

[0069] 1. Linearly sample the foreground area to ensure that the sampling points cover the foreground area of ​​the scene and meet the needs of high-quality foreground rendering;

[0070] 2. Inverse sampling of the background area, that is, using a nonlinear sampling method with an interval equal to the inverse of the distance to ensure coverage of the distant background.

[0071] In step S203, spatial parameterization is performed on the sampling space points to obtain space point parameters.

[0072] In the embodiment of the present application, spatial parameterization is performed on the sampled spatial points. Specifically, considering a scene centered at the origin, we divide the scene into foreground and background areas and separate them by a predefined boundary B. Given a 3D point, we first normalize it:

[0073] x=x / B

[0074] In this application example, the spatial contraction method is then applied:

[0075]

[0076] Wherein, x represents the coordinate of the spatial point, p represents the norm, and b represents the size of the space occupied by the background area (such as the outer circle in the figure). In a specific embodiment, the present invention sets p to 2 and b to 1.

[0077] In step S204, a hash feature extraction operation is performed on the spatial point parameters according to the multi-resolution hash grid to obtain hash feature data.

[0078] In an embodiment of the present application, a multi-resolution hash grid is an efficient data structure that can represent a scene with a high-resolution grid (e.g., 1024 or higher resolution), and randomly maps 3D points into a linear hash table of fixed size through a hash function. The parameter quantity of the multi-resolution hash grid is defined by L·T·F, where L is the number of resolutions, and T and F represent the hash table size and feature dimension at each resolution. In a specific embodiment, in order to balance efficiency and performance, the present invention sets L=16, T=2^19, and F=2.

[0079] In step S205 , a plane feature extraction operation is performed on the spatial point parameters according to the high-resolution plane to obtain plane feature data.

[0080] In the embodiment of the present application, the plane features are designed as three orthogonally placed planes with a resolution of N and a feature dimension of F. For the queried three-dimensional point, orthogonal projection is first performed on these three planes, and then two-dimensional plane features are obtained by bilinear interpolation. The three interpolated features are then spliced ​​into a feature vector with a length of 3·F.

[0081] In the embodiment of the present application, the plane is a multi-resolution plane (i.e., there are multiple planes with different resolutions. In the embodiment, we use 4 scales, and the plane resolutions N are 128, 256, 512, and 1024, respectively. The feature dimension F at each scale is 2, so the total feature dimension is equal to 8). High-resolution and low-resolution plane features are used, and global and local information are combined to make a trade-off between parameter quantity and performance.

[0082] In step S206, a feature mixing operation is performed on the hash feature data and the plane feature data to obtain mixed feature data.

[0083] In an embodiment of the present application, the two feature extraction methods mentioned above are combined to form a hybrid representation based on a hash grid and a high-resolution plane. Specifically, the present invention directly splices the two extracted features to form hybrid feature data.

[0084] In step S207, the mixed feature data is input into the lightweight network for color prediction operation to obtain the spatial point color.

[0085] In step S208, a target three-dimensional scene corresponding to the target scene is constructed according to the colors of the spatial points.

[0086] In an embodiment of the present application, a three-dimensional scene reconstruction method is provided, including: receiving a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene; performing a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampled spatial points; performing spatial parameterization on the sampled spatial points to obtain spatial point parameters; performing a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data; performing a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data; performing a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data; inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain spatial point colors; and constructing a target three-dimensional scene corresponding to the target scene according to the spatial point colors. Compared with the existing technology, this application obtains sampling space points through ray tracing, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted through a lightweight network, and the three-dimensional scene of the target scene is reconstructed by the predicted color value, which can realize the scene (approximately 300×300m) in 1.5 hours. 2 ) and ensures high quality of the three-dimensional scene.

[0087] In some optional implementations of this embodiment, the plane feature data includes vertical plane data, and after step S205, the following steps are further included:

[0088] Perform scaling operations on vertical plane data.

[0089] In the embodiment of this application, due to the large scale of the target scene, the height of the scene is often smaller than the horizontal length. In order to reduce the waste of features on the vertical plane, we use the camera height measurement to scale the vertical plane so that the spatial points of the scene fill the designed plane.

[0090] In some optional implementations of this embodiment, step S207 specifically includes the following steps:

[0091] The attribute prediction operation is performed on the mixed feature data according to the supervised loss function, where the supervised loss function is expressed as:

[0092]

[0093] Where R represents the ray set of sampling points in the mixed feature data; Represents the color of a spatial point; C(r) represents the true value of the pixel color.

[0094] In the embodiment of the present application, the mixed features are input into a lightweight network, and the attributes of the input three-dimensional space point are predicted by supervised learning. The attributes include the density value σ of the space point. i and color value c i Then, the color of each spatial point on the ray is accumulated by volume rendering to obtain the pixel color value corresponding to the ray.

[0095] In some optional implementations of this embodiment, step S207 specifically includes the following steps:

[0096] Input the mixed feature data into the lightweight network to obtain the predicted color value;

[0097] The predicted color values ​​are accumulated according to the volume rendering method to obtain the spatial point color.

[0098] In some optional implementations of this embodiment, the volume rendering method is expressed as follows:

[0099]

[0100] Among them, δ i Indicates the interval distance between adjacent sampling points; T i Indicates the cumulative transparency of the calculation, T i Expressed as:

[0101]

[0102] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When executed, the computer-readable instructions can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0103] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0104] Example 2

[0105] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a three-dimensional scene reconstruction device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 3 As shown, the 3D scene reconstruction device 200 of this embodiment includes: a request receiving module 210, a spatial point sampling module 220, a spatial parameterization module 230, a hash feature extraction module 240, a plane feature extraction module 250, a feature mixing module 260, a color prediction module 270, and a construction module 280. Among them:

[0107] a request receiving module 210, configured to receive a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene;

[0108] A spatial point sampling module 220 is configured to perform a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points;

[0109] A spatial parameterization module 230 is configured to perform spatial parameterization on the sampling spatial points to obtain spatial point parameters;

[0110] A hash feature extraction module 240 is configured to perform a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data;

[0111] A plane feature extraction module 250 is configured to perform a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data;

[0112] A feature mixing module 260 is configured to perform a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data;

[0113] A color prediction module 270 is configured to input the mixed feature data into a lightweight network for color prediction to obtain a spatial point color;

[0114] The construction module 280 is configured to construct a target three-dimensional scene corresponding to the target scene according to the color of the spatial point.

[0115] In the embodiments of the present application, the user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not used to limit the present invention.

[0116] Using the ray tracing method, a ray is emitted from the image pixel position, and then a three-dimensional space point is sampled on the ray to obtain the coordinates and direction of the space point.

[0117] In the embodiment of the present application, for sampling points, the space is limited and divided into foreground and background. The present invention adopts two sampling methods for the two areas respectively:

[0118] 1. Linearly sample the foreground area to ensure that the sampling points cover the foreground area of ​​the scene and meet the needs of high-quality foreground rendering;

[0119] 2. Inverse sampling of the background area, that is, using a nonlinear sampling method with an interval equal to the inverse of the distance to ensure coverage of the distant background.

[0120] In the embodiment of the present application, spatial parameterization is performed on the sampled spatial points. Specifically, considering a scene centered at the origin, we divide the scene into foreground and background areas and separate them by a predefined boundary B. Given a 3D point, we first normalize it:

[0121] x=x / B

[0122] In this application example, the spatial contraction method is then applied:

[0123]

[0124] Wherein, x represents the coordinate of the spatial point, p represents the norm, and b represents the size of the space occupied by the background area (such as the outer circle in the figure). In a specific embodiment, the present invention sets p to 2 and b to 1.

[0125] In an embodiment of the present application, a multi-resolution hash grid is an efficient data structure that can represent a scene with a high-resolution grid (e.g., 1024 or higher resolution), and randomly maps 3D points into a linear hash table of fixed size through a hash function. The parameter quantity of the multi-resolution hash grid is defined by L·T·F, where L is the number of resolutions, and T and F represent the hash table size and feature dimension at each resolution. In a specific embodiment, in order to balance efficiency and performance, the present invention sets L=16, T=2^19, and F=2.

[0126] In the embodiment of the present application, the plane features are designed as three orthogonally placed planes with a resolution of N and a feature dimension of F. For the queried three-dimensional point, orthogonal projection is first performed on these three planes, and then two-dimensional plane features are obtained by bilinear interpolation. The three interpolated features are then spliced ​​into a feature vector with a length of 3·F.

[0127] In the embodiment of the present application, the plane is a multi-resolution plane (i.e., there are multiple planes with different resolutions. In the embodiment, we use 4 scales, and the plane resolutions N are 128, 256, 512, and 1024, respectively. The feature dimension F at each scale is 2, so the total feature dimension is equal to 8). High-resolution and low-resolution plane features are used, and global and local information are combined to make a trade-off between parameter quantity and performance.

[0128] In an embodiment of the present application, the two feature extraction methods mentioned above are combined to form a hybrid representation based on a hash grid and a high-resolution plane. Specifically, the present invention directly splices the two extracted features to form hybrid feature data.

[0129] In an embodiment of the present application, a three-dimensional scene reconstruction device 200 is provided, including: a request receiving module 210, a spatial point sampling module 220, a spatial parameterization module 230, a hash feature extraction module 240, a plane feature extraction module 250, a feature mixing module 260, a color prediction module 270 and a construction module 280. Compared with the prior art, the present application obtains sampling space points by ray tracing, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted by a lightweight network, and the three-dimensional scene of the target scene is reconstructed by the predicted color value, which can realize the scene (approximately 300×300m 2 ) and ensures high quality of the three-dimensional scene.

[0130] In some optional implementations of this embodiment, the plane feature data includes vertical plane data, and the three-dimensional scene reconstruction device 200 further includes a scaling operation module, wherein:

[0131] The scaling operation module is used to perform scaling operations on vertical plane data.

[0132] In the embodiment of this application, due to the large scale of the target scene, the height of the scene is often smaller than the horizontal length. In order to reduce the waste of features on the vertical plane, we use the camera height measurement to scale the vertical plane so that the spatial points of the scene fill the designed plane.

[0133] In some optional implementations of this embodiment, the color prediction module 270 includes: an attribute prediction submodule, wherein:

[0134] The attribute prediction submodule is used to perform attribute prediction operations on mixed feature data based on the supervised loss function, where the supervised loss function is expressed as:

[0135]

[0136] Where R represents the ray set of sampling points in the mixed feature data; Represents the color of a spatial point; C(r) represents the true value of the pixel color.

[0137] In the embodiment of the present application, the mixed features are input into a lightweight network, and the attributes of the input three-dimensional space point are predicted by supervised learning. The attributes include the density value σ of the space point. i and color value c i Then, the color of each spatial point on the ray is accumulated by volume rendering to obtain the pixel color value corresponding to the ray.

[0138] In some optional implementations of this embodiment, the color prediction module 270 includes: a data input submodule and an accumulation operation submodule, wherein:

[0139] The data input submodule is used to input the mixed feature data into the lightweight network to obtain the predicted color value;

[0140] The accumulation operation submodule is used to perform an accumulation operation on the predicted color value according to the volume rendering method to obtain the spatial point color.

[0141] In some optional implementations of this embodiment, the volume rendering method is expressed as follows:

[0142]

[0143] Among them, δ i Indicates the interval distance between adjacent sampling points; Ti Indicates the cumulative transparency of the calculation, T i Expressed as:

[0144]

[0145] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0146] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows the computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0147] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0148] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions for a three-dimensional scene reconstruction method. In addition, the memory 310 can also be used to temporarily store various data that has been output or is about to be output.

[0149] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for executing the three-dimensional scene reconstruction method.

[0150] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0151] The computer device provided by this application obtains sampling space points through ray tracing, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted through a lightweight network, and the three-dimensional scene of the target scene is reconstructed based on the predicted color values. It can realize the scene (approximately 300×300m) in 1.5 hours. 2) and ensures high quality of the three-dimensional scene.

[0152] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the three-dimensional scene reconstruction method as described above.

[0153] The computer-readable storage medium provided by the present application obtains sampling space points through ray tracing, and performs spatial parameterization on the sampling space points to obtain space point parameters; then, feature extraction is performed on the space point parameters according to the multi-resolution hash grid and the high-resolution plane, and the extracted feature data is mixed to obtain mixed feature data; finally, the color of the mixed feature data is predicted through a lightweight network, and the three-dimensional scene of the target scene is reconstructed by the predicted color value, which can realize the scene (approximately 300×300m) in 1.5 hours. 2 ) and ensures high quality of the three-dimensional scene.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0155] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: The steps include: receiving a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene; Performing a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points; Performing spatial parameterization on the sampling space points to obtain spatial point parameters; Performing a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data; Performing a plane feature extraction operation on the spatial point parameters according to a high-resolution plane to obtain plane feature data, wherein the plane feature data is three orthogonally placed planes with a resolution of N and a feature dimension of F. For the queried three-dimensional point, first orthogonal projection is performed on these three planes, and then two-dimensional plane features are obtained by bilinear interpolation. The three interpolated features are then spliced ​​into a feature vector with a length of 3·F. Performing a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data; Inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the spatial point color; A target three-dimensional scene corresponding to the target scene is constructed according to the colors of the spatial points.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The plane feature data includes vertical plane data. After the step of performing a plane feature extraction operation on the spatial point parameters according to the high-resolution plane to obtain the plane feature data, the following steps are also included: A scaling operation is performed on the vertical plane data.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The step of inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the color of the spatial point specifically includes the following steps: The color prediction operation is performed on the mixed feature data according to a supervised loss function, wherein the supervised loss function is expressed as: Wherein, R represents the ray set of sampling points in the mixed feature data; Indicates the color of the spatial point; The true value representing the pixel color.

4. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The step of inputting the mixed feature data into a lightweight network to perform a color prediction operation to obtain the color of the spatial point specifically includes the following steps: Inputting the mixed feature data into the lightweight network to obtain a predicted color value; The predicted color values ​​are accumulated according to a volume rendering method to obtain the spatial point color.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that: The volume rendering method is expressed as: ； in, Indicates the interval distance between adjacent sampling points; Indicates the cumulative transparency of the calculation, the Expressed as: 。 6. A three-dimensional scene reconstruction device, characterized in that: include: a request receiving module, configured to receive a scene reconstruction request sent by a user terminal, wherein the scene reconstruction request includes a large-scale image corresponding to a target scene; A spatial point sampling module, configured to perform a spatial point sampling operation on the large-scale image according to a ray tracing method to obtain sampling spatial points; A spatial parameterization module, configured to perform spatial parameterization on the sampling spatial points to obtain spatial point parameters; A hash feature extraction module is used to perform a hash feature extraction operation on the spatial point parameters according to a multi-resolution hash grid to obtain hash feature data; A plane feature extraction module is used to perform a plane feature extraction operation on the spatial point parameters based on a high-resolution plane to obtain plane feature data, wherein the plane feature data is three orthogonally placed planes with a resolution of N and a feature dimension of F. For the queried three-dimensional point, orthogonal projection is first performed on these three planes, and then two-dimensional plane features are obtained by bilinear interpolation. The three interpolated features are then spliced ​​into a feature vector with a length of 3·F; A feature mixing module, configured to perform a feature mixing operation on the hash feature data and the plane feature data to obtain mixed feature data; A color prediction module is used to input the mixed feature data into a lightweight network to perform a color prediction operation to obtain the spatial point color; A construction module is used to construct a target three-dimensional scene corresponding to the target scene according to the color of the spatial point.

7. The three-dimensional scene reconstruction device according to claim 6, characterized in that: The plane feature data includes vertical plane data, and the device further includes: The scaling operation module is used to perform a scaling operation on the vertical plane data.

8. The three-dimensional scene reconstruction device according to claim 6, characterized in that: The color prediction module includes: A data input submodule, configured to input the mixed feature data into the lightweight network to obtain a predicted color value; The accumulation operation submodule is used to perform an accumulation operation on the predicted color value according to a volume rendering method to obtain the color of the spatial point.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 5 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 5.

Images