Real-time dynamic scene reconstruction method and system based on neural radiation field network

Abstract

The invention discloses a real-time dynamic scene reconstruction method and system based on a neural radiation field network, and belongs to the technical field of image data processing. The neural network hyper-parameters in the neural radiation field network are optimized by adopting an intelligent optimization algorithm, and the optimized neural radiation field network is determined, so that the global optimization capability and the optimization precision degree of the neural radiation field network can be effectively improved; the technical problem that in the prior art, the real-time dynamic scene reconstruction effect is poor is solved, meanwhile, the real-time image data serve as the input of the optimized neural radiation field network at the preset data sampling frequency, and real-time dynamic three-dimensional scene reconstruction is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and particularly relates to a real-time dynamic scene reconstruction method and system based on a neural radiance field network. Background Art

[0002] The Neural Radiance Fields (NeRF) is a computer vision model based on deep learning and volume rendering technology, aiming to generate high-quality three-dimensional scenes from multi-view 2D images. Its core idea is to implicitly learn the radiance field of the three-dimensional scene through a multi-layer perceptron (MLP), so as to synthesize images from new viewpoints. The input of NeRF includes multiple images and their corresponding camera parameters, and the output is a continuous three-dimensional radiance field, representing the color and density information of each point in the scene. This technology is widely used in fields such as 3D modeling, virtual reality, and augmented reality, with highly realistic three-dimensional reconstruction effects, but requires high computing resources for training. When using the neural radiance field network, it is generally necessary to first train it with sample data so that the neural radiance field network can achieve three-dimensional reconstruction. Existing technologies generally optimize the hyperparameters of the neural radiance field network through optimization algorithms such as gradient descent and Adam. Although certain optimization effects can be achieved, the optimization process is extremely prone to falling into local optima, resulting in the difficulty for the optimized neural radiance field network to accurately complete the three-dimensional reconstruction task, and causing the reconstruction effect of the neural radiance field network to be worse when dealing with real-time dynamic scenes. Summary of the Invention

[0003] The present invention provides a real-time dynamic scene reconstruction method and system based on a neural radiance field network to solve the technical problems existing in the prior art.

[0004] On the one hand, the present invention provides a real-time dynamic scene reconstruction method based on a neural radiance field network, including: Obtaining image sample data input by a staff through human-computer interaction and real color label data corresponding to the image sample data; Based on the image sample data and the real color label data corresponding to the image sample data, using an intelligent optimization algorithm to optimize the neural network hyperparameters in the neural radiance field network to determine the optimized neural radiance field network; Based on a preset data sampling frequency, collecting real-time image data and using the real-time image data as the input of the optimized neural radiance field network for real-time dynamic three-dimensional scene reconstruction.

[0005] In a possible implementation manner, the image sample data includes: depth information, sampling point positions, camera poses, and embedding vectors corresponding to a set of images of an object to be reconstructed.

[0006] In a possible implementation manner, based on the image sample data and the true color label data corresponding to the image sample data, an intelligent optimization algorithm is used to optimize the neural network hyperparameters in the neural radiance field network, and the optimized neural radiance field network is determined, including: Initialize the neural network hyperparameters in the neural radiance field network to determine multiple individuals; Construct a loss function, and based on the image sample data and the true color label data corresponding to the image sample data, use the loss function to obtain the loss function value corresponding to each individual; According to the loss function value corresponding to each individual, determine the optimal individual, the worst individual in the current optimization process, and the historical optimal position information corresponding to each individual; Based on the optimal individual in the current optimization process and the historical optimal position information corresponding to each individual, use an adaptive hybrid learning mechanism to optimize the individuals once to obtain the individuals after the first optimization; Based on the optimal individual in the current optimization process, use a multi-directional attenuation learning mechanism to perform a second optimization on the individuals after the first optimization to obtain the individuals after the second optimization; Based on the worst individual in the current optimization process, use a mutually exclusive acceleration learning mechanism to perform a third optimization on the individuals after the second optimization to obtain the individuals after the third optimization; Based on the individuals after the third optimization, determine the search center position, and according to the search center position, use a multi-information fusion mutation mechanism to perform a fourth optimization on the individuals after the third optimization to obtain the individuals after the fourth optimization; Repeat the process from the first optimization to the fourth optimization until the total number of optimizations reaches the maximum number of optimizations. Then, based on the individuals after the fourth optimization in the last optimization process, determine the optimized neural radiance field network.

[0007] In a possible implementation manner, initializing the neural network hyperparameters in the neural radiance field network to determine multiple individuals includes: For any hyperparameter of the neural network in the neural radiance field network, randomly initialize it between the upper limit and the lower limit corresponding to the hyperparameter, and encode the initialized hyperparameter as a vector to obtain an individual; After repeating the initialization and encoding multiple times, multiple individuals are obtained.

[0008] In a possible implementation manner, constructing a loss function includes: jointly constructing a loss function using color loss and depth loss.

[0009] In a possible implementation manner, based on the optimal individual of the current optimization process and the historical optimal position information corresponding to each individual, an adaptive hybrid learning mechanism is adopted to optimize the individual once to obtain the individual after the first optimization, including: Determine the hybrid learning rate control factor and the hybrid learning direction control factor; According to the hybrid learning rate control factor and the hybrid learning direction control factor, learn from the optimal individual of the current optimization process to determine the optimal learning information; According to the hybrid learning rate control factor and the hybrid learning direction control factor, learn from the historical optimal position information corresponding to the individual to determine the historical learning information; According to the historical learning information and the optimal learning information corresponding to the individual, optimize the individual once to determine the individual after the first optimization.

[0010] In a possible implementation manner, based on the optimal individual of the current optimization process, a multi-directional decay learning mechanism is adopted to perform a second optimization on the individual after the first optimization to obtain the individual after the second optimization, including: Based on the current number of optimization times, determine the adaptive decay weight factor and the adaptive decay learning factor; According to the adaptive decay weight factor, combined with the direction adjustment function, adjust the historical optimization speed to determine the adjusted historical optimization speed; According to the adaptive decay learning factor, learn from the optimal individual of the current optimization process to determine the adaptive decay learning information; According to the adjusted historical optimization speed and the adaptive decay learning information, perform a second optimization on the individual after the first optimization to obtain the individual after the second optimization.

[0011] In a possible implementation manner, based on the worst individual of the current optimization process, a mutually exclusive acceleration learning mechanism is adopted to perform a third optimization on the individual after the second optimization to obtain the individual after the third optimization, including: Randomly match an other individual for the individual after the second optimization to obtain the matching individual corresponding to the individual after the second optimization; Make the individual after the second optimization search for the position of the matching individual in the solution space, and at the same time move away from the position of the worst individual of the current optimization process in the solution space, so as to perform a third optimization on the individual after the second optimization to obtain the individual after the third optimization.

[0012] In a possible implementation manner, based on the individuals after three optimizations, the search center position is determined, and based on the search center position, a multi-information fusion mutation mechanism is adopted to perform four optimizations on the individuals after three optimizations to obtain the individuals after four optimizations, including: For any individual after three optimizations, three other individuals are randomly matched for the individual to obtain a first mutation information fusion individual, a second mutation information fusion individual, and a third mutation information fusion individual; Based on the first mutation information fusion individual, the second mutation information fusion individual, and the third mutation information fusion individual, the fusion mutation information is determined; Based on the individuals after three optimizations, the search center position is determined; Based on the search center position and the fusion mutation information, the individuals after three optimizations are subjected to four optimizations to obtain the individuals after four optimizations.

[0013] On the other hand, the present invention provides a real-time dynamic scene reconstruction system based on a neural radiance field network, including: a data acquisition module, a data optimization module, and a dynamic reconstruction module; The data acquisition module is used to acquire the image sample data input by the staff through human-computer interaction and the real color label data corresponding to the image sample data; The data optimization module is used to optimize the neural network hyperparameters in the neural radiance field network by using an intelligent optimization algorithm based on the image sample data and the real color label data corresponding to the image sample data, and determine the optimized neural radiance field network; The dynamic reconstruction module is used to collect real-time image data based on a preset data sampling frequency, and use the real-time image data as the input of the optimized neural radiance field network to perform real-time dynamic three-dimensional scene reconstruction.

[0014] A real-time dynamic scene reconstruction method and system provided by the present invention, by optimizing the neural network hyperparameters in the neural radiance field network by using an intelligent optimization algorithm based on the image sample data and the real color label data corresponding to the image sample data, can effectively improve the global optimization ability and optimization fineness of the neural radiance field network, avoid the technical problem of poor real-time dynamic scene reconstruction effect caused by the prior art, and at the same time use the real-time image data as the input of the optimized neural radiance field network at a preset data sampling frequency to realize real-time dynamic three-dimensional scene reconstruction. Description of the Drawings

[0015] The accompanying drawings here are incorporated into and constitute a part of this specification, showing embodiments consistent with the present invention and, together with the specification, are used to explain the principles of the present invention.

[0016] Figure 1 It is a flowchart of a real-time dynamic scene reconstruction method based on a neural radiance field network provided for an embodiment of the present invention.

[0017] Figure 2 It is a schematic structural diagram of a real-time dynamic scene reconstruction system based on a neural radiance field network provided for an embodiment of the present invention.

[0018] Through the above accompanying drawings, specific embodiments of the present invention have been shown, and there will be a more detailed description hereinafter. These drawings and the written description are not intended to limit the scope of the inventive concept in any way, but to illustrate the concept of the present invention to those skilled in the art by reference to specific embodiments. Detailed Embodiments

[0019] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0020] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] As Figure 1 shown, the embodiments of the present invention provide a real-time dynamic scene reconstruction method based on a neural radiance field network, including: S101. Obtain the image sample data input by a staff member through human-computer interaction and the corresponding true color label data of the image sample data; In a possible implementation manner, the image sample data includes: depth information, sampling point positions, camera poses, and embedding vectors corresponding to a set of images of an object to be reconstructed.

[0022] Depth information: The depth map can be obtained through a depth estimation network, and the sparse point cloud is obtained by the SFM method (COLMAP can also be used). Sampling point position: The position of the sampling point is obtained by Gaussian sampling based on depth, and the obtained position is fed into an MLP (Multi-Layer Perceptron) network for optimization after position encoding. Camera pose: Feature points are extracted from images taken from multiple perspectives by the SFM method, and feature matching is performed between images. The camera pose during shooting is obtained through the corresponding relationship of the feature points in space. In the few-shot scenario, no position encoding is performed on the pose information, and position encoding is only performed on the pose information in the sufficient-sample scenario. This is because position encoding of the pose in the few-shot scenario will lead to overfitting. Embedding vector: Each image has an embedding vector used to simulate the lighting conditions of the image. The dimension of this vector can be set through hyperparameters, and the dimension of the vector can be set to 48.

[0023] S102. Based on the image sample data and the corresponding true color label data of the image sample data, use an intelligent optimization algorithm to optimize the neural network hyperparameters in the neural radiance field network, and determine the optimized neural radiance field network. For example, optimize the hyperparameters of the MLP network in the neural radiance field network. More specifically, the connection weights and threshold parameters of the MLP network can be optimized to achieve the optimization of the neural radiance field network.

[0024] In the prior art, the hyperparameters of the neural radiance field network are generally optimized by optimization algorithms such as gradient descent and Adam. Although certain optimization effects can be achieved, the optimization process is extremely likely to fall into local optima, resulting in the difficulty for the optimized neural radiance field network to accurately complete the 3D reconstruction task, and the reconstruction effect is even worse when the neural radiance field network is used for real-time dynamic scenes. Therefore, the embodiments of the present invention use an intelligent optimization algorithm to optimize the neural network hyperparameters in the neural radiance field network, which can effectively solve the technical problems existing in the prior art and improve the reconstruction effect of real-time dynamic scenes.

[0025] S103. Based on a preset data sampling frequency, collect real-time image data, and use the real-time image data as the input of the optimized neural radiance field network to perform real-time dynamic 3D scene reconstruction.

[0026] It should be noted that the real-time image data and the image sample data should be of the same type and structure, so that the optimized neural radiance field network can successfully complete the 3D reconstruction task.

[0027] A real-time dynamic scene reconstruction method based on a neural radiance field network provided by the present invention, based on the image sample data and the corresponding real color label data of the image sample data, uses an intelligent optimization algorithm to optimize the neural network hyperparameters in the neural radiance field network, and determines the optimized neural radiance field network, which can effectively improve the global optimization ability and optimization fineness of the neural radiance field network, avoid the technical problem of poor real-time dynamic scene reconstruction effect caused by the prior art, and at the same time uses the real-time image data as the input of the optimized neural radiance field network at a preset data sampling frequency to realize real-time dynamic three-dimensional scene reconstruction.

[0028] In a possible implementation manner, based on the image sample data and the corresponding real color label data of the image sample data, using an intelligent optimization algorithm to optimize the neural network hyperparameters in the neural radiance field network, and determining the optimized neural radiance field network, includes: Initialize the neural network hyperparameters in the neural radiance field network to determine multiple individuals; Construct a loss function, and based on the image sample data and the corresponding real color label data of the image sample data, use the loss function to obtain the loss function value corresponding to each individual; According to the loss function value corresponding to each individual, determine the optimal individual, the worst individual in the current optimization process, and the historical optimal position information corresponding to each individual; Based on the optimal individual in the current optimization process and the historical optimal position information corresponding to each individual, use an adaptive hybrid learning mechanism to optimize the individuals once to obtain the individuals after the first optimization; Based on the optimal individual in the current optimization process, use a multi-directional attenuation learning mechanism to optimize the individuals after the first optimization twice to obtain the individuals after the second optimization; Based on the worst individual in the current optimization process, use a mutually exclusive acceleration learning mechanism to optimize the individuals after the second optimization three times to obtain the individuals after the third optimization; Based on the individuals after the third optimization, determine the search center position, and according to the search center position, use a multi-information fusion mutation mechanism to optimize the individuals after the third optimization four times to obtain the individuals after the fourth optimization; Repeat the process of the first optimization to the fourth optimization until the total number of optimizations reaches the maximum number of optimizations. Then, according to the individuals after the fourth optimization in the last optimization process, determine the optimized neural radiance field network. For example, the optimal individual can be re-determined according to the individuals after the fourth optimization in the last optimization process, and the hyperparameters included in the optimal individual can be used as the final hyperparameters to obtain the optimized neural radiance field network.

[0029] Optionally, after each optimization, the hyperparameters in the individual can be processed to prevent them from going out of bounds, ensuring that the hyperparameters always remain between their corresponding upper and lower limits. For example, an out-of-bounds hyperparameter can be randomly reset between the upper and lower limits, or it can be reset to the nearest limit value (e.g., when the out-of-bounds hyperparameter is closer to the upper limit, it can be reset to the upper limit).

[0030] In a possible implementation, the neural network hyperparameters in the neural radiance field network are initialized to determine multiple individuals, including: For any hyperparameter of the neural network in the neural radiance field network, it is randomly initialized between the upper and lower limits corresponding to the hyperparameter, and the hyperparameter after initialization is encoded as a vector to obtain an individual; After repeatedly initializing and encoding multiple times, multiple individuals are obtained.

[0031] In a possible implementation, a loss function is constructed, including: jointly constructing the loss function using color loss and depth loss.

[0032] It should be noted that it is also possible to construct the loss function only using color loss. The loss function is a relatively mature technology in the neural radiance field network. In the embodiments of the present invention, it is only used without improvement. Existing loss functions can be used here and will not be elaborated further.

[0033] In a possible implementation, based on the optimal individual in the current optimization process and the historical optimal position information corresponding to each individual, an adaptive hybrid learning mechanism is used to optimize the individual once to obtain the individual after one optimization, including: Determine the hybrid learning rate control factor and the hybrid learning direction control factor as:

[0034]

[0035] Among them, represents the hybrid learning rate control factor, t represents the current number of optimizations, T represents the preset maximum number of optimizations, represents a constant value, set to 2; represents the hybrid learning direction control factor, represents a random number between (0, 2).

[0036] According to the hybrid learning rate control factor and the hybrid learning direction control factor, learn from the optimal individual in the current optimization process to determine the optimal learning information as ; among them, represents the learning degree control factor, and , represents the preset maximum value of the learning degree control factor, represents the preset minimum value of the learning degree control factor, and the learning degree control factor is always between (0, 1), represents the optimal individual, represents the t th i individual in the i th optimization process, where = 1, 2, …, M, and M represents the total number of individuals; Learn the historical optimal position information corresponding to the individual according to the hybrid learning rate control factor and the hybrid learning direction control factor, and determine the historical learning information as ; where represents the historical optimal position information corresponding to the individual ; Optimize the individual once according to the historical learning information and the optimal learning information corresponding to the individual, and determine the individual after one optimization as: , ; where represents the hybrid optimization speed corresponding to the t th i individual in the th optimization process, t represents the hybrid optimization speed corresponding to the i +1 th individual in the optimization process, sin represents the sine function, cos represents the cosine function, represents the individual after one optimization The adaptive hybrid learning mechanism provided by the embodiments of the present invention can remember more of its own positions in the early stage of the algorithm and more optimal positions in the later stage of the algorithm, thereby effectively improving the convergence accuracy of the algorithm. And by introducing the hybrid learning rate control factor and the hybrid learning direction control factor, it can effectively guarantee the diversity of individuals, enable individuals to be more flexible when searching for the global optimal solution, improve the search ability of individuals, and further guide individuals to move in the direction of the global optimal solution.

[0037] In a possible implementation manner, based on the optimal individual in the current optimization process, perform a secondary optimization on the individual after one optimization by using a multi-directional decay learning mechanism to obtain the individual after the secondary optimization, including: Based on the current number of optimizations, determine the adaptive decay weight factor and the adaptive decay learning factor as:

[0038]

[0039] Among them, represents the adaptive decay weight factor, represents the preset minimum value of the adaptive decay weight factor, represents the preset maximum value of the adaptive decay weight factor, represents the pi, represents the constant term and is set to 0.448; represents the natural constant; a represents the first coefficient term, b represents the second coefficient term, represents the adaptive decay learning factor; According to the adaptive decay weight factor, combined with the direction adjustment function, adjust the historical optimization speed to determine the adjusted historical optimization speed; The direction adjustment function is set as: ; among them, R represents the direction adjustment control factor; The adjusted historical optimization speed is: ; among them, represents the t th optimization process, the optimization speed corresponding to the j th individual after the first optimization, j = 1, 2, …, M, where M represents the total number of individuals; According to the adaptive decay learning factor, learn from the optimal individual in the current optimization process to determine the adaptive decay learning information as ; among them, represents the t th optimization process, the j th individual after the first optimization, represents the first random number between (0, 1); According to the adjusted historical optimization speed and the adaptive decay learning information, perform a secondary optimization on the individual after the first optimization to obtain the individual after the secondary optimization as: , ; among them, represents the optimization speed corresponding to the t +1th optimization process, the j th individual after the first optimization, represents the individual after the secondary optimization .

[0040] The multi-directional attenuation learning mechanism provided by the embodiments of the present invention introduces an adaptive attenuation weight factor and an adaptive attenuation learning factor, making the algorithm slow in the early stage, faster in the middle stage, and gradually flattening in the later stage. The improved algorithm dynamically balances the global search ability in the early stage and the local search ability in the middle and later stages, which can prevent the optimization algorithm from falling into the local optimum prematurely and improve the optimization accuracy.

[0041] In a possible implementation manner, based on the worst individual in the current optimization process, a mutually exclusive accelerated learning mechanism is used to perform a third optimization on the individual after the second optimization to obtain the individual after the third optimization, including: Randomly match an other individual for the individual after the second optimization to obtain the matching individual corresponding to the individual after the second optimization; Enable the individual after the second optimization to search for the position of the matching individual in the solution space, and at the same time move away from the position of the worst individual in the current optimization process in the solution space, so as to realize the third optimization of the individual after the second optimization, and the individual after the third optimization obtained is: ; where represents the t th individual after the second optimization in the k th optimization process, k = 1, 2,..., M, M represents the total number of individuals, represents the individual corresponding matching individual, represents the second random number between (0, 1), represents the worst individual, represents the individual after the third optimization .

[0042] The mutually exclusive accelerated learning mechanism provided by the embodiments of the present invention can enable the individual to learn random position information while escaping from the worst position, thereby realizing the update of the position, improving the diversity of the individuals, and enabling the algorithm to jump out of the local optimum solution while accelerating the algorithm optimization.

[0043] In a possible implementation manner, based on the individual after the third optimization, determine the search center position, and according to the search center position, use a multi-information fusion mutation mechanism to perform a fourth optimization on the individual after the third optimization to obtain the individual after the fourth optimization, including: For any individual after the third optimization, randomly match three other individuals for the individual to obtain the first mutation information fusion individual, the second mutation information fusion individual, and the third mutation information fusion individual; According to the first mutation information fusion individual, the second mutation information fusion individual, and the third mutation information fusion individual, determine the fusion mutation information as ; where Represents the fusion mutation control factor between (0, 1). Represents the scaling factor between (0, 2). Represents the first individual for fusing mutation information. Represents the second individual for fusing mutation information. Represents the third individual for fusing mutation information; Based on the individuals after three optimizations, determine the search center position; each element in each dimension of the search center position is substantially the mean value of the elements of all individuals in that dimension; According to the search center position and the fusion mutation information, perform a fourth optimization on the individuals after three optimizations, and the individuals after the fourth optimization are obtained as: ; where Represents the m th individual after three optimizations during the t-th optimization process, m = 1, 2, …, M, where M represents the total number of individuals, Represents the individual after the fourth optimization , Represents the third random number between (0, 1), Represents the search center position.

[0044] The multi-information fusion mutation mechanism provided by the embodiments of the present invention can adjust the position by randomly selecting individuals in the solution space, thereby enhancing the diversity of the population and the global exploration ability, and thus being able to avoid falling into local optimality while ensuring the convergence speed of the algorithm.

[0045] Through the mutual cooperation of the above several mechanisms, the optimization of the neural network in the neural radiance field network is realized. Compared with the existing conventional gradient optimization methods, it has stronger global search ability and fine search ability, can effectively improve the learning effect on data, and enables the optimized neural radiance field network to reconstruct three-dimensional scenes more accurately.

[0046] As Figure 2 shown, the embodiments of the present invention provide a real-time dynamic scene reconstruction system based on a neural radiance field network, including: a data acquisition module 201, a data optimization module 202, and a dynamic reconstruction module 203; The data acquisition module 201 is used to acquire the image sample data input by the staff through human-computer interaction and the corresponding real color label data of the image sample data; The data optimization module 202 is used to optimize the neural network hyperparameters in the neural radiance field network based on the image sample data and the corresponding real color label data of the image sample data, and determine the optimized neural radiance field network; The dynamic reconstruction module 203 is configured to collect real-time image data based on a preset data sampling frequency, and use the real-time image data as the input of the optimized neural radiance field network for real-time dynamic three-dimensional scene reconstruction.

[0047] The embodiment of the present invention provides a real-time dynamic scene reconstruction system based on a neural radiance field network, which can execute the above method technical solution. The principle and beneficial effects are similar, and will not be elaborated here.

[0048] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0050] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0051] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0052] Those of ordinary skill in the art can understand that all or part of the steps in implementing the above facts and methods can be completed by instructing relevant hardware through a program. The program involved or the said program can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: At this time, the corresponding method steps are introduced. The said storage medium can be ROM / RAM, magnetic disk, optical disc, etc.

[0053] The specific embodiments described above have further elaborated on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time dynamic scene reconstruction method based on a neural radiation field network, characterized in that: include: Obtaining image sample data input by a staff member through human-computer interaction and real color label data corresponding to the image sample data; Based on the image sample data and the real color label data corresponding to the image sample data, an intelligent optimization algorithm is used to optimize the neural network hyperparameters in the neural radiation field network to determine the optimized neural radiation field network; Based on a preset data sampling frequency, real-time image data is collected, and the real-time image data is used as input to the optimized neural radiation field network to perform real-time dynamic three-dimensional scene reconstruction.

2. The real-time dynamic scene reconstruction method based on the neural radiation field network according to claim 1 is characterized in that: The image sample data includes: depth information, sampling point positions, camera poses and embedding vectors corresponding to a set of images of objects to be reconstructed.

3. The real-time dynamic scene reconstruction method based on the neural radiation field network according to claim 1 is characterized in that: Based on the image sample data and the real color label data corresponding to the image sample data, an intelligent optimization algorithm is used to optimize the neural network hyperparameters in the neural radiation field network to determine the optimized neural radiation field network, including: Initializing the hyperparameters of the neural network in the neural radiation field network and determining a plurality of individuals; Constructing a loss function, and based on the image sample data and the real color label data corresponding to the image sample data, using the loss function to obtain the loss function value corresponding to each individual; According to the loss function value corresponding to each individual, determine the best individual, the worst individual and the historical optimal position information corresponding to each individual in the current optimization process; Based on the optimal individual of the current optimization process and the historical optimal position information corresponding to each individual, an adaptive hybrid learning mechanism is used to optimize the individual once to obtain an optimized individual; Based on the optimal individual of the current optimization process, a multi-directional attenuation learning mechanism is used to perform a second optimization on the individual after the first optimization to obtain the individual after the second optimization; Based on the worst individual in the current optimization process, the individual after the second optimization is optimized three times by using a mutually exclusive acceleration learning mechanism to obtain an individual after the third optimization; Based on the individuals after three optimizations, the search center position is determined, and according to the search center position, the individuals after three optimizations are optimized four times by using a multi-information fusion mutation mechanism to obtain the individuals after four optimizations; The optimization process is repeated from one to four times until the total number of optimizations reaches the maximum number of optimizations. Then, the optimized neural radiation field network is determined based on the individual after four optimizations in the last optimization process.

4. The real-time dynamic scene reconstruction method based on the neural radiation field network according to claim 3 is characterized in that: Initialize the neural network hyperparameters in the neural radiation field network and determine multiple individuals, including: For any hyperparameter of the neural network in the neural radiation field network, randomly initialize it between the upper limit and the lower limit corresponding to the hyperparameter, and encode the initialized hyperparameter into a vector to obtain an individual; After repeated initialization and encoding, multiple individuals are obtained.

5. The real-time dynamic scene reconstruction method based on neural radiation field network according to claim 3 is characterized in that: Constructing a loss function, including: using color loss and depth loss to jointly construct a loss function.

6. The real-time dynamic scene reconstruction method based on neural radiation field network according to claim 5 is characterized in that: Based on the optimal individual of the current optimization process and the historical optimal position information corresponding to each individual, an adaptive hybrid learning mechanism is used to optimize the individual once to obtain the optimized individual, including: Determine the hybrid learning rate control factor and the hybrid learning direction control factor; According to the hybrid learning rate control factor and the hybrid learning direction control factor, the optimal individual of the current optimization process is learned to determine the optimal learning information; According to the hybrid learning rate control factor and the hybrid learning direction control factor, the historical optimal position information corresponding to the individual is learned to determine the historical learning information; According to the historical learning information and optimal learning information corresponding to the individual, the individual is optimized once, and the individual after the optimization is determined.

7. The real-time dynamic scene reconstruction method based on neural radiation field network according to claim 6 is characterized in that: Based on the optimal individual of the current optimization process, a multi-directional attenuation learning mechanism is used to perform a secondary optimization on the individual after the primary optimization to obtain the individual after the secondary optimization, including: Based on the current number of optimizations, determine the adaptive attenuation weight factor and the adaptive attenuation learning factor; According to the adaptive attenuation weight factor, in combination with the direction adjustment function, the historical optimization speed is adjusted to determine the adjusted historical optimization speed; According to the adaptive attenuation learning factor, the optimal individual of the current optimization process is learned to determine the adaptive attenuation learning information; According to the adjusted historical optimization speed and adaptive attenuation learning information, the individuals after the first optimization are optimized for a second time to obtain the individuals after the second optimization.

8. The real-time dynamic scene reconstruction method based on neural radiation field network according to claim 7 is characterized in that: Based on the worst individual in the current optimization process, the individuals after the second optimization are optimized three times by using a mutually exclusive acceleration learning mechanism to obtain individuals after the third optimization, including: Randomly match another individual to the second-optimized individual to obtain the matching individual corresponding to the second-optimized individual; The individuals after the second optimization are made to search for the positions of the matching individuals in the solution space, while being away from the positions of the worst individuals in the current optimization process in the solution space, so as to achieve the third optimization of the individuals after the second optimization and obtain the individuals after the third optimization.

9. The real-time dynamic scene reconstruction method based on neural radiation field network according to claim 3 is characterized in that: Based on the individuals after three optimizations, the search center position is determined, and according to the search center position, the individuals after three optimizations are optimized four times using a multi-information fusion mutation mechanism to obtain the individuals after four optimizations, including: For any individual after three optimizations, three other individuals are randomly matched for the individual to obtain the first variation information fusion individual, the second variation information fusion individual and the third variation information fusion individual; Determine fusion variation information according to the first variation information fusion individual, the second variation information fusion individual and the third variation information fusion individual; Based on the individuals after three optimizations, the search center position is determined; According to the search center position and the fusion variation information, the individual after the three optimizations is optimized four times to obtain the individual after the four optimizations.

10. A real-time dynamic scene reconstruction system based on a neural radiation field network, characterized in that: include: Data acquisition module, data optimization module and dynamic reconstruction module; The data acquisition module is used to acquire the image sample data input by the staff through human-computer interaction and the real color label data corresponding to the image sample data; The data optimization module is used to optimize the neural network hyperparameters in the neural radiation field network using an intelligent optimization algorithm based on the image sample data and the real color label data corresponding to the image sample data, and determine the neural radiation field network after optimization; The dynamic reconstruction module is used to collect real-time image data based on a preset data sampling frequency, and use the real-time image data as input of the optimized neural radiation field network to perform real-time dynamic three-dimensional scene reconstruction.

Images