Complex illumination-based AAN-NeRF three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to an AAN-NeRF three-dimensional reconstruction method and device based on complex illumination, electronic equipment and a storage medium, and the method comprises the steps: obtaining the position information and visual angle information of each pixel point in each image in an image sequence after coding; inputting the coded view angle information of each pixel point into an attention mechanism for feature fusion to obtain fused view angle information; and inputting the coded position information of each pixel point and the fused visual angle information into an initial neural radiation field model for training, and constructing an implicit radiation field model of the target scene. According to the method, the position information and view angle information of each pixel point in each image in an image sequence are coded, the extracted position information and view angle information are converted into high-dimensional feature vectors, and the view angle information of each pixel point after coding is fused through a multilayer perceptron network with complexity reduced by a pruning technology, so that the image quality is improved. Redundant information is reduced, and generalization and robustness of the model are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, in particular to a complex illumination-based AAN-NeRF three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] 3D reconstruction, a technique for processing 2D data to generate the 3D geometry and appearance of objects or scenes, is widely used in fields such as cultural heritage digitization, virtual reality, and augmented reality. Common 3D reconstruction methods include multi-view image-based reconstruction, depth-based reconstruction, and deep learning-based reconstruction.

[0003] Traditional 3D reconstruction methods based on multi-view images rely on image data acquired from different viewpoints and perform poorly in scenes with weak textures, reflections, or complex lighting, and the generated models may lack details.

[0004] The 3D reconstruction method based on depth information is implemented through a depth sensor, which is limited by the effective range of the depth sensor and its dependence on environmental conditions.

[0005] Although existing 3D reconstruction methods based on neural radiance fields (NeRFs) perform well in scene reconstruction, they still suffer from problems of blurred reconstruction and loss of details when processing high-frequency areas with rich textures and rapidly changing colors in complex lighting scenes. They usually require large-scale computing resources and have low training efficiency. At the same time, their ability to process redundant data and parameters is limited, making it difficult to meet the needs of efficient reconstruction. Summary of the Invention

[0006] The present invention provides an AAN-NeRF three-dimensional reconstruction method, device, electronic device and storage medium based on complex illumination, which overcomes the shortcomings of the above-mentioned existing technologies and can effectively solve the problem that the existing three-dimensional reconstruction method based on neural radiation field (NeRF) cannot reconstruct high-frequency information in complex scenes.

[0007] One of the technical solutions of the present invention is achieved by the following measures: an AAN-NeRF three-dimensional reconstruction method based on complex illumination, comprising:

[0008] Obtaining the encoded position and viewing angle information of each pixel in each image in an image sequence, where the image sequence is composed of multiple RGB images of the target scene acquired under different lighting conditions;

[0009] The perspective information encoded by each pixel is input into the attention mechanism for feature fusion to obtain the fused perspective information;

[0010] The encoded position information of each pixel and the fused perspective information are input into the initial neural radiation field model for training to obtain the color, density and light intensity control information of each pixel, and construct an implicit radiation field model of the target scene.

[0011] The following are further optimizations and / or improvements to the above technical solutions:

[0012] The above-mentioned acquisition of the encoded position information and viewing angle information of each pixel in each image in the image sequence includes:

[0013] Combining ray tracing technology with a progressive cone sampling method, the image sequence is coarsely and finely sampled to obtain the position and viewing angle information of each pixel in each image in the image sequence, where the image sequence is composed of multiple RGB images of the target scene acquired under different lighting conditions.

[0014] Combined with the high-frequency information position of the image sequence, the integrated position coding technology is applied to encode the position information and viewing angle information of each pixel to obtain the encoded position information and viewing angle information.

[0015] The process of obtaining the high-frequency information position of the above image sequence includes:

[0016] Get the position representation of each pixel in each image in the image sequence;

[0017] The position representation of each pixel is mapped into a high-dimensional space to form a new representation, namely the high-frequency information position of the image sequence.

[0018] The above attention mechanism is a multi-layer perceptron network that uses pruning technology to reduce network complexity. The pruning process is as follows:

[0019]

[0020] Where W is the weight of the linear layer, is the pruning threshold calculated based on the specified pruning ratio; W pruned Is the pruned weight matrix. If the absolute value of the linear layer |W| is greater than the pruning threshold, the weight is retained. If the absolute value of the linear layer |W| is less than or equal to the pruning threshold, the weight is pruned and set to zero.

[0021] The implicit radiation field model based on the target scene applies the volume rendering method to synthesize the target scene's perspective, generate 2D images of any unknown perspective, and complete the 3D reconstruction of the target scene through the synthesized 2D images.

[0022] The second technical solution of the present invention is achieved by the following measures: an AAN-NeRF three-dimensional reconstruction device based on complex illumination, comprising:

[0023] an information acquisition unit for acquiring encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions;

[0024] The encoding unit inputs the perspective information encoded by each pixel into the attention mechanism for feature fusion to obtain the fused perspective information;

[0025] The training unit inputs the encoded position information of each pixel and the fused perspective information into the initial neural radiation field model for training, obtains the color, density and light intensity control information of each pixel, and constructs the implicit radiation field model of the target scene.

[0026] The following are further optimizations and / or improvements to the above technical solutions:

[0027] The reconstruction unit, based on the implicit radiation field model of the target scene, introduces a loss function and applies a volume rendering method to perform perspective synthesis on the target scene, generating a 2D image of any unknown perspective. The 3D reconstruction of the target scene is completed through the 2D image synthesized from the perspective.

[0028] The third technical solution of the present invention is achieved through the following measures: an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the steps in the AAN-NeRF three-dimensional reconstruction method based on complex lighting.

[0029] The fourth technical solution of the present invention is achieved through the following measures: a storage medium, on which a computer program that can be read by a computer is stored, and the computer program is configured to execute the steps in the AAN-NeRF three-dimensional reconstruction method based on complex illumination when running.

[0030] The present invention improves the light path simulation process by introducing frustum projection technology, and combines the dynamic capture mechanism of multi-view features to collect the position information and perspective information of each pixel point in each image in the image sequence. By encoding the position information and perspective information of each pixel point in each image in the image sequence, the extracted position information and perspective information are converted into a higher-dimensional feature vector. The multi-layer perceptron network with reduced complexity through pruning technology is used to fuse the encoded perspective information of each pixel point, reduce redundant information, and enhance the generalization and robustness of the model. This solves the problem that the current three-dimensional reconstruction method based on neural radiation field is difficult to accurately reconstruct high-frequency information, and realizes high-precision three-dimensional reconstruction of the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Attachment Figure 1A schematic flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention.

[0032] Attachment Figure 2 A schematic flow chart of another three-dimensional reconstruction method provided by an embodiment of the present invention.

[0033] Attachment Figure 3 A schematic flow chart of a method for encoding position information and viewing angle information provided by an embodiment of the present invention.

[0034] Attachment Figure 4 A schematic diagram of the structure used in the progressive cone sampling method provided in an embodiment of the present invention.

[0035] Attachment Figure 5 A schematic diagram for comparing reconstruction effects provided by an embodiment of the present invention.

[0036] Attachment Figure 6 A schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of the present invention.

[0037] Attachment Figure 7 A schematic structural diagram of another 3D reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention is not limited to the following embodiments, and specific implementation methods can be determined based on the technical solutions of the present invention and actual conditions.

[0039] Those skilled in the art will understand that, unless otherwise stated, in the embodiments of the present invention, a "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] In addition, in the embodiments of the present invention, “plurality” refers to two or more than two, and “first” and “second” are used for distinguishing descriptions and should not be understood as implying relative importance.

[0041] Embodiments of the present invention provide a complex illumination-based AAN-NeRF 3D reconstruction method, apparatus, electronic device, and storage medium. This complex illumination-based AAN-NeRF 3D reconstruction apparatus can be integrated into a computer device, such as a server or a terminal. Alternatively, the apparatus can be jointly executed by a terminal and a server. The above examples should not be construed as limiting the present invention.

[0042] The above-mentioned terminals may include mobile phones, wearable smart devices, tablet computers, laptop computers, personal computers (PCs), and vehicle-mounted computers, etc., and the present invention does not impose any restrictions on this. The present invention does not impose any restrictions on the number of terminal devices.

[0043] The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. The present invention does not impose any restrictions on this.

[0044] For example, a computer device obtains the encoded position information and perspective information of each pixel in each image in an image sequence, where the image sequence is composed of multiple RGB images of a target scene obtained under different lighting conditions; the encoded perspective information of each pixel is input into an attention mechanism for feature fusion to obtain fused perspective information; the encoded position information and fused perspective information of each pixel are input into an initial neural radiation field model for training to obtain the color, density, and light intensity control information of each pixel, and to construct an implicit radiation field model of the target scene; based on the implicit radiation field model of the target scene, a volume rendering method is applied to perform perspective synthesis on the target scene to generate a 2D image of any unknown perspective, and the three-dimensional reconstruction of the target scene is completed through the perspective-synthesized 2D image.

[0045] Based on this, the technical solution of the present invention will be introduced and explained with reference to several examples below.

[0046] Example 1: As shown in the attached Figure 1 As shown, an embodiment of the present invention discloses an AAN-NeRF 3D reconstruction method based on complex illumination, comprising:

[0047] Step S110, obtaining encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions;

[0048] In the above steps, the acquisition of the image sequence includes reconstructing the target scene using a camera, shooting around the target in low-light to high-light environments, collecting multiple images with a size of 500x375 RGB images, and constructing an image sequence containing the target scene.

[0049] In the above steps, the position information and perspective information of each pixel in each image in the image sequence are encoded, thereby converting the extracted position information and perspective information into a higher-dimensional feature vector, which facilitates the subsequent initial neural radiation field model to accurately express high-frequency detail information.

[0050] Step S120: Input the encoded perspective information of each pixel into the attention mechanism for feature fusion to obtain fused perspective information;

[0051] In the above steps, the attention mechanism is introduced to fuse the perspective information encoded by each pixel point. The attention mechanism dynamically assigns weights, so that subsequent model training pays more attention to task-related information, reduces redundant information, thereby reducing the computational burden and enhancing the generalization and robustness of the model.

[0052] In the above steps, the attention mechanism is a multi-layer perceptron network that uses pruning technology to reduce network complexity. The pruning process is as follows:

[0053]

[0054] Where W is the weight of the linear layer, is the pruning threshold calculated based on the specified pruning ratio; W pruned Is the pruned weight matrix. If the absolute value of the linear layer |W| is greater than the pruning threshold, the weight is retained. If the absolute value of the linear layer |W| is less than or equal to the pruning threshold, the weight is pruned and set to zero.

[0055] In the above steps, the main expressions of fusion are as follows:

[0056]

[0057] The encoded view information is processed and mapped into query Q, key K and value V through linear transformation. The similarity between query and key is calculated to derive the attention matrix QK T , where K T Represents the transpose of the key vector. The dimension of the key vector is denoted as d, and √d is used for scaling to prevent the dot product result from becoming too large, thereby mitigating the risk of gradient vanishing caused by the Softmax function. Subsequently, the attention weights are calculated through Softmax normalization to ensure that the contributions of different features are appropriately weighted. The values V are then weighted according to the attention weights to obtain the weighted sum output. Finally, the output is mapped back to the same feature dimension as the input through a linear transformation and added to the original input to form a residual connection.

[0058] In step S130, the encoded position information of each pixel and the fused viewing angle information are input into the initial neural radiation field model for training to obtain the color, density and light intensity control information of each pixel, and construct an implicit radiation field model of the target scene.

[0059] In the above steps, the loss function used to optimize rendering during training includes four components: NeRF rendering loss L it-mse And three unsupervised brightness correction losses (Lde, Lco, Lcc). The loss function analyzes the loss between the rendered color and the true color. The specific overall training loss is shown below:

[0060]

[0061] Among them, λ1, λ2, and λ3 are three non-negative parameters for balancing the total loss weight, which can be set to 1e respectively. -3 、1e -3 、1e -8 .

[0062] An embodiment of the present invention discloses an AAN-NeRF 3D reconstruction method based on complex illumination. By encoding the position information and perspective information of each pixel point in each image in an image sequence, the extracted position information and perspective information are converted into a higher-dimensional feature vector. A multi-layer perceptron network with reduced complexity is used through pruning technology to fuse the perspective information encoded for each pixel point, thereby reducing redundant information and enhancing the generalization and robustness of the model. This solves the problem that the current 3D reconstruction method based on neural radiation field has difficulty in accurately reconstructing high-frequency information, and realizes high-precision 3D reconstruction of the target scene.

[0063] Example 2: As shown in the attached Figure 2 As shown, an embodiment of the present invention discloses an AAN-NeRF 3D reconstruction method based on complex illumination, comprising:

[0064] Step S210, obtaining the encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions;

[0065] Step S220: Input the encoded perspective information of each pixel into the attention mechanism for feature fusion to obtain fused perspective information;

[0066] Step S230: Input the encoded position information of each pixel and the fused viewing angle information into the initial neural radiation field model for training, obtain the color, density, and light intensity control information of each pixel, and construct an implicit radiation field model of the target scene;

[0067] Step S240 , based on the implicit radiation field model of the target scene, a volume rendering method is applied to perform perspective synthesis on the target scene to generate a 2D image of any unknown perspective, and the 3D reconstruction of the target scene is completed through the perspective-synthesized 2D image.

[0068] The above steps specifically include:

[0069] (1) After obtaining the implicit radiation field model of the target scene, ray tracing technology is used to perform pixel-by-pixel and layer-by-layer fine sampling along each ray for any unknown camera position and viewing angle. For these sampling points on the ray, the obtained implicit radiation field model is used to accurately extract the color, density, and light intensity control information corresponding to each sampling point;

[0070] (2) Use volume rendering to combine the color and density corresponding to each sampling point and calculate the RGB color value formed when the light passes through each pixel on the virtual imaging plane. The formula for volume rendering is:

[0071]

[0072] Among them, c i , σ i is the color and density of the pixel predicted by the implicit radiation field model, δ j =t i +1-t i is the distance between adjacent pixels, T i As shown below:

[0073]

[0074] (3) After volume rendering, the target view reconstruction of the target scene under the unknown new perspective can be obtained, thereby completing the three-dimensional reconstruction of the scene.

[0075] The comparison of the reconstruction results achieved by the existing 3D reconstruction method based on NeRF and this embodiment is shown in the attached figure. Figure 5 As shown, attached Figure 5 The two pictures on the left are the results of reconstruction using the 3D reconstruction method based on Neural Radiation Field (NeRF). Figure 5 The two figures on the right are the results of reconstruction using this embodiment. By comparison, it can be seen that the attached Figure 5 The two pictures on the left correspond to the fuzzy reconstruction of the high-frequency area in the red frame, while the two pictures on the side can reconstruct the high-frequency area well.

[0076] Example 3: As shown in the attached Figure 3 As shown, the embodiment of the present invention is a further optimization of the above embodiment, wherein obtaining the encoded position information and viewing angle information of each pixel point in each image in the image sequence includes:

[0077] Step S310: performing coarse sampling and fine sampling on the image sequence using a progressive cone sampling method in combination with ray tracing technology to obtain position information and viewing angle information of each pixel in each image in the image sequence, wherein the image sequence is composed of multiple RGB images of the target scene acquired under different lighting conditions;

[0078] In the above steps, the image sequence is coarsely and finely sampled using a progressive cone sampling method combined with ray tracing technology. Specifically, ray tracing is used to coarsely sample the scene's position coordinates in the 3D world coordinate system and to precisely sample the view coordinates along a cone from the camera's optical center. The view information encoded at each pixel is then fed into an attention mechanism for feature fusion. This pruning of the coarse and fine sampling results reduces the complexity of the multilayer perceptron network.

[0079] As attached Figure 4 As shown, the main expressions for the cone sampling using Gaussian distribution to approximate the frustum volume properties are as follows:

[0080]

[0081] Where μ is the center point of the conical frustum, hw is the half width of the conical frustum, t0 and t1 are the depth intervals of the light respectively;

[0082]

[0083] Among them, t mean is the Gaussian mean, which provides the expected depth value within the sampling range of the conical frustum and is used as the center position of the sampling point;

[0084]

[0085] Among them, t var is the Gaussian variance, used to model the depth range of the target area;

[0086]

[0087] Where r2 is the radius of the cone at the light source, r var is the radius-dependent variance to quantify the lateral diffusion properties of the frustum;

[0088] Step S320 , combining the high-frequency information position of the image sequence, and applying the integrated position coding technology to encode the position information and viewing angle information of each pixel to obtain encoded position information and viewing angle information.

[0089] In the above steps, the process of obtaining the high-frequency information position of the image sequence includes:

[0090] (1) Obtain the position representation (x, y, z) of each pixel in each image in the image sequence;

[0091] (2) The position representation of each pixel is mapped into a high-dimensional space to form a new representation, namely the high-frequency information position of the image sequence.

[0092] The new representation here may contain multiple dimensions, each of which corresponds to a high-frequency component of the original position information; these high-frequency components are optimized and used as input to the neural network to learn the radiation field of the target scene.

[0093] Example 4: As shown in the attached Figure 6 As shown, the embodiment of the present invention discloses an AAN-NeRF 3D reconstruction device based on complex illumination, including:

[0094] an information acquisition unit for acquiring encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions;

[0095] The encoding unit inputs the perspective information encoded by each pixel into the attention mechanism for feature fusion to obtain the fused perspective information;

[0096] The training unit inputs the encoded position information of each pixel and the fused perspective information into the initial neural radiation field model for training, obtains the color, density and light intensity control information of each pixel, and constructs the implicit radiation field model of the target scene.

[0097] Example 5: As shown in the attached Figure 7 As shown, the embodiment of the present invention discloses an AAN-NeRF 3D reconstruction device based on complex illumination, including:

[0098] an information acquisition unit for acquiring encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions;

[0099] The encoding unit inputs the perspective information encoded by each pixel into the attention mechanism for feature fusion to obtain the fused perspective information;

[0100] The training unit inputs the encoded position information of each pixel and the fused perspective information into the initial neural radiation field model for training, obtains the color, density and light intensity control information of each pixel, and constructs the implicit radiation field model of the target scene;

[0101] The reconstruction unit, based on the implicit radiation field model of the target scene, applies the volume rendering method to synthesize the perspective of the target scene, generates a 2D image of any unknown perspective, and completes the three-dimensional reconstruction of the target scene through the 2D image synthesized from the perspective.

[0102] Embodiment 6: The embodiment of the present invention discloses a storage medium, on which a computer program readable by a computer is stored. The computer program is configured to execute an AAN-NeRF three-dimensional reconstruction method based on complex illumination when running.

[0103] The above-mentioned storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory, a mobile hard disk, a magnetic disk, or an optical disk.

[0104] Example 7: An embodiment of the present invention discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement an AAN-NeRF three-dimensional reconstruction method based on complex illumination.

[0105] The processor may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. It may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. Memory may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memories, removable hard drives, magnetic disks, or optical disks.

[0106] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] The above content is only a specific implementation method of the present invention, which has strong adaptability and implementation effect, but the scope of protection of the present invention is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present invention.

Claims

1. A complex illumination-based AAN-NeRF 3D reconstruction method, characterized in that: include: Obtaining the encoded position and viewing angle information of each pixel in each image in an image sequence, where the image sequence is composed of multiple RGB images of the target scene acquired under different lighting conditions; The perspective information encoded by each pixel is input into the attention mechanism for feature fusion to obtain the fused perspective information; The encoded position information of each pixel and the fused perspective information are input into the initial neural radiation field model for training to obtain the color, density and light intensity control information of each pixel, and construct an implicit radiation field model of the target scene.

2. The AAN-NeRF 3D reconstruction method based on complex illumination according to claim 1, characterized in that: The step of obtaining the encoded position information and viewing angle information of each pixel in each image in the image sequence includes: Combining ray tracing technology with a progressive cone sampling method, the image sequence is coarsely and finely sampled to obtain the position and viewing angle information of each pixel in each image in the image sequence, where the image sequence is composed of multiple RGB images of the target scene acquired under different lighting conditions. Combined with the high-frequency information position of the image sequence, the integrated position coding technology is applied to encode the position information and viewing angle information of each pixel to obtain the encoded position information and viewing angle information.

3. The AAN-NeRF 3D reconstruction method based on complex illumination according to claim 2, characterized in that: The process of obtaining the high-frequency information position of the image sequence includes: Get the position representation of each pixel in each image in the image sequence; The position representation of each pixel is mapped into a high-dimensional space to form a new representation, namely the high-frequency information position of the image sequence.

4. The AAN-NeRF 3D reconstruction method based on complex illumination according to claim 1, 2 or 3, characterized in that: The attention mechanism is a multi-layer perceptron network that uses pruning technology to reduce network complexity. The pruning process is as follows: Where W is the weight of the linear layer, is the pruning threshold calculated based on the specified pruning ratio; W pruned Is the pruned weight matrix. If the absolute value of the linear layer |W| is greater than the pruning threshold, the weight is retained. If the absolute value of the linear layer |W| is less than or equal to the pruning threshold, the weight is pruned and set to zero.

5. The AAN-NeRF 3D reconstruction method based on complex illumination according to any one of claims 1 to 4, characterized in that: It also includes an implicit radiation field model based on the target scene, using a volume rendering method to synthesize the target scene's perspective, generating a 2D image of any unknown perspective, and completing the three-dimensional reconstruction of the target scene through the synthesized 2D image.

6. An application of the AAN-NeRF 3D reconstruction device based on complex illumination according to any one of claims 1 to 5, characterized in that: include: an information acquisition unit for acquiring encoded position information and viewing angle information of each pixel in each image in an image sequence, wherein the image sequence is composed of multiple RGB images of a target scene acquired under different lighting conditions; The encoding unit inputs the perspective information encoded by each pixel into the attention mechanism for feature fusion to obtain the fused perspective information; The training unit inputs the encoded position information of each pixel and the fused perspective information into the initial neural radiation field model for training, obtains the color, density and light intensity control information of each pixel, and constructs the implicit radiation field model of the target scene.

7. The AAN-NeRF 3D reconstruction device based on complex illumination according to claim 6, characterized in that: It also includes a reconstruction unit, which introduces a loss function and applies a volume rendering method to synthesize the perspective of the target scene based on the implicit radiation field model of the target scene, generates a 2D image of any unknown perspective, and completes the three-dimensional reconstruction of the target scene through the 2D image synthesized from the perspective.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the steps in the method according to any one of claims 1 to 5.

9. A storage medium, characterized in that: The storage medium stores a computer program that can be read by a computer, and the computer program is configured to execute the steps of the method according to any one of claims 1 to 5 when run.

Images