A 3D model reconstruction method based on a hybrid pulse neural network

Abstract

This invention discloses a 3D model reconstruction method based on a hybrid spiking neural network, comprising the following steps: acquiring a target image; preprocessing the target image to obtain an initial feature map; constructing and training a hybrid spiking neural network model to obtain a target model, wherein the target model includes several hybrid spiking layers, each including a mixing module, an MLP module, and a merging module; inputting the initial feature map into the mixing module for spatial position mixing, inputting the spatially mixed initial feature map into the MLP module for channel mixing, inputting the channel-mixed initial feature map into the merging module for image stitching to obtain a final feature map; and inputting the final feature map into a Nerf renderer to reconstruct the corresponding 3D model of the target image. This invention combines a hybrid spiking neural network with neural radiation fields, enabling accurate 3D model reconstruction with low resource consumption, demonstrating high practicality and facilitating further application.

Description

Technical Field

[0001] This invention relates to the field of 3D model reconstruction, and more particularly to a 3D model reconstruction method based on a hybrid spiking neural network. Background Technology

[0002] 3D model reconstruction, which involves reconstructing a 3D scene from a 2D image to generate a corresponding 3D model, is a core task in computer vision. Current technologies typically utilize neural radiance fields (Nerf) and multi-layer perceptrons (MLPs) to describe the color and density of each point in a 2D image from various viewpoints. For each pixel, a camera ray is used to traverse the scene and generate sampling points. Combining the ray direction, sampling location, and the color and density calculated by the MLP, volume rendering is used to obtain the corresponding 3D scene pixels from the original 2D image. By combining all the 3D scene pixels, the 3D scene can be reconstructed and the corresponding 3D model rebuilt.

[0003] However, although high-quality and realistic 3D models can be obtained through Nerf, its simple structural design makes it extremely demanding on input data. Otherwise, a lot of computing resources and time are required for training, resulting in poor overall practicality, high energy consumption, and uneconomical 3D model reconstruction technology in the current technology, which cannot be widely applied.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a 3D model reconstruction method based on a hybrid spiking neural network. By combining a hybrid spiking neural network with Nerf, this method solves the problems of poor overall practicality, high energy consumption, and uneconomical operation of existing 3D model reconstruction technologies, which hinder their further promotion and application.

[0006] The technical solution adopted by this invention to solve the problem is as follows:

[0007] In a first aspect, embodiments of the present invention provide a 3D model reconstruction method based on a hybrid spiking neural network, wherein the method includes:

[0008] Acquire the target image;

[0009] The target image is preprocessed to obtain an initial feature map;

[0010] A hybrid spiking neural network model is constructed and trained to obtain a target model. The target model includes several hybrid spiking layers, and each hybrid spiking layer includes a hybrid module, an MLP module, and a merging module.

[0011] The initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain the final feature map;

[0012] The final feature map is input into the Nerf renderer to reconstruct the 3D model corresponding to the target image.

[0013] In one implementation, after acquiring the target image, the process further includes:

[0014] Extract the pixel values ​​of each pixel in the target image;

[0015] The pixel value of each pixel in the target image is compared with the pixel value of its neighboring pixels. Pixels with different pixel values ​​from their neighboring pixels are assigned a value of 1, and pixels with the same pixel value as their neighboring pixels are assigned a value of 0, thus obtaining the boundary image of the object in the target image.

[0016] In one implementation, the target image is preprocessed to obtain an initial feature map, specifically including:

[0017] The target image and the boundary image are each divided into images of the same size;

[0018] The corresponding images in the target image and the boundary image are fused and embedded into an n-dimensional vector;

[0019] The initial feature map is obtained by combining the n-dimensional vectors.

[0020] In one implementation, a hybrid spiking neural network model is constructed, and the hybrid spiking neural network model is trained to obtain a target model, specifically including:

[0021] Construct a hybrid spiking neural network model with several initial LIF neurons, and determine the initial spiking threshold of the initial LIF neurons;

[0022] When the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, resets the initial LIF neuron, and determines the reset membrane potential and pulse threshold, and replaces the initial LIF neuron with the reset LIF neuron;

[0023] By applying gradient descent and repeatedly replacing LIF neurons until the full-precision information meets the requirements, the target model is obtained.

[0024] In one embodiment, after constructing a hybrid spiking neural network model with a plurality of initial LIF neurons, the method includes: connecting all the initial LIF neurons to form a group of neurons, wherein the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, resetting all the initial LIF neurons in the group of neurons, wherein the reset LIF neurons in the group of neurons have the same membrane potential and pulse threshold.

[0025] In one embodiment, the group of neurons includes LIF neurons connected and running in a vertical direction and LIF neurons connected and running in a horizontal direction.

[0026] In one implementation, the initial feature map is input into the mixing module for spatial position mixing, the spatially mixed initial feature map is input into the MLP module for channel mixing, and the channel-mixed initial feature map is input into the merging module for image stitching to obtain the final feature map, specifically including:

[0027] The initial feature map is input into the mixing module, which runs independently on each channel and communicates between different spatial locations to complete spatial location mixing.

[0028] The initial feature map after spatial location mixing is input into the MLP module, which independently operates on the spatial location and communicates between different channels to complete channel mixing;

[0029] The initial feature map after channel mixing is input into the merging module. The features of adjacent images in the initial feature map after channel mixing are separated into four channels. After merging, the number of channels is halved through a linear layer to complete the image stitching and obtain the final feature map.

[0030] In one implementation, the final feature map is input into a Nerf renderer to reconstruct a 3D model corresponding to the target image, specifically including:

[0031] Extract the position coordinates and viewing direction information of each pixel in the final feature map;

[0032] Based on the position coordinate information and the gaze direction information, the density and color of the corresponding pixels are generated through a multilayer perceptron;

[0033] Based on the density and color, and combined with the volume rendering equation, the cumulative light amount, voxel density and expected color of the corresponding pixel are rendered.

[0034] By combining the expected colors of all pixels, a 3D model corresponding to the target image is reconstructed.

[0035] Secondly, embodiments of the present invention also provide a 3D model reconstruction system based on a hybrid spiking neural network, the system comprising:

[0036] The image acquisition module is used to acquire the target image;

[0037] The preprocessing module is used to preprocess the target image to obtain an initial feature map;

[0038] The model training module is used to construct a hybrid spiking neural network model, train the hybrid spiking neural network model to obtain a target model, wherein the target model includes several hybrid spiking layers, and the hybrid spiking layers include a hybrid module, an MLP module and a merging module;

[0039] The image processing module is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain the final feature map;

[0040] The model reconstruction module is used to input the final feature map into the Nerf renderer to reconstruct a 3D model corresponding to the target image.

[0041] Thirdly, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a 3D model reconstruction program based on a hybrid spiking neural network. When the 3D model reconstruction program based on a hybrid spiking neural network is executed by a processor, it implements the steps of the 3D model reconstruction method based on a hybrid spiking neural network as described above.

[0042] This invention discloses a 3D model reconstruction method based on a hybrid spiking neural network, comprising the following steps: acquiring a target image; preprocessing the target image to obtain an initial feature map; constructing a hybrid spiking neural network model and training the model to obtain a target model, wherein the target model includes several hybrid spiking layers, each including a mixing module, an MLP module, and a merging module; inputting the initial feature map into the mixing module for spatial position mixing, inputting the spatially mixed initial feature map into the MLP module for channel mixing, inputting the channel-mixed initial feature map into the merging module for image stitching to obtain a final feature map; and inputting the final feature map into a Nerf renderer to reconstruct a 3D model corresponding to the target image. This invention combines a hybrid spiking neural network with neural radiation fields, enabling accurate 3D model reconstruction with low resource consumption, demonstrating high practicality and facilitating further application. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the steps of the 3D model reconstruction method based on a hybrid spiking neural network in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram comparing the target image and the boundary image in an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram comparing an unfused boundary image and a fused boundary image in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of LIF neurons in different directions in an embodiment of the present invention.

[0048] Figure 5 This is a flowchart illustrating the 3D model reconstruction method based on a hybrid spiking neural network in an embodiment of the present invention.

[0049] Figure 6 This is a flowchart illustrating another form of the 3D model reconstruction method based on a hybrid spiking neural network in this embodiment of the invention.

[0050] Figure 7 This is a schematic diagram comparing the transmit pulse rates of GN and LIF in an embodiment of the present invention.

[0051] Figure 8 This is a block diagram of a 3D model reconstruction system based on a hybrid spiking neural network in an embodiment of the present invention.

[0052] Figure 9 This is a block diagram of the smart terminal in an embodiment of the present invention. Detailed Implementation

[0053] This invention discloses a 3D model reconstruction method based on a hybrid spiking neural network. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0054] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0056] Neural radiance field (Nerf) is a deep learning model for modeling 3D implicit space. It can capture a scene from a known perspective without requiring intermediate 3D reconstruction. It can synthesize images from a new perspective based solely on pose intrinsic parameters and the original image, thus completing the 3D model reconstruction task. However, while Nerf has a simple overall structure, it requires significant computational resources and training time to obtain the necessary model for specific 3D model reconstruction tasks, resulting in poor practicality and hindering further expansion of applications.

[0057] Spiking Neural Networks (SNNs) are third-generation neural network models that excel at handling sparse data. Through LIF (Leaky Integrate and Fire) neurons, they can communicate between data points at different time steps, making them frequently used in neural computing, sensory processing, and event-driven systems. However, the training and inference algorithms for SNNs are highly complex, limiting their practical applications.

[0058] This invention combines the simulation effect of SNN on real biological neurons with the simplification effect of LIF neurons, using SNN and Nerf in synergy. SNN is used as an optimization constraint for Nerf. First, SNN is used to optimize the input data of Nerf. By training SNN, optimized feature map data is provided to Nerf, thereby simplifying the training time and computational resources required for Nerf. This allows for the reconstruction of 3D models from 2D images to 3D models with lower energy consumption, providing a more economical and practical 3D model reconstruction method that can complete 3D model reconstruction quickly and accurately with low energy consumption.

[0059] Specifically, such as Figure 1 As shown, the 3D model reconstruction method based on a hybrid spiking neural network according to the present invention includes the following steps:

[0060] S100, Acquire the target image.

[0061] Specifically, the target image is an original two-dimensional image, and image features are extracted from the target image for subsequent 3D model reconstruction. While the boundary data between the model and the background is easily extracted, neural network models often struggle to extract pixel-level boundary information within the image during training and use. In particular, existing technologies often use RGB images with limited boundary information, undoubtedly posing a significant challenge to extracting corresponding image features. Therefore, to address the problem of insufficient boundary information in the target image, this invention first generates a boundary image corresponding to the object in the target image. Using this boundary image as auxiliary information, both the target image and the corresponding boundary image are used as data sources for subsequent model training and use to extract and generate corresponding feature maps, thereby making the target object clearer and its boundaries more defined.

[0062] Therefore, after obtaining the target image in step S100, obtaining the boundary image of the object corresponding to the target image includes the following steps:

[0063] S110. Extract the pixel values ​​of each pixel in the target image.

[0064] S120. Compare the pixel value of each pixel in the target image with the pixel value of its neighboring pixels. Assign a value of 1 to pixels that are different from those of their neighboring pixels and a value of 0 to pixels that are the same as those of their neighboring pixels to obtain the boundary image.

[0065] Specifically, to extract the boundary image as auxiliary information, an instance boundary map is first calculated and each boundary is analyzed. If a pixel in the boundary image is different from its four neighboring nodes, it is set to 1; otherwise, it is set to 0 within a certain range. Finally, the boundary image is merged with the original target image and fed back into the neural network to complete the training and use of the neural network. Further, for the target image I, the scheme for generating the corresponding boundary image B can be expressed by the equation:

[0066]

[0067] Among them, B i,j Let I be the pixel value of each pixel in the boundary image B. i,j Let i and j represent the pixel values ​​of each pixel in the target image I, where i and j correspond to the row and column numbers of the specific pixel, respectively. Therefore, by using this equation, pixels in the target image with pixel values ​​equal to those of its four adjacent pixels in each of the four directions are changed to pixels with a pixel value of 0, and pixels in the target image with pixel values ​​different from those of at least one of its four adjacent pixels in each of the four directions are changed to pixels with a pixel value of 1, thus obtaining the boundary image corresponding to the target image. For example... Figure 2 As shown, Figure 2 (a) in the image is the target image. Figure 2 (b) in the figure is the boundary image of the object in the corresponding target image obtained by the scheme of the present invention. It can be seen that the boundary image clearly shows the boundary information of the target image. Using the boundary image as auxiliary information helps to quickly separate and extract the boundary information in the subsequent training and use of the neural network model.

[0068] Furthermore, such as Figure 1 As shown, in the 3D model reconstruction method based on a hybrid spiking neural network described in this invention, after step S100, the method further includes the following step:

[0069] S200. The target image is preprocessed to obtain an initial feature map.

[0070] For a target image with complete boundary information, preprocessing can determine the boundary information of objects in the target image, thereby generating an initial feature map that can be directly used for subsequent neural network training. Optionally, for a target image lacking complete boundary information, the boundary image corresponding to the object in the target image is used as auxiliary information. After integration, a significant improvement can be observed, making the target object clearer and its boundaries more defined. Specifically, after acquiring the target image and the boundary image, a fusion processing is performed on the target image and the boundary image to generate a feature map that can be used for subsequent neural network model training and testing. Figure 5 As shown, the 3D model reconstruction method based on a hybrid spiking neural network of the present invention uses a parameterless boundary encoder to preprocess the target image and the boundary image, thereby enhancing the boundary integrity in the data. Specifically, step S200 includes:

[0071] S210. Divide the target image and the boundary image into images of the same size;

[0072] S220. Merge the corresponding images in the target image and the boundary image and embed them into an n-dimensional vector;

[0073] S230. Combine the n-dimensional vectors to obtain the initial feature map.

[0074] The target image and the boundary image are segmented into several images of size 3×w×h, where w is the width and h is the height of the image, and there is a one-to-one correspondence between the target image and the boundary image. After the segmentation operation, all the corresponding images of the target image and the boundary image are fused together. An embedding layer is used to embed each fused image into an n-dimensional vector. All images with embedded n-dimensional vectors are combined to generate an initial feature map of size n×Hp×Wp, which is then used as input data for a spiking neural network model. Figure 3 As shown, Figure 3 In the image, (a) is the image that retains only the target image. Figure 3 Image (b) is an image that combines the target image and the boundary image. It can be seen that... Figure 3 In (b), the image that combines the target image and the boundary image shows better line continuity and clearer boundary texture between the model and the background.

[0075] Furthermore, such as Figure 1 As shown, in the 3D model reconstruction method based on a hybrid spiking neural network described in this invention, after step S200, the method further includes the following step:

[0076] S300. Construct a hybrid spiking neural network model, train the hybrid spiking neural network model to obtain a target model, wherein the target model includes several hybrid spiking layers, and the hybrid spiking layers include a hybrid module, an MLP module and a merging module.

[0077] Commonly used neurons in hybrid spiking neural network models include LIF neurons, HH (Hodgkin-Huxley) neurons, and Izhikevich neurons. The hybrid spiking neural network model described in this invention uses LIF neurons to form the overall framework, so as to take advantage of the simplicity and ease of application of LIF neurons.

[0078] The mathematical form of a LIF neuron is:

[0079]

[0080] Where V m (t) represents the membrane potential of the neuron at time t, X t For presynaptic input, τ m V is the membrane time constant. rest This is the reset potential after the emission pulse. When the membrane potential V t Exceeding the pulse threshold V th At that time, it emits a pulse. This continuously differential LIF neuron can be rewritten in discrete form:

[0081]

[0082] S t+1 =Θ(m t+1 -V th );

[0083] V t+1 =S t+1 V rest +(1-S t+1 )H t+1 ;

[0084] Among them, H t V is the membrane potential after the neurodynamic process at time t. t S is the membrane potential after the pulse is generated at time t. t Let θ be the pulse output at time t, and let Θ() be the Heaviside step function, defined as Θ(x) = 1 when x ≥ 0 and Θ(x) = 0 when x < 0.

[0085] Specifically, the steps in step S300 for training the target model can be summarized as follows:

[0086] S310. Construct a hybrid spiking neural network model with several initial LIF neurons, and determine the initial spiking threshold of the initial LIF neurons;

[0087] S320. When the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, resets the initial LIF neuron, and determines the reset membrane potential and pulse threshold, and replaces the initial LIF neuron with the reset LIF neuron.

[0088] S330. Apply gradient descent and repeat the step of replacing LIF neurons until the full-precision information meets the requirements to obtain the target model.

[0089] In this invention, LIF neurons have been improved to better adapt to real-world environments:

[0090]

[0091] o = 1, u = u reset ,u≥V th ;

[0092] Where u is the membrane potential, I is the input from the upper layer neurons, τ is the time coefficient, o is the output, and V th This is the pulse threshold of the current neuron. When a pulse is generated, the membrane potential is reset to the reset potential u. reset Training is completed iteratively by replacing the original LIF neurons with updated ones.

[0093] y n,t+1 =XW T x;

[0094] u n,t+1 =τu n,t (1-o n,t )+y n,t+1 ;

[0095] o n,t+1 =u n,t+1 >V th ;

[0096] Here, time t and time t+1 are two consecutive time steps, n is the layer index, and W, x, and y are the weights, input, and output, respectively. To avoid precision loss, a full-precision formula for the output is introduced:

[0097] r n,t+1 =max(u n,t+1 V th );

[0098] It represents a fully accurate output at time t+1, at which point o n,t+1The time coefficient τ and the pulse threshold V are temporary variables used to record the output state at time t+1. th These are the coefficients learned during the training process. Thus, as training progresses, the final target model possesses fully accurate iterative LIF neurons and can communicate across different images. Specifically, the average gradient of the IF neurons is used to normalize information across multiple neurons, reducing problems during training.

[0099] Furthermore, after constructing a hybrid spiking neural network model with several initial LIF neurons, the model further includes connecting all the initial LIF neurons to form a group of neurons, and the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, all the initial LIF neurons in the group of neurons are reset, and the reset LIF neurons in the group of neurons have the same membrane potential and pulse threshold.

[0100] Optionally, such as Figure 4 As shown, the group of neurons includes neurons connected in the vertical direction and neurons connected in the horizontal direction, thereby enabling more effective communication between images.

[0101] Among these, group neurons (GNs) provide stronger expressive power, thereby achieving smaller mapping errors. For example... Figure 7 The diagram shows a comparison of the transmission rates using GN and LIF alone, where the horizontal axis represents the time step and the vertical axis represents the transmission rate. Figure 7 It can be seen that when the time step is set to 4, the curves of both GN and LF neurons are stepped, but GN shows more details and is closer to the ReLU curve of Artificial Neural Network (ANN).

[0102] Specifically, the threshold of the i-th neuron in GN is expressed as:

[0103]

[0104] The threshold of GN itself is:

[0105]

[0106] Where I is the layer number, and when used as a superscript, it represents the layer index, and τ is the time coefficient.

[0107] If all neurons in a GN have the same membrane potential, then the neural dynamics equation in a GN is:

[0108] p l (t)=v l (t-1)+W l xl (t);

[0109]

[0110] In layer I, the membrane potential of all neurons before firing a pulse is p. l (t), and use v l (t) represents the membrane potential of all neurons after the firing pulse, and the charging current of all neurons in GN is the same. The process of the i-th neuron firing a pulse is represented by v. l (t) represents the reset process, using This represents the pulse aggregation process, which involves pooling the pulses emitted by all neurons in GN at the current time step.

[0111] Furthermore, the hybrid spiking neural network model incorporates a loss function through regularization optimization to ensure that the difference between the expected output and the actual output remains within an ideal range. Specifically, the final feature map is compared with the actual feature map to obtain the loss function:

[0112] l = l rgb +λ1L v +λ2L GN +λ3L 边界 ;

[0113] Where λ1, λ2, and λ3 are weights, and λ2 and λ3 are the weights of the regularization term; L rgb Let L be the color loss function. v Let L be the membrane potential loss function. 边界 Let L be the boundary loss function. GN The loss function for group neurons can be expressed as:

[0114]

[0115] By optimizing the loss function, multiple dimensions are optimized simultaneously, thereby effectively training the hybrid spiking neural network model to obtain the target model.

[0116] Furthermore, such as Figure 1 As shown, in the 3D model reconstruction method based on a hybrid spiking neural network described in this invention, after step S300, the method further includes the following step:

[0117] S400. The initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain the final feature map.

[0118] Specifically, step S400 includes:

[0119] S410. Input the initial feature map into the mixing module, run it independently on each channel, and communicate between different spatial locations to complete spatial location mixing;

[0120] S420. Input the initial feature map after spatial location mixing into the MLP module, operate on the spatial location independently, and communicate between different channels to complete channel mixing;

[0121] S430. Input the initial feature map after channel mixing into the merging module, separate the features of adjacent images in the initial feature map after channel mixing into four channels, and then halve the number of channels through a linear layer after merging to complete the image stitching and obtain the final feature map.

[0122] like Figure 6 As shown, after fusing and embedding the target image and boundary image to form an initial feature map, the initial feature map is input into a hybrid spiking neural network model. It undergoes processing by a mixing module, an MLP module, and a merging module to obtain the final feature map. Optionally, the final feature map is input into a feedforward neural network (FNN), where the mixing module includes vertical LIF (VLIF) and horizontal LIF (HLIF). Optionally, the hybrid spiking neural network model includes several sets of mixing modules—MLP modules—and the merging module is set at the final output of the final feature map.

[0123] Furthermore, such as Figure 1 As shown, in the 3D model reconstruction method based on a hybrid spiking neural network described in this invention, after step S500, the method further includes the following step:

[0124] S500. Input the final feature map into the Nerf renderer to reconstruct the 3D model corresponding to the target image.

[0125] Specifically, Nerf (Neural radiance field) uses MLP (Multi-layer perceptron) to describe the color and density of each point in a 2D image at different viewpoints. For each pixel, a camera ray is used to pass through the scene and generate sampling points. The ray direction, sampling position, and color and density calculated by MLP are combined, and the corresponding 3D scene pixels of each pixel in the original 2D image are obtained through volume rendering. By combining all the 3D scene pixels, the 3D scene can be restored and the corresponding 3D model can be reconstructed.

[0126] Specifically, step S500 includes:

[0127] S510. Extract the position coordinate information and viewing direction information of each pixel in the final feature map;

[0128] S520. Based on the position coordinate information and the viewing direction information, the density and color of the corresponding pixels are generated through a multilayer perceptron.

[0129] S530. Based on the density and the color, and combined with the volume rendering equation, the cumulative light amount, voxel density and expected color of the corresponding pixel are rendered.

[0130] S540. Combine the expected colors of all pixels to reconstruct a 3D model corresponding to the target image.

[0131] Among them, for the obtained pixel coordinates and perspective direction The generated output is density and color

[0132] σ=MLP θ (p);

[0133] c = MLP γ (e,v);

[0134] Here, θ and γ represent the parameters of two independent parts in the MLP, and e represents the embedded feature. Then, light ray r is projected from the camera towards the pixel, and K sampling points are set along the ray direction. That is, the equation is repeated K times to obtain the color and density of each sampling point, resulting in a total of K colors and K densities. The pixel can then be rendered.

[0135] a i =1-exp(-σ i δ i );

[0136]

[0137] Among them, a i Let T be the voxel density at point i. i Let i be the cumulative amount of light at point i. For the desired color, c i Let σ be the color of point i. i Let δ be the density at point i. i Let i be the distance between point i and its adjacent point i+1.

[0138] Once all pixels have been rendered, the corresponding 3D scene of the target image can be obtained, allowing for rapid reconstruction of the 3D model. Furthermore, the Nerf renderer is trained by minimizing the mean squared error loss between the rendered pixel color and the real pixel color. The loss function is:

[0139]

[0140] Where |R| represents the total number of pixels in the image, and C(r) is the color of the actual pixel.

[0141] By combining SNN and Nerf, the trained SNN optimizes the input data of Nerf, thereby assisting Nerf in completing 3D model reconstruction, reducing the energy consumption of the entire reconstruction process, and achieving fast, economical and accurate 3D model reconstruction.

[0142] Based on the above embodiments, the present invention also provides a 3D model reconstruction system based on a hybrid spiking neural network, such as... Figure 8 As shown, the system includes:

[0143] Image acquisition module 1000 is used to acquire target images;

[0144] The preprocessing module 2000 is used to preprocess the target image to obtain an initial feature map;

[0145] The model training module 3000 is used to construct a hybrid spiking neural network model, train the hybrid spiking neural network model to obtain a target model, wherein the target model includes several hybrid spiking layers, and the hybrid spiking layers include a hybrid module, an MLP module and a merging module;

[0146] Image processing module 4000 is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain the final feature map;

[0147] The model reconstruction module 5000 is used to input the final feature map into the Nerf renderer to reconstruct a 3D model corresponding to the target image.

[0148] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 9As shown, the smart terminal includes a processor, a computer-readable storage medium, a network interface, and a display screen connected via a system bus. The processor provides computing and control capabilities. The computer-readable storage medium includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D model reconstruction method based on a hybrid spiking neural network. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0149] Those skilled in the art will understand that the principle block diagram shown in the figure is only a block diagram of a part of the structure related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0150] The present invention also provides a non-volatile computer-readable storage medium storing a 3D model reconstruction program based on a hybrid spiking neural network. When the 3D model reconstruction program based on a hybrid spiking neural network is executed by a processor, it implements the steps of the 3D model reconstruction method based on a hybrid spiking neural network described in the present invention.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] In summary, this invention discloses a 3D model reconstruction method based on a hybrid spiking neural network, comprising the following steps: acquiring a target image; preprocessing the target image to obtain an initial feature map; constructing a hybrid spiking neural network model and training the model to obtain a target model, wherein the target model includes several hybrid spiking layers, each including a mixing module, an MLP module, and a merging module; inputting the initial feature map into the mixing module for spatial position mixing, inputting the spatially mixed initial feature map into the MLP module for channel mixing, inputting the channel-mixed initial feature map into the merging module for image stitching to obtain a final feature map; and inputting the final feature map into a Nerf renderer to reconstruct a 3D model corresponding to the target image. This invention combines a hybrid spiking neural network with neural radiation fields, enabling accurate 3D model reconstruction with low resource consumption, demonstrating high practicality and facilitating further application.

[0153] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A 3D model reconstruction method based on a hybrid spiking neural network, characterized in that, The method includes: Acquire the target image; The target image is preprocessed to obtain an initial feature map; A hybrid spiking neural network model is constructed and trained to obtain a target model. The target model includes several hybrid spiking layers, and each hybrid spiking layer includes a hybrid module, an MLP module, and a merging module. The initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain the final feature map; The final feature map is input into the Nerf renderer to reconstruct a 3D model corresponding to the target image; Constructing a hybrid spiking neural network model, training the hybrid spiking neural network model to obtain the target model, specifically including: Construct a hybrid spiking neural network model with several initial LIF neurons, and determine the initial spiking threshold of the initial LIF neurons; When the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, resets the initial LIF neuron, and determines the reset membrane potential and pulse threshold, and replaces the initial LIF neuron with the reset LIF neuron; By applying gradient descent and repeating the step of replacing LIF neurons until the full-precision information meets the requirements, the target model is obtained. After constructing a hybrid spiking neural network model with several initial LIF neurons, the method includes: connecting all the initial LIF neurons to form a group of neurons, and the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, resetting all the initial LIF neurons in the group of neurons, and the reset LIF neurons in the group of neurons have the same membrane potential and pulse threshold. The group of neurons includes LIF neurons that connect and run in a vertical direction and LIF neurons that connect and run in a horizontal direction.

2. The 3D model reconstruction method based on a hybrid spiking neural network according to claim 1, characterized in that, After acquiring the target image, the following is also included: Extract the pixel values ​​of each pixel in the target image; The pixel value of each pixel in the target image is compared with the pixel value of its neighboring pixels. Pixels with different pixel values ​​from their neighboring pixels are assigned a value of 1, and pixels with the same pixel value as their neighboring pixels are assigned a value of 0, thus obtaining the boundary image of the object in the target image.

3. The 3D model reconstruction method based on a hybrid spiking neural network according to claim 2, characterized in that, The target image is preprocessed to obtain an initial feature map, specifically including: The target image and the boundary image are each divided into images of the same size; The corresponding images in the target image and the boundary image are fused and embedded into an n-dimensional vector; The initial feature map is obtained by combining the n-dimensional vectors.

4. The 3D model reconstruction method based on a hybrid spiking neural network according to claim 1, characterized in that, The initial feature map is input into the mixing module for spatial mixing, the spatially mixed initial feature map is input into the MLP module for channel mixing, and the channel-mixed initial feature map is input into the merging module for image stitching to obtain the final feature map, specifically including: The initial feature map is input into the mixing module, which runs independently on each channel and communicates between different spatial locations to complete spatial location mixing. The initial feature map after spatial location mixing is input into the MLP module, which independently operates on the spatial location and communicates between different channels to complete channel mixing; The initial feature map after channel mixing is input into the merging module. The features of adjacent images in the initial feature map after channel mixing are separated into four channels. After merging, the number of channels is halved through a linear layer to complete the image stitching and obtain the final feature map.

5. The 3D model reconstruction method based on a hybrid spiking neural network according to claim 1, characterized in that, The final feature map is input into the Nerf renderer to reconstruct a 3D model corresponding to the target image, specifically including: Extract the position coordinates and viewing direction information of each pixel in the final feature map; Based on the position coordinate information and the gaze direction information, the density and color of the corresponding pixels are generated through a multilayer perceptron; Based on the density and color, and combined with the volume rendering equation, the cumulative light amount, voxel density and expected color of the corresponding pixel are rendered. By combining the expected colors of all pixels, a 3D model corresponding to the target image is reconstructed.

6. A 3D model reconstruction system based on a hybrid spiking neural network, to implement the 3D model reconstruction method based on a hybrid spiking neural network as described in any one of claims 1-5, characterized in that, The system includes: The image acquisition module is used to acquire the target image; The preprocessing module is used to preprocess the target image to obtain an initial feature map; The model training module is used to construct a hybrid spiking neural network model, train the hybrid spiking neural network model to obtain a target model, wherein the target model includes several hybrid spiking layers, and the hybrid spiking layers include a hybrid module, an MLP module and a merging module; The image processing module is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain the final feature map; The model reconstruction module is used to input the final feature map into the Nerf renderer to reconstruct a 3D model corresponding to the target image.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a 3D model reconstruction program based on a hybrid spiking neural network. When the 3D model reconstruction program based on a hybrid spiking neural network is executed by a processor, it implements the steps of the 3D model reconstruction method based on a hybrid spiking neural network as described in any one of claims 1-5.

Images