3D model reconstruction method based on hybrid pulse neural network

Abstract

The invention discloses a 3D model reconstruction method based on a hybrid pulse neural network. The method comprises the following steps: acquiring a target image; preprocessing the target image to obtain an initial feature map; a mixed pulse neural network model is constructed and trained, a target model is obtained, the target model comprises a plurality of mixed pulse layers, and each mixed pulse layer comprises a mixing module, an MLP module and a merging module; inputting the initial feature map into a mixing module for spatial position mixing, inputting the initial feature map after spatial position mixing into an MLP module for channel mixing, and inputting the initial feature map after channel mixing into a merging module for picture splicing to obtain a final feature map; and inputting the final feature map into a Nerf renderer, and reconstructing a 3D model corresponding to the target image. According to the method, the hybrid pulse neural network and the neural radiation field are combined, accurate 3D model reconstruction can be realized under the condition of low consumption, the practicability is high, and further popularization and application are facilitated.

Description

Technical Field

[0001] The present invention relates to the field of 3D model reconstruction, and in particular to a 3D model reconstruction method based on a hybrid pulse neural network. Background Art

[0002] 3D model reconstruction, that is, restoring the 3D scene based on a 2D image to generate a corresponding 3D model, is the core task of computer vision tasks. In the prior art, neural radiance field (Nerf) is generally used to describe the color and density of each point in a 2D image at each viewing angle through a multi-layer perceptron (MLP). For each pixel, a camera ray is used to pass through the scene and generate a sampling point. The color and density calculated by combining the ray direction, sampling position and MLP are used to obtain the 3D scene pixels corresponding to each pixel in the original 2D image through volume rendering. Combining all 3D scene pixels can restore the 3D scene and reconstruct the corresponding 3D model.

[0003] However, although high-quality and realistic 3D models can be obtained through Nerf, due to the simple structural design of Nerf itself, it has extremely high requirements for input data. Otherwise, it will take a lot of computing resources and time to train. As a result, the 3D model reconstruction technology in the existing technology has poor overall practicality, high energy consumption, and is uneconomical, and cannot be promoted and applied.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a 3D model reconstruction method based on a mixed spiking neural network. By combining the mixed spiking neural network (Mixed Spiking Neural Network) with Nerf, the problem of poor overall practicality, high energy consumption and uneconomicality of the 3D model reconstruction technology in the prior art, which leads to the inability to further promote and apply, is solved.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a 3D model reconstruction method based on a hybrid spiking neural network, wherein the method comprises:

[0008] Get the target image;

[0009] Preprocessing the target image to obtain an initial feature map;

[0010] Constructing a hybrid pulse neural network model, training the hybrid pulse neural network model to obtain a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layer includes a hybrid module, an MLP module and a merging module;

[0011] The initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain a final feature map;

[0012] The final feature map is input into a Nerf renderer to reconstruct a 3D model corresponding to the target image.

[0013] In one embodiment, after acquiring the target image, the method further includes:

[0014] Extracting the pixel value of each pixel in the target image;

[0015] The pixel value of each pixel in the target image is compared with the pixel values ​​of adjacent pixels, pixels with different pixel values ​​from adjacent pixels are assigned a value of 1, and pixels with the same pixel values ​​as adjacent pixels are assigned a value of 0, to obtain a boundary image corresponding to the object in the target image.

[0016] In one implementation, preprocessing the target image to obtain an initial feature map specifically includes:

[0017] Dividing the target image and the boundary image into pictures of the same size respectively;

[0018] Merge the target image and the corresponding pictures in the boundary image and embed them into an n-dimensional vector;

[0019] The n-dimensional vectors are combined to obtain the initial feature map.

[0020] In one embodiment, constructing a hybrid spiking neural network model and training the hybrid spiking neural network model to obtain a target model specifically includes:

[0021] Constructing a hybrid spiking neural network model with a plurality of initial LIF neurons, and determining an initial spiking threshold of the initial LIF neurons;

[0022] When the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, the initial LIF neuron is reset and the membrane potential and pulse threshold after the reset are determined, and the initial LIF neuron is replaced with the reset LIF neuron;

[0023] Applying the gradient descent method, the step of replacing the LIF neurons is repeated until the full-precision information meets the requirements, thereby obtaining the target model.

[0024] In one embodiment, after constructing a hybrid pulse neural network model with several initial LIF neurons, it includes: connecting all the initial LIF neurons to form a group of neurons, and the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, resetting all the initial LIF neurons in the group of neurons, and the reset LIF neurons in the group of neurons have the same membrane potential and pulse threshold.

[0025] In one embodiment, the group of neurons includes LIF neurons connected and running along a vertical direction and LIF neurons connected and running along a horizontal direction.

[0026] In one implementation, the initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain the final feature map, specifically including:

[0027] Input the initial feature map into the mixing module, operate independently on each channel, and communicate between different spatial positions to complete spatial position mixing;

[0028] The initial feature map after spatial position mixing is input into the MLP module, the spatial position is operated independently, and communication is performed between different channels to complete channel mixing;

[0029] The initial feature map after channel mixing is input into the merging module, the features of adjacent pictures in the initial feature map after channel mixing are separated into four channels, and after merging, the number of channels is halved through a linear layer to complete the picture stitching and obtain the final feature map.

[0030] In one implementation, the final feature map is input into a Nerf renderer to reconstruct a 3D model corresponding to the target image, specifically comprising:

[0031] Extracting position coordinate information and sight direction information of each pixel in the final feature map;

[0032] Based on the position coordinate information and the sight direction information, generating density and color of corresponding pixels through a multi-layer perceptron;

[0033] According to the density and the color, combined with a volume rendering equation, rendering obtains the light accumulation amount, voxel density and expected color of the corresponding pixel;

[0034] The expected colors of all pixels are combined to reconstruct a 3D model corresponding to the target image.

[0035] In a second aspect, an embodiment of the present invention further provides a 3D model reconstruction system based on a hybrid spiking neural network, the system comprising:

[0036] An image acquisition module, used for acquiring a target image;

[0037] A preprocessing module, used for preprocessing the target image to obtain an initial feature map;

[0038] A model training module, used for constructing a hybrid pulse neural network model, training the hybrid pulse neural network model, and obtaining a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layers include a hybrid module, an MLP module, and a merging module;

[0039] An image processing module is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map;

[0040] The model reconstruction module is used to input the final feature map into the Nerf renderer to reconstruct a 3D model corresponding to the target image.

[0041] In the third aspect, an embodiment of the present invention also provides a non-temporary computer-readable storage medium, which stores a 3D model reconstruction program based on a hybrid pulse neural network. When the 3D model reconstruction program based on a hybrid pulse neural network is executed by a processor, the steps of the 3D model reconstruction method based on a hybrid pulse neural network as described above are implemented.

[0042] The present invention discloses a 3D model reconstruction method based on a hybrid pulse neural network, comprising the steps of: acquiring a target image; preprocessing the target image to obtain an initial feature map; constructing a hybrid pulse neural network model, training the hybrid pulse neural network model to obtain a target model, wherein the target model comprises a plurality of hybrid pulse layers, the hybrid pulse layer comprises a hybrid module, an MLP module and a merging module; inputting the initial feature map into the hybrid module for spatial position mixing, inputting the initial feature map after spatial position mixing into the MLP module for channel mixing, inputting the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map; inputting the final feature map into a Nerf renderer to reconstruct a 3D model corresponding to the target image. The present invention combines a hybrid pulse neural network with a neural radiation field, can realize accurate 3D model reconstruction under low consumption, has high practicality, and is conducive to further promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a flowchart of the steps of a 3D model reconstruction method based on a hybrid pulse neural network in an embodiment of the present invention.

[0045] Figure 2 Schematic diagram of comparison between a target image and a boundary image in an embodiment of the present invention.

[0046] Figure 3 3 is a schematic diagram comparing an unfused boundary image and a fused boundary image in an embodiment of the present invention.

[0047] Figure 4 Schematic diagram of LIF neurons in different directions in an embodiment of the present invention.

[0048] Figure 5 It is a flow chart of a 3D model reconstruction method based on a hybrid pulse neural network in an embodiment of the present invention.

[0049] Figure 6 It is a flow chart of another form of a 3D model reconstruction method based on a hybrid pulse neural network in an embodiment of the present invention.

[0050] Figure 7 Schematic diagram of comparison of emission pulse rates of GN and LIF in an embodiment of the present invention.

[0051] Figure 8 It is a block diagram of a 3D model reconstruction system based on a hybrid pulse neural network in an embodiment of the present invention.

[0052] Fig. 9 It is a block diagram of an intelligent terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention discloses a 3D model reconstruction method based on a hybrid spiking neural network. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0056] Neural radiance field (Nerf) is a deep learning model for three-dimensional implicit space modeling. It can capture scenes from a known perspective without the need for an intermediate three-dimensional reconstruction process. It can synthesize images from a new perspective based on pose internal parameters and images, thereby completing the task of 3D model reconstruction. However, Nerf itself has a simple overall structure. When it comes to specific 3D model reconstruction tasks, it requires a lot of computing resources and training time to obtain the required model, resulting in poor practicality and not conducive to further expansion of applications.

[0057] Spiking Neural Network (SNN) is a third-generation neural network model that is good at processing sparse data and can communicate between data at different time steps through LIF (Leaky Integrate and Fire) neurons. Therefore, it is often used in fields such as neural computing, perception processing, and event-driven systems. However, the training and inference algorithms of SNN are very complex, and its actual application scenarios are still limited.

[0058] The present invention combines the effect of SNN on simulating real biological neurons and uses the effect of LIF neurons in simplifying the amount of calculation, uses SNN in collaboration with Nerf, and uses SNN as an optimization restriction of Nerf. First, SNN is used to complete the optimization task of Nerf input data, and SNN is trained to provide Nerf with optimized feature map data, thereby simplifying Nerf training time and the required computing resources, thereby completing the 3D model reconstruction task from a two-dimensional image to a three-dimensional model with lower energy consumption, providing a more economical and more practical 3D model reconstruction method, which can complete 3D model reconstruction quickly and accurately with low energy consumption.

[0059] Specifically, Figure 1 As shown, the 3D model reconstruction method based on a hybrid pulse neural network described in the present invention includes the following steps:

[0060] S100: Acquire a target image.

[0061] Specifically, the target image is an original two-dimensional image, and the image features in the target image are extracted for subsequent 3D model reconstruction. Among the image features, although the boundary data between the model and the background is easy to extract, during training and use, the neural network model often finds it difficult to extract pixel-level boundary information in the image. In particular, the prior art often uses RGB images with less boundary information, which undoubtedly causes great difficulties in extracting corresponding image features. Therefore, in order to solve the problem that the target image used may have too little boundary information and cannot extract enough feature data, the present invention first generates a boundary image corresponding to the object in the target image based on the target image, uses the boundary image as auxiliary information, and uses the target image and the corresponding boundary image together as the data source for subsequent model training and use, extracts and generates the corresponding feature map, so that the target object is clearer and the boundary is clearer.

[0062] Therefore, after acquiring the target image in step S100, acquiring the boundary image corresponding to the object in the target image includes the following steps:

[0063] S110: Extracting the pixel value of each pixel in the target image.

[0064] S120, comparing the pixel value of each pixel in the target image with the pixel values ​​of adjacent pixels, assigning a pixel having a different pixel value from the adjacent pixels to 1, and assigning a pixel having the same pixel value as the adjacent pixels to 0, to obtain the boundary image.

[0065] Specifically, in order to extract the boundary image as auxiliary information, the instance boundary map is first calculated and each boundary is analyzed. If a pixel in the boundary image is different from its four adjacent nodes, it is set to 1; otherwise, it is set to 0 within the range. In this way, the boundary image is finally merged with the original target image and fed back to the neural network to complete the training and use of the neural network. Further, for the target image I, the scheme for generating the corresponding boundary image B can be expressed by the equation:

[0066]

[0067] Among them, B i,j is the pixel value of each pixel in the boundary image B, I i,j is the pixel value of each pixel in the target image I, and i and j correspond to the row and column number of the specific pixel in the image. Therefore, by using this equation, the pixel values ​​of the target image that are equal to the adjacent pixel values ​​in the four directions are changed to the pixel value of 0, and the pixel values ​​of the target image that are different from at least one adjacent pixel in the four directions are changed to the pixel value of 1, and the boundary image corresponding to the target image can be obtained. Figure 2 As shown, Figure 2 (a) is the target image. Figure 2 (b) is the boundary image of the object in the corresponding target image extracted by the solution of the present invention. It can be seen that the boundary image clearly shows the boundary information of the target image. Using the boundary image as auxiliary information is helpful to quickly separate and extract boundary information during the subsequent training and use of the neural network model.

[0068] Furthermore, if Figure 1 As shown, in the 3D model reconstruction method based on hybrid pulse neural network described in the present invention, after step S100, it also includes the following steps:

[0069] S200: Preprocess the target image to obtain an initial feature map.

[0070] For a target image with complete boundary information, the boundary information of the object in the target image can be determined through preprocessing, thereby generating an initial feature map that can be directly used for subsequent neural network training. Optionally, for a target image that lacks complete boundary information, the boundary image corresponding to the object in the target image is used as auxiliary information. After integration, significant improvements can be observed, making the target object clearer and the boundary clearer. Specifically, after acquiring the target image and the boundary image, the target image and the boundary image are fused to generate a feature map that can be used for subsequent neural network model training and testing. Figure 5 As shown, in the 3D model reconstruction method based on hybrid pulse neural network of the present invention, a parameter-free boundary encoder is set to pre-process the target image and the boundary image, so as to enhance the boundary integrity in the data. Specifically, step S200 includes:

[0071] S210, dividing the target image and the boundary image into pictures of the same size respectively;

[0072] S220, fusing the target image and the corresponding pictures in the boundary image and embedding them into an n-dimensional vector;

[0073] S230, combining the n-dimensional vectors to obtain the initial feature map.

[0074] The target image and the boundary image are segmented into several images of 3×w×h size, where w is the width of the image and h is the height of the image, and there is a one-to-one correspondence between the target image and the boundary image. After the segmentation operation, all the images of the target image and the boundary image are fused, and the fused images are respectively embedded into n-dimensional vectors through the embedding layer. All the images embedded into n-dimensional vectors are combined to generate an initial feature map of n×Hp×Wp, which can be used as input data for the pulse neural network model later. Figure 3 As shown, Figure 3 (a) in the figure is the image that only retains the target image. Figure 3 (b) is an image that combines the target image and the boundary image. It can be seen that Figure 3 In (b), the image that combines the target image and the boundary image has better line continuity and the boundary texture between the model and the background is clearer.

[0075] Furthermore, if Figure 1 As shown, in the 3D model reconstruction method based on hybrid pulse neural network described in the present invention, after step S200, it also includes the following steps:

[0076] S300, constructing a hybrid pulse neural network model, training the hybrid pulse neural network model, and obtaining a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layers include a hybrid module, an MLP module, and a merging module.

[0077] Commonly used neurons in the hybrid pulse neural network model include LIF neurons, HH (Hodgkin-Huxley) neurons and Izhikevich neurons. The hybrid pulse neural network model described in the present invention forms an overall framework through LIF neurons to utilize the simple and easy-to-use characteristics of LIF neurons.

[0078] The mathematical form of a LIF neuron is:

[0079]

[0080] Where V m (t) is the membrane potential of the neuron at time t, X t is the presynaptic input, τ m is the membrane time constant, V rest is the reset potential after the emission pulse. t Exceeding the pulse threshold V th When , a pulse is fired. This LIF neuron in continuous differential form can be rewritten in discrete form:

[0081]

[0082] S t+1 =Θ(m t+1 -V th );

[0083] V t+1 =S t+1 V rest +(1-S t+1 )H t+1 ;

[0084] Among them, H t is the membrane potential after the neural dynamics process at time t, V t is the membrane potential after the pulse is generated at time t, S t is the pulse output at time t, Θ() is the Heaviside step function, which is defined as Θ(x) = 1 when x ≥ 0 and Θ(x) = 0 when x < 0.

[0085] Specifically, the steps of training to obtain the target model in step S300 can be summarized as follows:

[0086] S310, constructing a hybrid pulse neural network model with a plurality of initial LIF neurons, and determining an initial pulse threshold of the initial LIF neurons;

[0087] S320: when the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, the initial LIF neuron is reset and the membrane potential and pulse threshold after the reset are determined, and the initial LIF neuron is replaced with the reset LIF neuron;

[0088] S330, applying the gradient descent method, repeating the step of replacing the LIF neurons until the full-precision information meets the requirements, thereby obtaining the target model.

[0089] Among them, the present invention improves the LIF neuron to better adapt to the actual environment:

[0090]

[0091] o=1,u=u reset ,u≥V th ;

[0092] Where u is the membrane potential, I is the input from the upper layer neurons, τ is the time coefficient, o is the output, V th is the pulse threshold of the current neuron. When a pulse is generated, the membrane potential is reset to the reset potential u reset Through iteration, the original LIF neurons are replaced with updated LIF neurons to complete the training:

[0093] y n,t+1 =XW T x;

[0094] u n,t+1 =τu n,t (1-o n,t )+y n,t+1 ;

[0095] o n,t+1 =u n,t+1 >V th ;

[0096] Among them, time t and time t+1 are two consecutive time steps, n is the layer index, W, x and y are weight, input and output respectively. In order to avoid precision loss, the full precision formula of output is introduced:

[0097] r n,t+1 =max(u n,t+1 ,V th );

[0098] It represents the full-precision output at time t+1, when o n,t+1As temporary variables for recording the output state at time t+1, the time coefficient τ and the pulse threshold V th are coefficients learned during the training process. As the training process progresses, the final target model has iterative LIF neurons with full precision and can communicate between different images. Specifically, the average IF neuron gradient is used to achieve information normalization between multiple neurons to reduce problems during training.

[0099] Furthermore, after constructing a hybrid pulse neural network model with several initial LIF neurons, it also includes connecting all the initial LIF neurons to form a group neuron, and the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, all the initial LIF neurons in the group neuron are reset, and the reset LIF neurons in the group neuron have the same membrane potential and pulse threshold.

[0100] Alternatively, if Figure 4 As shown, the group of neurons includes vertically connected neurons and horizontally connected neurons, thereby achieving more effective communication between pictures.

[0101] Among them, the group neuron (GN) provides stronger expression ability, thereby achieving smaller mapping errors. Figure 7 The figure shows the comparison of the pulse emission rates using GN and LIF, respectively, where the horizontal axis is the time step and the vertical axis is the emission rate. Figure 7 It can be seen that when the time step is set to 4, the GN and LF neuron curves are both stepped, but GN reflects more details and is closer to the ReLU curve of the artificial neural network (ANN).

[0102] Specifically, the threshold of the i-th neuron in GN is expressed as:

[0103]

[0104] The threshold of GN itself is:

[0105]

[0106] Where I is the number of layers, t as a superscript represents the layer index, and τ is the time coefficient.

[0107] The membrane potential of all neurons in GN is the same, so the neural dynamics equation in GN is:

[0108] p l (t) = v l (t-1)+W l xl (t);

[0109]

[0110] Among them, in layer I, the membrane potential of all neurons before firing a pulse is p l (t), and use v l (t) represents the membrane potential of all neurons after firing a pulse, and the charging current of all neurons in GN is the same, Represents the process of the ith neuron firing a pulse, with v l (t) represents the reset process, represents the pulse aggregation process, which aggregates the pulses fired by all neurons in the GN at the current time step.

[0111] Furthermore, a loss function is introduced into the hybrid spiking neural network model through regularization optimization to ensure that the difference between the expected output and the actual output is within an ideal range. Specifically, the final feature map is compared with the actual feature map to obtain a loss function:

[0112] l=l rgb +λ1L v +λ2L GN +λ3L 边界 ;

[0113] Among them, λ1, λ2 and λ3 are weights, and λ2 and λ3 are the weights of the regularization term; L rgb is the color loss function, L v is the membrane potential loss function, L 边界 is the boundary loss function, L GN is the group neuron loss function, which can be expressed as:

[0114]

[0115] By optimizing the loss function, multiple dimensions are optimized simultaneously, thereby effectively training the hybrid pulse neural network model to obtain the target model.

[0116] Furthermore, if Figure 1 As shown, in the 3D model reconstruction method based on hybrid pulse neural network described in the present invention, after step S300, it also includes the following steps:

[0117] S400, inputting the initial feature map into the mixing module for spatial position mixing, inputting the initial feature map after spatial position mixing into the MLP module for channel mixing, and inputting the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map.

[0118] Specifically, step S400 includes:

[0119] S410, inputting the initial feature map into the mixing module, operating independently on each channel, and communicating between different spatial positions to complete spatial position mixing;

[0120] S420, inputting the initial feature map after spatial position mixing into the MLP module, independently operating the spatial position, and communicating between different channels to complete channel mixing;

[0121] S430, inputting the initial feature map after channel mixing into the merging module, separating the features of adjacent pictures in the initial feature map after channel mixing into four channels, and after merging, halving the number of channels through a linear layer to complete picture stitching and obtain the final feature map.

[0122] like Figure 6 As shown, after the target image and the boundary image are fused and embedded to form an initial feature map, the initial feature map is input into the hybrid pulse neural network model, and is processed by the hybrid module, the MLP module and the merging module respectively to obtain the final feature map. Optionally, the final feature map is input into a feedforward neural network (FNN), and the hybrid module includes a vertical LIF (VLIF) and a horizontal LIF (HLIF). Optionally, the hybrid pulse neural network model includes several groups of hybrid modules-MLP modules, and the merging module is set at the final output of the final feature map.

[0123] Furthermore, if Figure 1 As shown, in the 3D model reconstruction method based on hybrid pulse neural network described in the present invention, after step S500, it also includes the following steps:

[0124] S500: Input the final feature map into a Nerf renderer to reconstruct a 3D model corresponding to the target image.

[0125] Specifically, Nerf (Neural radiance field) describes the color and density of each point in a two-dimensional image at each viewing angle through MLP (Multi-layer perceptron). For each pixel, a camera ray is used to pass through the scene and generate a sampling point. The color and density calculated by the MLP are combined with the ray direction, sampling position, and volume rendering to obtain the 3D scene pixels corresponding to each pixel in the original two-dimensional image. By combining all the 3D scene pixels, the 3D scene can be restored and the corresponding 3D model can be reconstructed.

[0126] Specifically, step S500 includes:

[0127] S510, extracting position coordinate information and sight direction information of each pixel in the final feature map;

[0128] S520, generating density and color of corresponding pixels through a multi-layer perceptron based on the position coordinate information and the sight direction information;

[0129] S530, rendering to obtain the light accumulation amount, voxel density and expected color of the corresponding pixel in combination with the volume rendering equation according to the density and the color;

[0130] S540 , combining the expected colors of all pixels to reconstruct a 3D model corresponding to the target image.

[0131] Among them, for the pixel coordinates obtained and viewing direction The generated output is the density and color

[0132] σ=MLP θ (p);

[0133] c=MLP γ (e,v);

[0134] Among them, θ and γ represent the parameters of two independent parts in MLP, and e represents the embedded feature. Then project the light r from the camera toward the pixel point, and set K sampling points along the direction of the light, that is, repeat the equation K times, obtain the color and density of each sampling point respectively, and obtain K colors and K densities in total, then the pixel can be rendered:

[0135] a i =1-exp(-σ i δ i );

[0136]

[0137] Among them, a i is the voxel density at point i, T i is the accumulated amount of light at point i, is the expected color, c i is the color of point i, σ i is the density of point i, δ i is the distance between point i and the adjacent point i+1.

[0138] After all pixels are rendered, the 3D scene corresponding to the target image can be obtained, so that the 3D model can be quickly reconstructed. Furthermore, the Nerf renderer is trained by minimizing the mean square error loss between the color of the rendered pixel and the color of the real pixel. The loss function is:

[0139]

[0140] Among them, |R| represents the total number of all pixels in the image, and C(r) is the color of the actual pixel.

[0141] In this way, by combining SNN and Nerf, the trained SNN is used to optimize the input data of Nerf, thereby assisting Nerf to complete 3D model reconstruction, reducing the energy consumption of the entire reconstruction process, and achieving fast, economical and accurate completion of 3D model reconstruction.

[0142] Based on the above embodiments, the present invention also provides a 3D model reconstruction system based on a hybrid pulse neural network, such as Figure 8 As shown, the system comprises:

[0143] An image acquisition module 1000 is used to acquire a target image;

[0144] A preprocessing module 2000 is used to preprocess the target image to obtain an initial feature map;

[0145] A model training module 3000 is used to construct a hybrid pulse neural network model, train the hybrid pulse neural network model, and obtain a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layers include a hybrid module, an MLP module, and a merging module;

[0146] The image processing module 4000 is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map;

[0147] The model reconstruction module 5000 is used to input the final feature map into the Nerf renderer to reconstruct the 3D model corresponding to the target image.

[0148] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Fig. 9As shown. The intelligent terminal includes a processor, a computer-readable storage medium, a network interface and a display screen connected via a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The computer-readable storage medium of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a 3D model reconstruction method based on a hybrid pulse neural network is implemented. The display screen of the terminal can be a liquid crystal display or an electronic ink display.

[0149] Those skilled in the art will understand that the principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] The present invention also provides a non-volatile computer-readable storage medium, which stores a 3D model reconstruction program based on a hybrid pulse neural network. When the 3D model reconstruction program based on a hybrid pulse neural network is executed by a processor, the steps of the 3D model reconstruction method based on a hybrid pulse neural network described in the present invention are implemented.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0152] In summary, the present invention discloses a 3D model reconstruction method based on a hybrid pulse neural network, comprising the steps of: acquiring a target image; preprocessing the target image to obtain an initial feature map; constructing a hybrid pulse neural network model, training the hybrid pulse neural network model to obtain a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layer includes a hybrid module, an MLP module, and a merging module; inputting the initial feature map into the hybrid module for spatial position mixing, inputting the initial feature map after spatial position mixing into the MLP module for channel mixing, and inputting the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map; inputting the final feature map into a Nerf renderer to reconstruct a 3D model corresponding to the target image. The present invention combines a hybrid pulse neural network with a neural radiation field, can realize accurate 3D model reconstruction under low consumption, has high practicality, and is conducive to further promotion and application.

[0153] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A 3D model reconstruction method based on a hybrid spiking neural network, characterized in that: The method comprises: Get the target image; Preprocessing the target image to obtain an initial feature map; Constructing a hybrid pulse neural network model, training the hybrid pulse neural network model to obtain a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layer includes a hybrid module, an MLP module and a merging module; The initial feature map is input into the mixing module for spatial position mixing, the initial feature map after spatial position mixing is input into the MLP module for channel mixing, and the initial feature map after channel mixing is input into the merging module for image stitching to obtain a final feature map; The final feature map is input into a Nerf renderer to reconstruct a 3D model corresponding to the target image.

2. The 3D model reconstruction method based on hybrid pulse neural network according to claim 1, characterized in that: After acquiring the target image, it also includes: Extracting the pixel value of each pixel in the target image; The pixel value of each pixel in the target image is compared with the pixel values ​​of adjacent pixels, pixels with different pixel values ​​from adjacent pixels are assigned a value of 1, and pixels with the same pixel values ​​as adjacent pixels are assigned a value of 0, to obtain a boundary image corresponding to the object in the target image.

3. The 3D model reconstruction method based on hybrid pulse neural network according to claim 2 is characterized in that: Preprocessing the target image to obtain an initial feature map specifically includes: Dividing the target image and the boundary image into pictures of the same size respectively; Merge the target image and the corresponding pictures in the boundary image and embed them into an n-dimensional vector; The n-dimensional vectors are combined to obtain the initial feature map.

4. The 3D model reconstruction method based on hybrid pulse neural network according to claim 1, characterized in that: Constructing a hybrid pulse neural network model and training the hybrid pulse neural network model to obtain a target model, specifically including: Constructing a hybrid spiking neural network model with a plurality of initial LIF neurons, and determining an initial spiking threshold of the initial LIF neurons; When the membrane potential of the initial LIF neuron is greater than the initial pulse threshold, the initial LIF neuron generates a pulse, the initial LIF neuron is reset and the membrane potential and pulse threshold after the reset are determined, and the initial LIF neuron is replaced with the reset LIF neuron; Applying the gradient descent method, the step of replacing the LIF neurons is repeated until the full-precision information meets the requirements, thereby obtaining the target model.

5. The 3D model reconstruction method based on hybrid pulse neural network according to claim 4 is characterized in that: After constructing a hybrid pulse neural network model with several initial LIF neurons, it includes: connecting all the initial LIF neurons to form a group of neurons, and the membrane potential of each initial LIF neuron is equal; when one of the initial LIF neurons generates a pulse, resetting all the initial LIF neurons in the group of neurons, and the reset LIF neurons in the group of neurons have the same membrane potential and pulse threshold.

6. The 3D model reconstruction method based on hybrid pulse neural network according to claim 5, characterized in that: The group of neurons includes LIF neurons connected and running along a vertical direction and LIF neurons connected and running along a horizontal direction.

7. The 3D model reconstruction method based on hybrid pulse neural network according to claim 4, characterized in that: Inputting the initial feature map into the mixing module for spatial position mixing, inputting the initial feature map after spatial position mixing into the MLP module for channel mixing, and inputting the initial feature map after channel mixing into the merging module for image stitching to obtain the final feature map, specifically including: Input the initial feature map into the mixing module, operate independently on each channel, and communicate between different spatial positions to complete spatial position mixing; The initial feature map after spatial position mixing is input into the MLP module, the spatial position is operated independently, and communication is performed between different channels to complete channel mixing; The initial feature map after channel mixing is input into the merging module, the features of adjacent pictures in the initial feature map after channel mixing are separated into four channels, and after merging, the number of channels is halved through a linear layer to complete the picture stitching and obtain the final feature map.

8. The 3D model reconstruction method based on hybrid pulse neural network according to claim 1, characterized in that: The final feature map is input into the Nerf renderer to reconstruct a 3D model corresponding to the target image, specifically including: Extracting position coordinate information and sight direction information of each pixel in the final feature map; Based on the position coordinate information and the sight direction information, generating density and color of corresponding pixels through a multi-layer perceptron; According to the density and the color, combined with a volume rendering equation, rendering obtains the light accumulation amount, voxel density and expected color of the corresponding pixel; The expected colors of all pixels are combined to reconstruct a 3D model corresponding to the target image.

9. A 3D model reconstruction system based on a hybrid spiking neural network, characterized in that: The system comprises: An image acquisition module, used for acquiring a target image; A preprocessing module, used for preprocessing the target image to obtain an initial feature map; A model training module, used for constructing a hybrid pulse neural network model, training the hybrid pulse neural network model, and obtaining a target model, wherein the target model includes a plurality of hybrid pulse layers, and the hybrid pulse layers include a hybrid module, an MLP module, and a merging module; An image processing module is used to input the initial feature map into the mixing module for spatial position mixing, input the initial feature map after spatial position mixing into the MLP module for channel mixing, and input the initial feature map after channel mixing into the merging module for image stitching to obtain a final feature map; The model reconstruction module is used to input the final feature map into the Nerf renderer to reconstruct a 3D model corresponding to the target image.

10. A non-transitory computer-readable storage medium, characterized in that: The non-temporary computer-readable storage medium stores a 3D model reconstruction program based on a hybrid pulse neural network. When the 3D model reconstruction program based on a hybrid pulse neural network is executed by a processor, the steps of the 3D model reconstruction method based on a hybrid pulse neural network as described in any one of claims 1-8 are implemented.

Images