A visual signal representation device and method for micro-wavelet elements

Abstract

The present invention provides a visual signal representation device and method for tiny wavelets. The device includes a wavelet element generation module, a differentiable rasterization module, and a physical process guidance module. The wavelet element generation module is used to initialize and parameterize wavelet elements. The differentiable rasterization module is used to map wavelet elements into pixel representations and perform forward rendering and reverse gradient propagation processes. The physical process guidance module performs numerical processing related to a preset physical process on the rasterized wavelet elements to adaptively represent multi-dimensional visual signals. The present invention can adaptively model the frequency characteristics of visual signals in real scenes, and without the need to additionally introduce frequency constraints, it can achieve general and fast multi-dimensional visual signal representation tasks.

Description

Technical Field

[0001] The present invention belongs to the field of explicit visual signal representation, and particularly relates to a visual signal representation device and method with micro-wavelet primitives. Background Art

[0002] The continuous function representation of visual signals uses implicit or explicit continuous function structures to characterize signals, and has the advantages of end-to-end differentiability and continuous queryability. Therefore, such continuous representations can be seamlessly integrated with various imaging and rendering functions and are widely used in optimizing inverse problems.

[0003] However, existing representation methods have problems of spectral loss or slow rendering speed. For example, Implicit Neural Representation (INR) is often limited by spectral bias in representation ability, and only frequencies defined by hyperparameters can be effectively learned, resulting in the loss of information in other frequency bands. In addition, the complex network architecture of INR requires a large amount of computing resources to decode signal attributes, which will affect the processing speed in various visual tasks.

[0004] In addition to implicit neural representation methods, explicit visual signal representation methods have also received increasing attention. Among them, 3D Gaussian Splatting (3DGS) models visual signals by using multiple three-dimensional Gaussian functions to achieve fast and high-quality visual signal representation. However, the inherent low-pass characteristic of the Gaussian function makes it unable to effectively capture high-frequency details. To solve these low-pass characteristics, frequency regularization or modulation loss is usually adopted, which requires a large amount of professional knowledge and engineering experience to adjust the hyperparameters of different categories of visual signals. Summary of the Invention

[0005] Object of the Invention: The technical problem to be solved by the present invention is to provide a visual signal representation device with micro-wavelet primitives in view of the deficiencies of the prior art, which can adaptively model the frequency characteristics of visual signals in real scenes and achieve general and fast multi-dimensional visual signal representation. Another object of the present invention is to provide a method and application of the above device.

[0006] The present invention first provides a visual signal representation device with micro-wavelet primitives, including a wavelet primitive generation module, a differentiable rasterization module, and a physical process guidance module;

[0007] The wavelet primitive generation module is used to initialize and parameterize wavelet primitives;

[0008] The differentiable rasterization module is used to rasterize the wavelet primitives. The rasterization includes: mapping the wavelet primitives to pixel representations and performing forward rendering and backward gradient propagation. The input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in screen space, as well as the depth sorting table of all grids in screen space and the corresponding wavelet primitives.

[0009] The physical process guidance module performs numerical processing related to preset physical processes (such as two-dimensional image fitting, five-dimensional neural radiance field reconstruction, and six-dimensional dynamic neural radiance field reconstruction) on the rasterized wavelet primitives to adaptively represent multi-dimensional visual signals. The input of the physical process guidance module is the depth sorting table of grids and the corresponding wavelet primitives and the wavelet primitives in screen space, and the output is the multi-dimensional visual signal value.

[0010] In the wavelet primitive generation module, the initialization method of wavelet primitives is random initialization or initialization through the structure from motion algorithm. The parametric expression formula of wavelet primitives is:

[0011] ,

[0012] ,

[0013] ,

[0014] where represents the wavelet function expression in -dimensional space, represents the position vector of a point in the -dimensional space coordinate system, represents the position vector of the wavelet primitive in the -dimensional space coordinate system, represents the modulation frequency vector of the wavelet primitive in the represents the Gaussian distribution in the -dimensional space coordinate system, and respectively represent the rotation matrix and scaling matrix of size in the

[0015] The differentiable rasterization module performs rasterization using general wavelet sputtering, and the expression of the general wavelet sputtering is:

[0016] ,

[0017] After simplification, it is obtained:

[0018] ,

[0019] where represents the mapping from the -dimensional space coordinate system to the -dimensional space coordinate system, respectively represent the position vector of a point, the position vector of the wavelet basis element, the modulation frequency vector of the wavelet basis element, and the covariance matrix of size in the real number space, represents a certain dimension for mapping in the

[0020] The differentiable rasterization module uses a fast rasterization method for the forward rendering process, including: dividing the screen into grids, represents the number of pixels occupied by the grid in the screen coordinate system, calculating the intersection area between the circumscribed equilateral rectangle of the wavelet basis element in the middle of the screen and all grids;

[0021] For each wavelet basis element, check whether the circumscribed equilateral rectangle of the wavelet basis element intersects with the boundary region of the grid . If the intersection area is greater than 0, it is determined that the grid intersects with the wavelet basis element; add the grids that satisfy to the set C to obtain the sorting table of the wavelet basis element and the grid; then re-sort the sorting table of the wavelet basis element and the grid.

[0022] Furthermore, the intersection area between the circumscribed equilateral rectangle of the wavelet basis element in the middle of the screen and all grids is calculated by the following formula:

[0023] ,

[0024] ,

[0025] ,

[0026] ,

[0027] Where \(C\) represents the set of all grids intersecting with the wavelet basis elements, represents the th row and th column grid in the screen space, represents the area of the rectangle where the wavelet basis element intersects with the grid; , respectively represent the length and width of the rectangle where the wavelet basis element intersects with the grid, \(r\) represents the side length of the circumscribed equilateral rectangle of the wavelet basis element, respectively represent the abscissa and ordinate of the wavelet basis element in the screen coordinate system.

[0028] Furthermore, the differentiable rasterization module can also use other methods for the forward rendering process, such as treating the wavelet basis element as volume data to perform volume rasterization, hierarchical sampling to calculate the coverage rate estimation method, etc.

[0029] Furthermore, the differentiable rasterization module uses a custom backpropagation process to optimize the parameters. Taking the modulation frequency vector in the 3D experiment as an example, its backpropagation gradient can be calculated by the following formula:

[0030] ,

[0031] ,

[0032] where, represents the gradient of the loss function with respect to the frequency vector, used to optimize the frequency vector; represents the gradient of the loss function with respect to the entire wavelet basis element, provided by the Pytorch automatic differentiator; represents the gradient of the wavelet basis element with respect to the frequency vector .

[0033] The sorting table for reordering wavelet primitives and grids specifically includes: The differentiable rasterization module first encodes the depth values of wavelet primitives using 32-bit numbers (the depth value of a wavelet primitive is directly the coordinate value corresponding to the projection axis of each wavelet primitive before projecting onto the screen coordinate system. By default, the coordinate value of the z-axis is used as the depth value of the wavelet primitive. The encoding process method is a prior art and is consistent with the 3D Gaussian Splatting method), obtaining the sequence number value wavelet_idx of the wavelet primitive; encodes all grids in the screen space using 32-bit numbers, obtaining the tile index tile_idx of the wavelet primitive and the intersecting grids; combines the sequence number value wavelet_idx and the tile index tile_idx to obtain a 64-bit encoding in the form of (wavelet_idx, tile_idx), rearranges the 64-bit encoding, exchanges the encoding sequence numbers of the wavelet primitive and the grid, obtaining a 64-bit encoding in the form of (tile_idx, wavelet_idx); for different wavelet primitives in the grids with the same sequence number, sorts them in ascending order according to the last 32-bit encoding value, obtaining the depth sorting table of each grid and the corresponding wavelet primitive; uses the depth sorting table of the grid and the corresponding wavelet primitive and the wavelet primitive in the screen space as the input of the physical process guiding module.

[0034] Further, the physical process guiding module performs numerical processing related to a preset physical process on the rasterized wavelet primitives using a physical process algorithm, and the physical process algorithm includes: cumulative synthesis, alpha synthesis, and deformation field.

[0035] Further, the wavelet primitives of the present invention are not limited to the three representative physical processes of cumulative synthesis, alpha synthesis, and deformation field. Combining wavelet primitives in different dimensions with corresponding physical process processing modules can achieve other tasks such as Mesh reconstruction and digital human reconstruction.

[0036] The present invention also provides a visual signal representation method for differentiable wavelet primitives, including:

[0037] Explicitly represent the visual signal using differentiable wavelet primitives, and initialize and parameterize the wavelet primitives.

[0038] For the wavelet primitives, use the differentiable rasterization module to map the wavelet primitives to the screen space.

[0039] Use the physical process guiding module to perform numerical processing related to a preset physical process on the wavelet primitives in the screen space to achieve adaptive multi-dimensional visual signal representation.

[0040] Further, the visual signal representation method for differentiable wavelet primitives is applied to the two-dimensional image fitting task.

[0041] For two-dimensional image fitting, the wavelet basis elements are two-dimensional wavelet basis elements, the numerical processing method of the preset physical process is cumulative synthesis, and the output is RGB pixel values.

[0042] Furthermore, the visual signal representation method of the differentiable wavelet basis elements is applied to the five-dimensional neural radiance field reconstruction task.

[0043] For five-dimensional neural radiance field reconstruction, the wavelet basis elements are three-dimensional wavelet basis elements, the numerical processing method of the preset physical process is alpha synthesis, and the output is the RGB pixel values of the image in a given viewing direction where is the pitch angle, and is the yaw angle.

[0044] Furthermore, the visual signal representation method of the differentiable wavelet basis elements is applied to the six-dimensional dynamic neural radiance field reconstruction task.

[0045] For six-dimensional dynamic neural radiance field reconstruction, the wavelet basis elements are three-dimensional wavelet basis elements, the numerical processing method of the preset physical process is the deformation field, and the output is the RGB pixel values of the image in a given viewing direction at a specific time where is the pitch angle, and is the yaw angle.

[0046] Furthermore, the wavelet basis element generation module of the present invention can further optimize the fitting effect of static and dynamic neural radiance fields by combining schemes such as depth supervision, exposure compensation, and anti-aliasing, and achieve a new view synthesis task with higher fidelity.

[0047] The present invention also provides an electronic device, including a processor and a memory, where the memory stores program codes, and when the program codes are executed by the processor, the processor is caused to execute the steps of the method.

[0048] The present invention also provides a storage medium, storing a computer program or instruction, and when the computer program or instruction runs on a computer, the steps of the method are executed.

[0049] The present invention has the following beneficial effects: (1) The present invention proposes a general differentiable wavelet basis element, which can adaptively model various frequency characteristics of visual signals in real scenes without the need to additionally introduce frequency constraints.

[0050] (2) The present invention proposes a fast differentiable wavelet rasterizer and a general wavelet sputtering method, which can achieve anisotropic wavelet basis element sputtering and perform efficient backpropagation optimization, and can be applied to different dimensions to achieve general and effective visual representation.

[0051] (3) The present invention has shown excellent quality and speed performance in the applications of 2D image fitting, 5D neural radiance field reconstruction, and 6D dynamic neural radiance field reconstruction, outperforming the previous implicit neural representations and explicit 3D Gaussian point cloud representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The following further detailed description of the present invention will be made in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0053] Figure 1 Schematic diagram of the device for implementing the method of the present invention.

[0054] Figure 2 Comparison diagram of the results of the present invention in the 2D image fitting task.

[0055] Figure 3 Comparison diagram of the results of the present invention in the 5D neural radiance field reconstruction task.

[0056] Figure 4 Comparison diagram of the results of the present invention in the 6D dynamic neural radiance field reconstruction task. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Referring to Figure 1 As shown, the embodiment of the present invention provides a visual signal representation device with micro-wavelet primitives, including a wavelet primitive generation module, a differentiable rasterization module, and a physical process guidance module;

[0058] Among them, the wavelet primitive generation module is used to initialize and parameterize wavelet primitives;

[0059] The differentiable rasterization module is used to rasterize the wavelet primitives. The rasterization includes: mapping the wavelet primitives into pixel representations and performing forward rendering and reverse gradient propagation; the input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in screen space, as well as the raster T in screen space (i.e., Figure 1 the four rasters shown in , Figure 1 in representing the area of the intersection rectangle between the wavelet primitive and the corresponding raster ), and the depth sorting table of the corresponding wavelet primitives;

[0060] The physical process guidance module performs numerical processing related to a preset physical process on the rasterized wavelet primitives to adaptively represent multi-dimensional visual signals; the input of the physical process guidance module is the raster and the corresponding depth sorting table of the wavelet primitives and the wavelet primitives in screen space, and the output is the multi-dimensional visual signal value.

[0061] The wavelet primitive, denoted as Wavelet Primitive in English, is a signal representation primitive and a basic unit for multi-dimensional data modeling and visual signal representation based on wavelet functions. Wavelet functions have locality and multi-resolution characteristics, enabling good localization properties in both the time domain (or spatial domain) and the frequency domain. Each wavelet primitive is described by parameters such as position, frequency, rotation, and scaling, and its shape and frequency characteristics in multi-dimensional space are defined by a modulation frequency vector and a covariance matrix.

[0062] The preset physical processes include, for example, two-dimensional image fitting, five-dimensional neural radiance field reconstruction, and six-dimensional dynamic neural radiance field reconstruction.

[0063] By combining multiple wavelet primitives, complex signal models can be constructed, which are suitable for tasks such as two-dimensional image fitting, five-dimensional neural radiance field reconstruction, and six-dimensional dynamic scene modeling. Compared with traditional primitives (such as Gaussian functions), wavelet primitives are more suitable for characterizing the fine-grained characteristics of signals and can achieve fast rendering and adaptive optimization while maintaining high fidelity. In addition, during the implementation of wavelet primitives, by utilizing the differentiability of their derivatives, they not only support efficient forward modeling but also can be adaptively optimized through gradient propagation, enabling them to play an important role in modern machine learning and computer vision.

[0064] In the wavelet primitive generation module, the initialization method of wavelet primitives is random initialization or initialization through the structure from motion algorithm. The parametric expression formula of wavelet primitives is:

[0065] ,

[0066] ,

[0067] ,

[0068] where, represents the expression of the wavelet function in the represents the transpose operator, represents the position vector of a point in the represents the position vector of the wavelet primitive in the represents the Gaussian distribution in the represents the covariance matrix of size in the and respectively represent the covariance matrix of size Rotation matrix and scaling matrix. Denote as the modulation frequency vector of the wavelet basis element in the d-dimensional space coordinate system, where the center position of the frequency response of the wavelet basis element changes with 's movement and appears as a band-pass filter in the frequency domain. Therefore, different modulation frequency vectors can be used to efficiently represent different frequency features of visual signals. While the traditional Gaussian basis element has a frequency response concentrated at zero frequency, decaying on both sides in a bell shape, and appears as a low-pass filter in the frequency domain. It can only represent low-frequency visual signals and is prone to causing excessive computational load when representing high-frequency features of visual signals.

[0069] This embodiment also provides a method for representing multi-dimensional visual signals using the above device. The specific steps include: explicitly representing visual signals using differentiable wavelet basis elements. First, initialize and parameterize the wavelet basis elements using a random or structure-from-motion algorithm. Then, for the wavelet basis elements, use a differentiable rasterizer to map the wavelet basis elements to screen space. Finally, use the wavelet basis elements in screen space to perform various physical processes (such as two-dimensional image fitting, five-dimensional neural radiance field reconstruction, and six-dimensional dynamic neural radiance field reconstruction), and adaptively optimize and update the position vectors of the wavelet basis elements during the training process. Rotation matrix , scaling matrix and modulation frequency vector to accurately capture the frequency details of visual signals and achieve efficient representation of various visual signals.

[0070] Among them, the specific implementation steps of the differentiable rasterizer include:

[0071] Step a1, calculate the parameter vector of the wavelet basis element in screen space after general wavelet sputtering ; Denote respectively the position vector of a point in the d-dimensional space coordinate system, the position vector of the wavelet basis element, the modulation frequency vector of the wavelet basis element, and the covariance matrix of size ; where general wavelet sputtering can support d-dimensional wavelet projection to d - 1 dimensions and efficiently implement the interaction between wavelet basis elements and pixels within the screen.

[0072] Step a2, calculate the intersection area between the wavelet basis element in the middle of the screen and all grids. For each wavelet basis element, check whether the circumscribed equilateral rectangle of the wavelet basis element intersects with the boundary region of the grid . If the intersection area is greater than 0, it is determined that the grid intersects with the wavelet basis element. Those satisfying The grids are added to the set C to obtain a sorting table of wavelet basis elements and grids;

[0073] Step a3, re-sort the sorting table of wavelet basis elements and grids to obtain a sorting table of grids and wavelet basis elements, which is used as the input of the physical process guiding module.

[0074] The re-sorting of the sorting table of wavelet basis elements and grids specifically includes: The differentiable rasterization module first uses 32-bit digital encoding for the depth values of wavelet basis elements to obtain the serial number value wavelet_idx of wavelet basis elements; uses 32-bit digital encoding for all grids in the screen space to obtain the number tile_idx of the wavelet basis element and the intersecting grids; combines the serial number value wavelet_idx with the number tile_idx to obtain a 64-bit encoding in the form of (wavelet_idx, tile_idx), re-arrange the 64-bit encoding, and exchange the encoding serial numbers of wavelet basis elements and grids to obtain a 64-bit encoding in the form of (tile_idx, wavelet_idx); for different wavelet basis elements in grids with the same serial number, sort them in ascending order according to the last 32-bit encoding value to obtain a depth sorting table of each grid and its corresponding wavelet basis element; use the depth sorting table of each grid and its corresponding wavelet basis element and the wavelet basis elements in the screen space as the input of the physical process guiding module.

[0075] In the present invention, the numerical processing of the physical process guiding module can be three types of physical process numerical processing methods, for example, cumulative synthesis, alpha synthesis, and deformation field. The specific implementation steps of the numerical processing of the physical process guiding module include:

[0076] Step b1, select the corresponding physical process guidance according to different tasks. For the two-dimensional image fitting task, use the cumulative synthesis algorithm to guide the wavelet basis elements; for the five-dimensional neural radiance field reconstruction task, use the alpha synthesis algorithm to guide the wavelet basis elements; for the six-dimensional dynamic neural radiance field reconstruction task, use the deformation field algorithm to guide the wavelet basis elements.

[0077] In the embodiment of the present application, the cumulative synthesis algorithm includes: In the execution of the two-dimensional image fitting task, it is necessary to perform numerical processing on all two-dimensional wavelet basis elements in the plane to calculate the image pixel values. The image pixel value refers to the pixel color value of the image. Each pixel color value in the image can be obtained by cumulatively weighting the color values of multiple wavelet basis elements covering the pixel, and the calculation formula is:

[0078] ,

[0079] where N represents the total number of wavelet basis elements currently covering the pixel, represents the color value of the wavelet basis element, represents the opacity value of the wavelet basis element, Represents the current image pixel value.

[0080] In the embodiments of the present application, the alpha compositing refers to that in the execution of five-dimensional neural radiance field reconstruction, numerical processing needs to be performed on all three-dimensional wavelet basis elements in space to render a reconstructed image with multi-view consistency. For example, a single pixel in an image at a certain view can be composed of multiple wavelet basis elements covering the pixel:

[0081] ;

[0082] In the embodiments of the present application, the deformation field refers to that in the execution of six-dimensional dynamic neural radiance field reconstruction tasks, for a given time t, various attribute values (position vector , frequency vector and covariance matrix ) of the wavelet basis elements are predicted by means of a fully connected network MLP, and then the alpha compositing is continued to obtain the image pixel value at the current moment t.

[0083] Step b2, calculate the loss between the obtained visual signal and the real visual signal, and perform the backpropagation process using an explicit gradient calculation formula. In the embodiments of the present application, the backpropagation adopts the Pytorch custom differential framework. Through the Pytorch custom differential framework, custom backpropagation can be realized, thereby improving the parameter optimization efficiency of the wavelet basis elements.

[0084] The differentiable rasterizer and the physical process guiding method of the present invention can accurately capture the frequency details of visual signals and achieve a more accurate display representation of visual signals with complex frequency distributions. Referring to the differentiable rasterizer shown in Figure 1 , it shows that the wavelet basis elements are mapped to the screen space, and the calculation formula of the mapped wavelet basis elements in the screen space is:

[0085] ,

[0086] After simplification, it is obtained:

[0087] ,

[0088] where represents the mapping from the -dimensional space coordinate system to the -dimensional space coordinate system, represents the real number space, represents A certain dimension for mapping in the N-dimensional space coordinate system. According to the frequency response characteristics of wavelet basis elements, different modulation frequency vectors can enable wavelet basis elements to efficiently and accurately represent the high and low frequency characteristics of visual signals. By adaptively optimizing the modulation frequency vector and other parameter vectors during training, the differentiable wavelet basis elements can effectively capture the time-frequency characteristics of visual signals, thereby representing visual signals with complex frequencies more precisely and efficiently.

[0089] An embodiment of the present invention also provides a method for representing visual signals with differentiable wavelet basis elements, including: explicitly representing visual signals using differentiable wavelet basis elements. First, initialize and parameterize the wavelet basis elements, then, for the wavelet basis elements, use a differentiable rasterization module to map the wavelet basis elements to the screen space, and finally, perform various physical processes using the wavelet basis elements in the screen space, and adaptively optimize and update the parameters of the wavelet basis elements during the training process.

[0090] The differentiable rasterization module uses a fast rasterization method to divide the screen into grids, represents the number of pixels occupied by the grid in the screen coordinate system, and the following formula is used to calculate the intersection area between the wavelet basis element in the middle of the screen and all grids:

[0091] ,

[0092] ,

[0093] ,

[0094] ,

[0095] where C represents the set of all grids intersecting with the wavelet basis element, represents the th row and th column grid in the screen space, represents the area of the intersection rectangle between the wavelet basis element and the grid; , respectively represent the length and width of the intersection rectangle between the wavelet basis element and the grid, r represents the side length of the circumscribed equilateral rectangle of the wavelet basis element, respectively represent the abscissa and ordinate of the wavelet basis element in the screen coordinate system.

[0096] The device and method of the present invention can be applied to various visual signal representation tasks. In this embodiment, it is denoted as wavelet visual primitives. The benchmark methods for comparison in the two-dimensional image fitting task are: Gaussian visual primitives and wavelet implicit neural representation. The benchmark methods for comparison in the five-dimensional neural radiance field reconstruction task are: Gaussian visual primitives, unbounded anti-aliasing neural radiance field, all-optical voxel radiance field, and hash neural primitives. The benchmark methods for comparison in the six-dimensional dynamic neural radiance field reconstruction task are: Gaussian visual primitives, dynamic implicit neural representation, and dynamic specular implicit neural representation. The specific implementation details vary due to different tasks. Therefore, the specific implementation of different tasks will be described below, including two-dimensional image representation, five-dimensional neural radiance field reconstruction, and six-dimensional dynamic neural radiance field reconstruction.

[0097] For the two-dimensional image fitting task, all comparison methods adopt the default parameter settings. The wavelet visual primitives adopt two-dimensional wavelet primitives and are guided by the cumulative synthesis algorithm. The number of wavelet visual primitives is the same as the training settings of Gaussian visual primitives, that is, the primitive parameters are randomly initialized, the number is set to 70k, and the training iteration is 50,000 times. Figure 2 Shows the result comparison of two-dimensional image fitting ( Figure 2 The privacy information involved is blurred in . The true value image is an image with a resolution of

[0098] Among the three methods of wavelet visual primitives (i.e., the method of the present invention), Gaussian visual primitives, and wavelet implicit neural representation, the method of the present invention achieves clearer and more accurate results. For example, the tiles and sculptures of the temple and the windows and street lights of the church shown in the enlarged view. However, these parts are over-smoothed in the results of the Gaussian visual primitive method and the wavelet implicit neural representation method. Figure 3 Shows the result comparison of indoor and outdoor scenes in the dataset ( Figure 3(The privacy information involved is blurred). The method of the present invention demonstrates advantages in representing aspects of scenes with complex frequency components. For example, the frames of portraits and plants in potted plants in indoor scenes, the windshields of trucks and transmission towers in the distance of trains in outdoor scenes. The highlight reflections or complex textures in these areas are over-smoothed or the reconstruction fails in the reconstruction of the four benchmark methods of Gaussian visual primitives, unbounded anti-aliased neural radiance fields, all-optical voxel radiance fields, and hash neural primitives. Only the wavelet visual primitives used in the present invention can reconstruct clear details in these areas.

[0099] For the six-dimensional dynamic neural radiance field reconstruction task, all comparison methods adopt default parameter settings. The wavelet visual primitives adopt three-dimensional wavelet primitives and are guided by the deformation field algorithm. The deformation field uses a neural network with 8 hidden layers, each layer having 256 neurons, to model the matching function between the parameter vector of the wavelet visual primitives and time t. The input of the network is the parameter vector of the wavelet primitive and time t, and the output is the offset of the parameter vector . The wavelet visual primitives adopt the same densification method and training parameter vector settings as the Gaussian visual primitives, that is, the primitives are initialized by the structure from motion algorithm, the number of pre-training iterations is 3k, the total number of training iterations is 20k, the gradient densification threshold is set to 0.0002, the starting iteration number for densification is 500, the ending iteration number for densification is 15k, the densification iteration interval is 100, the number of training iterations is 20k, and the loss function is supervised by a mixed loss of L1-Loss and SSIM. Training and testing are carried out on the virtual synthetic dataset D-NeRF and the real-scene dataset NeRF-DS. It should be particularly noted that the learning rate of the modulation frequency vector of the wavelet visual primitives here is set to 0.0025. Figure 4 The result comparison is shown ( Figure 4 The privacy information involved is blurred). The method of the present invention demonstrates the advantage of simultaneously representing high-frequency and low-frequency components in the scene. For example, the sleeve part when the virtual person jumps, the cuffed part of the pants when the virtual person squats, and the tail part of the dinosaur in the virtual synthetic dataset, the curtain and corner parts in the real dataset. The two benchmark methods of Gaussian visual primitives and dynamic mirror implicit neural representation cannot accurately reconstruct low-frequency and high-frequency components simultaneously. The present invention can capture and represent rich frequency components in dynamic scenes simultaneously.

[0100] The present invention provides a visual signal representation device and method for micro-wave elements. There are many ways to specifically implement this technical solution. The above description is only a preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by existing technologies.

Claims

1. A visual signal representation device capable of representing microwavelet primitives, characterized in that: It includes wavelet primitive generation module, differentiable rasterization module and physical process guidance module; The wavelet primitive generation module is used to initialize and parameterize the wavelet primitive; The differentiable rasterization module is used to rasterize the wavelet primitives, and the rasterization includes: mapping the wavelet primitives into pixel representations, and performing forward rendering and reverse gradient propagation; the input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in the screen space, and a depth sorting table of all grids in the screen space and the corresponding wavelet primitives; The physical process guiding module performs numerical processing of a preset physical process on the rasterized wavelet primitives to adaptively represent multi-dimensional visual signals; the input of the physical process guiding module is a depth sorting table of grids and corresponding wavelet primitives and the wavelet primitives in the screen space, and the output is a multi-dimensional visual signal value; The wavelet primitive initialization method in the wavelet primitive generation module is random initialization or initialization by motion recovery structure algorithm. The parameterized expression formula of the wavelet primitive is: in, represents the wavelet function expression in d-dimensional space, represents the transpose operator, represents the position vector of a point in the d-dimensional space coordinate system, represents the position vector of the wavelet primitive in the d-dimensional space coordinate system, represents the modulation frequency vector of the wavelet primitive in the d-dimensional space coordinate system, represents the Gaussian distribution of the d-dimensional space coordinate system, Σ represents the covariance matrix of size d×d in the d-dimensional space coordinate system, R and S represent the rotation matrix and scaling matrix of size d×d in the d-dimensional space coordinate system, respectively, and e represents a natural constant; The differentiable rasterization module uses general wavelet sputtering for rasterization, and the expression of the general wavelet sputtering is: After simplification, we get: in Represents the mapping from the d-dimensional space coordinate system to the d-1-dimensional space coordinate system, Σ′ represents the position vector of the point in the d-1 dimensional space coordinate system, the position vector of the wavelet primitive, the modulation frequency vector of the wavelet primitive, and the covariance matrix of size (d-1)×(d-1). represents the real number space, x d Represents a dimension of mapping in the d-dimensional space coordinate system.

2. The device according to claim 1, characterized in that The differentiable rasterization module adopts a fast rasterization method to perform a forward rendering process, including: dividing the screen into n×n grids, where n represents the number of pixels occupied by the grid in the screen coordinate system, and calculating the intersection area between the circumscribed equilateral rectangle of the wavelet primitive in the middle of the screen and all the n×n grids; For each wavelet primitive, check whether the bounding equilateral rectangle of the wavelet primitive is consistent with the grid T. ij The boundary areas of the two sets intersect if the area of ​​the intersection is S. ij If it is greater than 0, then the grid T ij Intersect with the wavelet primitive; will satisfy S ij >0 are added to set C to obtain the sorting table of wavelet primitives and grids; then the sorting table of wavelet primitives and grids is re-sorted.

3. The device according to claim 2, characterized in that The intersection area of ​​the equilateral rectangle of the wavelet primitive in the middle of the screen and all n×n grids is calculated by the following formula: C={T ij ∣S ij >0}, S ij =ΔxΔy, Δx=min{μ x +r,n(i+1)}-max{μ x -r,ni}, Δy=min{μ y +r,n(j+1)}-max{μ y -r,nj}, Where C represents the set of all grids intersecting with the wavelet primitives, T ij represents the grid at row i and column j in screen space, S ij represents the area of ​​the rectangle where the wavelet primitive and the grid intersect; Δx, Δy represent the length and width of the rectangle where the wavelet primitive and the grid intersect, ni represents the product of n and i, nj represents the product of n and j, r represents the side length of the equilateral rectangle circumscribing the wavelet primitive, μ x ,μ y They respectively represent the horizontal and vertical coordinates of the wavelet primitive in the screen coordinate system.

4. The device according to claim 3, characterized in that The reordering of the sorting table of wavelet primitives and grids specifically includes: the differentiable rasterization module first uses 32-bit numbers to encode the depth value of the wavelet primitive to obtain the serial number value wavelet_idx of the wavelet primitive; uses 32-bit numbers to encode all grids in the screen space to obtain the serial number tile_idx of the wavelet primitive and the intersecting grid; combines the serial number value wavelet_idx with the serial number tile_idx to obtain a 64-bit code in the form of (wavelet_idx, tile_idx), rearranges the 64-bit code, exchanges the coding serial numbers of the wavelet primitive and the grid, and obtains a 64-bit code in the form of (tile_idx, wavelet_idx); for different wavelet primitives in the grid with the same serial number, sorts them from small to large according to the last 32-bit coding value to obtain a depth sorting table of each grid and the corresponding wavelet primitive; and uses the depth sorting table of the grid and the corresponding wavelet primitive and the wavelet primitive in the screen space as inputs of the physical process guidance module.

5. A method for representing visual signals in microwavelet primitives implemented by the device according to any one of claims 1 to 4, characterized in that: include: Use minimizable wavelet primitives to explicitly represent visual signals, initialize and parameterize the wavelet primitives; For the wavelet primitive, a differentiable rasterization module is used to map the wavelet primitive to a screen space; A physical process guided module is used to perform numerical processing of preset physical processes on wavelet primitives in screen space to achieve adaptive multi-dimensional visual signal representation.

6. The method according to claim 5, characterized in that The visual signal representation method using microwavelet primitives is applied to two-dimensional image fitting tasks. For two-dimensional image fitting, the wavelet primitive is a two-dimensional wavelet primitive, the numerical processing method of the preset physical process is cumulative synthesis, and the output is an RGB pixel value.

7. The method according to claim 6, characterized in that The visual signal representation method of the micro-wavelet primitives is applied to the five-dimensional neural radiation field reconstruction task. For the reconstruction of the five-dimensional neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is alpha synthesis, and the output is the RGB pixel value of the image in a given observation direction.

8. The method according to claim 7, characterized in that The visual signal representation method using microwavelet primitives is applied to the task of reconstructing a six-dimensional dynamic neural radiation field. For the reconstruction of the six-dimensional dynamic neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is a deformation field, and the output is the RGB pixel value of the image in a given observation direction at time t.

Images