Visual signal representation device and method capable of realizing tiny wave primitives

Abstract

The invention provides a visual signal representation device and method capable of realizing tiny wave primitives. The device comprises a wavelet primitive generation module, a differential rasterization module and a physical process guide module, the wavelet primitive generation module is used for initializing and parameterizing wavelet primitives; the differentiable rasterization module is used for mapping wavelet elements into pixel representation and executing forward rendering and reverse gradient propagation processes; and the physical process guiding module executes numerical processing related to a preset physical process on the rasterized wavelet elements for self-adaptive multi-dimensional visual signal representation. According to the method, the frequency characteristics of the visual signals in a real scene can be modeled in a self-adaptive mode, frequency constraint does not need to be additionally introduced, and a universal and rapid multi-dimensional visual signal representation task is achieved.

Description

Technical Field

[0001] The invention belongs to the field of explicit visual signal representation, and in particular relates to a visual signal representation device and method capable of representing microwavelet primitives. Background Art

[0002] The continuous function representation of visual signals uses implicit or explicit continuous function structures to characterize signals, which has the advantages of end-to-end differentiability and continuous query. Therefore, this type of continuous representation can be seamlessly integrated with various imaging and rendering functions and is widely used in optimizing inverse problems.

[0003] However, existing representation methods suffer from spectral loss or slow rendering speed. For example, implicit neural representation (INR) is often limited in representation ability by spectral bias, and only the frequencies defined by hyperparameters can be effectively learned, resulting in the loss of information in other frequency bands. In addition, the complex network architecture of INR requires a lot of computing resources to decode signal properties, which affects processing speed in various visual tasks.

[0004] In addition to implicit neural representation methods, explicit visual signal representation methods are also gaining more and more attention. Among them, 3D Gaussian Splatting (3DGS) achieves fast and high-quality visual signal representation by modeling visual signals using multiple three-dimensional Gaussian functions. However, the inherent low-pass characteristics of the Gaussian function make it unable to effectively capture high-frequency details. In order to solve these low-pass characteristics, frequency regularization or modulation loss is usually used, but this requires a lot of expertise and engineering experience to adjust the hyperparameters of different categories of visual signals. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a visual signal representation device that can represent microwavelet primitives, which can adaptively model the frequency characteristics of visual signals in real scenes and realize universal and fast multi-dimensional visual signal representation. Another purpose of the present invention is to provide a method and application of the above device.

[0006] The present invention firstly provides a visual signal representation device that can be divided into wavelet primitives, including a wavelet primitive generation module, a differentiable rasterization module and a physical process guidance module; The wavelet primitive generation module is used to initialize and parameterize the wavelet primitive; The differentiable rasterization module is used to rasterize the wavelet primitives, and the rasterization includes: mapping the wavelet primitives into pixel representations, and performing forward rendering and reverse gradient propagation; the input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in the screen space, and a depth sorting table of all grids in the screen space and the corresponding wavelet primitives; The physical process guidance module performs numerical processing related to preset physical processes (such as two-dimensional image fitting, five-dimensional neural radiation field reconstruction and six-dimensional dynamic neural radiation field reconstruction) on the rasterized wavelet primitives for adaptive multi-dimensional visual signal representation; the input of the physical process guidance module is a depth sorting table of the grid and the corresponding wavelet primitives and the wavelet primitives in the screen space, and the output is a multi-dimensional visual signal value.

[0007] The wavelet primitive initialization method in the wavelet primitive generation module is random initialization or initialization by motion recovery structure algorithm. The parameterized expression formula of the wavelet primitive is: , , , in, express The wavelet function expression in dimensional space, represents the transpose operator, express The position vector of a point in the dimensional space coordinate system, express The position vector of the wavelet primitive in the dimensional space coordinate system, express The modulation frequency vector of the wavelet primitive in the dimensional space coordinate system, express Gaussian distribution of the dimensional space coordinate system, express The size of the coordinate system in the dimensional space is The covariance matrix of and Respectively The size of the coordinate system in the dimensional space is The rotation matrix and scaling matrix of , e represents a natural constant. The wavelet primitives generated by the wavelet primitive generation module of the present invention are parameterized using Morlet wavelet expressions, and support continuous wavelet families such as Shannon wavelet and Ricker wavelet.

[0008] The differentiable rasterization module uses general wavelet sputtering for rasterization, and the expression of the general wavelet sputtering is: , After simplification, we get: , in express dimensional space coordinate system to dimensional space coordinate system, Respectively The position vector of the point in the dimensional space coordinate system, the position vector of the wavelet primitive, the modulation frequency vector of the wavelet primitive, and the magnitude of The covariance matrix of represents the real number space, express A dimension in the dimensional space coordinate system for mapping.

[0009] The differentiable rasterization module uses a fast rasterization method to perform a forward rendering process, including: dividing the screen into The grid, Represents the number of pixels occupied by the grid in the screen coordinate system, and calculates the equilateral rectangle of the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid; For each wavelet primitive, check whether the bounding rectangle of the wavelet primitive is consistent with the grid. The boundary areas of the two sets intersect if the area of ​​the intersecting area is If it is greater than 0, the grid is judged Intersect with the wavelet primitives; will satisfy The grid is added to set C to obtain the sorting table of wavelet primitives and grids; then the sorting table of wavelet primitives and grids is re-sorted.

[0010] Further, the equilateral rectangle of the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid is: , , , , Where C represents the set of all grids intersecting with the wavelet primitives, Indicates the screen space Line A grid of columns, Represents the area of ​​the rectangle where the wavelet primitive intersects the grid; , They represent the length and width of the rectangle where the wavelet primitive and the grid intersect, r represents the side length of the equilateral rectangle circumscribing the wavelet primitive, They respectively represent the horizontal and vertical coordinates of the wavelet primitive in the screen coordinate system.

[0011] Furthermore, the differentiable rasterization module may also use other methods to perform the forward rendering process, such as treating wavelet primitives as volume data to perform the volume rasterization process, layered sampling calculation coverage estimation method, etc.

[0012] Furthermore, the differentiable rasterization module uses a custom back-propagation process to optimize parameters. For example, the back propagation gradient can be calculated by the following formula: , , in, Represents the gradient of the loss function with respect to the frequency vector, which is used to optimize the frequency vector; represents the gradient of the loss function with respect to the entire wavelet primitive, provided by the Pytorch automatic differentiator; Represents the frequency vector of wavelet primitives gradient.

[0013] The reordering of the sorting table of wavelet primitives and grids specifically includes: the differentiable rasterization module first uses 32-bit digital encoding of wavelet primitive depth values ​​(the wavelet primitive depth value is the coordinate value corresponding to the projection axis of each wavelet primitive directly taken before projection to the screen coordinate system, and the coordinate value of the z-axis is used as the depth value of the wavelet primitive by default. The encoding process method is the existing technology and is consistent with the 3D Gaussian Splatting method) to obtain the sequence value wavelet_idx of the wavelet primitive; uses 32-bit digital encoding of all grids in the screen space to obtain the tile_idx of the wavelet primitive and the intersecting grid; combines the sequence value wavelet_idx with the tile_idx to obtain a 64-bit code in the form of (wavelet_idx, tile_idx), rearranges the 64-bit code, and exchanges the coding sequence numbers of the wavelet primitive and the grid to obtain a (tile_idx, wavelet_idx); for different wavelet primitives in the same sequence number grid, sort them from small to large according to the last 32-bit code value, and get the depth sorting table of each grid and the corresponding wavelet primitive; the depth sorting table of the grid and the corresponding wavelet primitive and the wavelet primitive in the screen space are used as the input of the physical process guidance module.

[0014] Furthermore, the physical process guiding module uses a physical process algorithm to perform numerical processing related to a preset physical process on the rasterized wavelet primitives, and the physical process algorithm includes: cumulative synthesis, alpha synthesis and deformation field.

[0015] Furthermore, the wavelet primitives of the present invention are not limited to the three representative physical processes of cumulative synthesis, alpha synthesis and deformation field. By combining wavelet primitives of different dimensions with corresponding physical process processing modules, other tasks such as Mesh reconstruction and digital human reconstruction can be achieved.

[0016] The present invention also provides a method for representing visual signals using microwavelet primitives, comprising: Visual signals are explicitly represented using minimizable wavelet primitives, which are initialized and parameterized.

[0017] For the wavelet primitives, a differentiable rasterization module is used to map the wavelet primitives to the screen space.

[0018] A physical process guided module is used to perform numerical processing related to preset physical processes on wavelet primitives in screen space to achieve adaptive multi-dimensional visual signal representation.

[0019] Furthermore, the visual signal representation method using microwavelet primitives is applied to two-dimensional image fitting tasks.

[0020] For two-dimensional image fitting, the wavelet primitive is a two-dimensional wavelet primitive, the numerical processing method of the preset physical process is cumulative synthesis, and the output is an RGB pixel value.

[0021] Furthermore, the visual signal representation method of microwavelet primitives is applied to the five-dimensional neural radiation field reconstruction task.

[0022] For the reconstruction of the five-dimensional neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is alpha synthesis, and the output is a given observation direction. The image RGB pixel value is Pitch angle, is the yaw angle.

[0023] Furthermore, the visual signal representation method using microwavelet primitives is applied to the task of reconstructing six-dimensional dynamic neural radiation fields.

[0024] For the reconstruction of the six-dimensional dynamic neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is a deformation field, and the output is a specific time A given viewing direction The image RGB pixel value, where is the pitch angle, is the yaw angle.

[0025] Furthermore, the wavelet primitive generation module of the present invention can be combined with depth supervision, exposure compensation, anti-aliasing and other schemes to further optimize the fitting effect of static and dynamic neural radiation fields, and achieve new perspective synthesis tasks with higher fidelity.

[0026] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0027] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0028] The present invention has the following beneficial effects: (1) The present invention proposes a universal microwavelet primitive that can adaptively model various frequency characteristics of visual signals in real scenes without introducing additional frequency constraints.

[0029] (2) The present invention proposes a fast and reproducible wavelet rasterizer and a universal wavelet sputtering method, which can realize anisotropic wavelet primitive sputtering and perform efficient back-propagation optimization. It can be applied to different dimensions to achieve universal and effective visual representation.

[0030] (3) The present invention has good quality and speed performance in the applications of two-dimensional image fitting, five-dimensional neural radiation field reconstruction, and six-dimensional dynamic neural radiation field reconstruction, surpassing the previous implicit neural representation and explicit 3D Gaussian point cloud representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0032] Figure 1 The figure is a schematic diagram of a device for implementing the method of the present invention.

[0033] Figure 2 This is a comparison chart of the results of the present invention in the two-dimensional image fitting task.

[0034] Figure 3 This is a comparison chart of the results of the present invention in the five-dimensional neural radiation field reconstruction task.

[0035] Figure 4 This is a comparison chart of the results of the six-dimensional dynamic neural radiation field reconstruction task of the present invention. DETAILED DESCRIPTION

[0036] Reference Figure 1 As shown, an embodiment of the present invention provides a visual signal representation device that can be divided into wavelet primitives, including a wavelet primitive generation module, a differentiable rasterization module and a physical process guidance module; Wherein, the wavelet primitive generation module is used to initialize and parameterize the wavelet primitive; The differentiable rasterization module is used to rasterize the wavelet primitives, and the rasterization includes: mapping the wavelet primitives into pixel representations, and performing forward rendering and reverse gradient propagation; the input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in the screen space and the grid T in the screen space (that is, Figure 1 The four grids shown in , Figure 1 middle Represents wavelet primitives and corresponding grids The area of ​​the intersecting rectangle) and the depth sorting table of the corresponding wavelet primitives; The physical process guiding module performs numerical processing related to preset physical processes on the rasterized wavelet primitives for adaptive multi-dimensional visual signal representation; the input of the physical process guiding module is a depth sorting table of grids and corresponding wavelet primitives and the wavelet primitives in screen space, and the output is a multi-dimensional visual signal value.

[0037] The wavelet primitive is a signal characterization primitive, which is a basic unit for multidimensional data modeling and visual signal representation based on wavelet functions. Wavelet functions have locality and multi-resolution characteristics, and can have good localization characteristics in both the time domain (or space domain) and the frequency domain. Each wavelet primitive is described by parameters such as position, frequency, rotation and scaling, and its shape and frequency characteristics in multidimensional space are defined by the modulation frequency vector and covariance matrix.

[0038] The preset physical process includes, for example, two-dimensional image fitting, five-dimensional neural radiation field reconstruction, and six-dimensional dynamic neural radiation field reconstruction.

[0039] By combining multiple wavelet primitives, complex signal models can be constructed, which are suitable for tasks such as two-dimensional image fitting, five-dimensional neural radiation field reconstruction, and six-dimensional dynamic scene modeling. Compared with traditional primitives (such as Gaussian functions), wavelet primitives are more suitable for characterizing the fine-grained characteristics of signals and achieving fast rendering and adaptive optimization while maintaining high fidelity. In addition, in the implementation process, wavelet primitives use the differentiability of their derivatives to not only support efficient forward modeling, but also perform adaptive optimization through gradient propagation, enabling them to play an important role in modern machine learning and computer vision.

[0040] The wavelet primitive initialization method in the wavelet primitive generation module is random initialization or initialization by motion recovery structure algorithm. The parameterized expression formula of the wavelet primitive is: , , , in, express The wavelet function expression in dimensional space, represents the transpose operator, express The position vector of a point in the dimensional space coordinate system, express The position vector of the wavelet primitive in the dimensional space coordinate system, express Gaussian distribution of the dimensional space coordinate system, express The size of the coordinate system in the dimensional space is The covariance matrix of and Respectively The size of the coordinate system in the dimensional space is The rotation matrix and scaling matrix. express The modulation frequency vector of the wavelet primitive in the dimensional space coordinate system. The center position of the frequency response of the wavelet primitive varies with The frequency domain behaves as a bandpass filter. It can efficiently represent the different frequency characteristics of visual signals. The frequency response is concentrated at zero frequency and decays to both sides in a bell shape. It behaves as a low-pass filter in the frequency domain and can only characterize low-frequency visual signals. It is very easy to cause excessive computational load when characterizing the high-frequency characteristics of visual signals.

[0041] This embodiment also provides a method for representing a multi-dimensional visual signal using the device, and the specific steps include: using a micro-wavelet primitive to explicitly represent the visual signal, first, using a random or motion recovery structure algorithm to initialize and parameterize the wavelet primitive, then, for the wavelet primitive, using a differentiable rasterizer to map the wavelet primitive to the screen space, and finally, using the wavelet primitive in the screen space to perform various physical processes (such as two-dimensional image fitting, five-dimensional neural radiation field reconstruction, and six-dimensional dynamic neural radiation field reconstruction), and adaptively optimizing and updating the position vector of the wavelet primitive during the training process. Rotation Matrix , scaling matrix and the modulation frequency vector , accurately capture the frequency details of visual signals to achieve efficient representation of various visual signals.

[0042] The specific implementation steps of the differentiable rasterizer include: Step a1, calculate the parameter vector of the wavelet primitive after general wavelet splashing in the screen space ; Respectively The position vector of the point in the dimensional space coordinate system, the position vector of the wavelet primitive, the modulation frequency vector of the wavelet primitive, and the magnitude of The covariance matrix of ; Among them, the general wavelet sputtering can support d-dimensional wavelet projection d-1 dimension, and efficiently realize the interaction between wavelet primitives and pixels on the screen.

[0043] Step a2, calculate the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid For each wavelet primitive, check whether the bounding equilateral rectangle of the wavelet primitive is consistent with the grid The boundary areas of If is greater than 0, the grid is judged to intersect with the wavelet primitive. Add the grids of to the set C to get the sorting table of wavelet primitives and grids; Step a3, re-sorting the sorting table of wavelet primitives and grids to obtain a sorting table of grids and wavelet primitives, which is used as an input of the physical process guiding module.

[0044] The reordering of the sorting table of wavelet primitives and grids specifically includes: the differentiable rasterization module first uses 32-bit numbers to encode the depth value of the wavelet primitive to obtain the serial number value wavelet_idx of the wavelet primitive; uses 32-bit numbers to encode all grids in the screen space to obtain the serial number tile_idx of the wavelet primitive and the intersecting grid; combines the serial number value wavelet_idx with the serial number tile_idx to obtain a 64-bit code in the form of (wavelet_idx, tile_idx), rearranges the 64-bit code, exchanges the coding serial numbers of the wavelet primitive and the grid, and obtains a 64-bit code in the form of (tile_idx, wavelet_idx); for different wavelet primitives in the grid with the same serial number, sorts them from small to large according to the last 32-bit coding value to obtain a depth sorting table of each grid and the corresponding wavelet primitive; and uses the depth sorting table of the grid and the corresponding wavelet primitive and the wavelet primitive in the screen space as inputs of the physical process guidance module.

[0045] In the present invention, the numerical processing of the physical process guidance module can be three types of physical process numerical processing methods, such as cumulative synthesis, alpha synthesis and deformation field. The specific implementation steps of the numerical processing of the physical process guidance module include: Step b1, select the corresponding physical process guidance according to different tasks. For the two-dimensional image fitting task, the cumulative synthesis algorithm is used to guide the wavelet primitives. For the five-dimensional neural radiation field reconstruction task, the alpha synthesis algorithm is used to guide the wavelet primitives. For the six-dimensional dynamic neural radiation field reconstruction task, the deformation field algorithm is used to guide the wavelet primitives.

[0046] In the embodiment of the present application, the cumulative synthesis algorithm includes: in performing a two-dimensional image fitting task, it is necessary to perform numerical processing on all two-dimensional wavelet primitives in the plane to calculate the image pixel value. The image pixel value refers to the pixel color value of the image. The color value of each pixel in the image can be obtained by accumulating and weighting the color values ​​of multiple wavelet primitives covering the pixel, and the calculation formula is: , Where N represents the total number of wavelet primitives currently covering the pixel, represents the color value of the wavelet primitive, represents the opacity value of the wavelet primitive, Represents the current image pixel value.

[0047] In the embodiment of the present application, the alpha synthesis means that in the five-dimensional neural radiation field reconstruction, it is necessary to perform numerical processing on all three-dimensional wavelet primitives in the space to render a reconstructed image with multi-view consistency. For example, a single pixel of an image at a certain viewing angle may be composed of multiple wavelet primitives covering the pixel: ; In the embodiment of the present application, the deformation field refers to the prediction of various attribute values ​​(position vectors) of wavelet primitives by using the fully connected network MLP for a given time t in the six-dimensional dynamic neural radiation field reconstruction task. , frequency vector and the covariance matrix ), and then continue to use alpha synthesis to obtain the image pixel value at the current time t.

[0048] Step b2, calculate the loss of the obtained visual signal and the real visual signal, and perform the back propagation process using an explicit gradient calculation formula. In the embodiment of the present application, the back propagation adopts the Pytorch custom differential framework. Through the Pytorch custom differential framework, custom back propagation can be implemented, thereby improving the parameter optimization efficiency of the wavelet primitives.

[0049] The differentiable rasterizer and physical process guidance method of the present invention can accurately capture the frequency details of visual signals and realize more accurate display representation of visual signals with complex frequency distribution. Figure 1 The differentiable rasterizer shown shows that the wavelet primitives are mapped to the screen space. The calculation formula of the mapped wavelet primitives in the screen space is: , After simplification, we get: , in express dimensional space coordinate system to dimensional space coordinate system, represents the real number space, express A dimension that is mapped in a dimensional space coordinate system. According to the frequency response characteristics of the wavelet primitive, different modulation frequency vectors can enable the wavelet primitive to efficiently and accurately represent the high and low frequency characteristics of the visual signal. By adaptively optimizing the modulation frequency vector and other parameter vectors during training, the differentiable wavelet primitive can effectively capture the time-frequency characteristics of the visual signal, thereby more accurately and efficiently representing the visual signal with complex frequency.

[0050] An embodiment of the present invention also provides a method for representing visual signals using microwavelet primitives, comprising: using microwavelet primitives to explicitly represent visual signals, first, initializing and parameterizing the wavelet primitives, then, for the wavelet primitives, using a differentiable rasterization module to map the wavelet primitives to the screen space, and finally, using the wavelet primitives in the screen space to execute various physical processes, and adaptively optimizing and updating various parameters of the wavelet primitives during the training process.

[0051] The differentiable rasterization module uses a fast rasterization method to divide the screen into The grid, Represents the number of pixels occupied by the grid in the screen coordinate system. The following formula is used to calculate the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid is: , , , , Where C represents the set of all grids intersecting with the wavelet primitives, Indicates the screen space Line A grid of columns, Represents the area of ​​the rectangle where the wavelet primitive intersects the grid; , They represent the length and width of the rectangle where the wavelet primitive and the grid intersect, r represents the side length of the equilateral rectangle circumscribing the wavelet primitive, They respectively represent the horizontal and vertical coordinates of the wavelet primitive in the screen coordinate system.

[0052] The device and method described in the present invention can be applied to various visual signal representation tasks. This embodiment is recorded as wavelet visual primitives. The benchmark methods compared in the two-dimensional image fitting task are: Gaussian visual primitives, wavelet implicit neural representation, the benchmark methods compared in the five-dimensional neural radiation field reconstruction task are: Gaussian visual primitives, unbounded anti-aliasing neural radiation field, full-light voxel radiation field, hash neural primitives, and the benchmark methods compared in the six-dimensional dynamic neural radiation field reconstruction task are: Gaussian visual primitives, dynamic implicit neural representation, dynamic mirror implicit neural representation. The specific implementation details vary depending on the task, so the specific implementation of different tasks will be described below, including two-dimensional image representation, five-dimensional neural radiation field reconstruction, and six-dimensional dynamic neural radiation field reconstruction.

[0053] For the two-dimensional image fitting task, all comparison methods use default parameter settings, and the wavelet vision primitive uses two-dimensional wavelet primitives, guided by the cumulative synthesis algorithm. The number of wavelet vision primitives is the same as the Gaussian vision primitive training setting, that is, the primitive parameters are randomly initialized, the number is set to 70k, and the training iterations are 50,000. Figure 2 The results of the two-dimensional image fitting are compared ( Figure 2 The privacy information involved is blurred in the image), where the true value image is a resolution of Among the three methods of wavelet visual primitives (i.e., the method of the present invention), Gaussian visual primitives, and wavelet implicit neural representation, the method of the present invention achieves clearer and more accurate results, such as the tiles and sculptures of the temple and the windows and street lights of the church that are magnified, which are over-smoothed in the results of the Gaussian visual primitives method and the wavelet implicit neural representation method.

[0054] For the five-dimensional static neural radiation field reconstruction task, all comparison methods use default parameter settings, and the wavelet visual primitive uses a three-dimensional wavelet primitive, guided by the alpha synthesis algorithm. The wavelet visual primitive uses the same densification method and training parameter vector settings as the Gaussian visual primitive, that is, the primitive is initialized by the motion recovery structure algorithm, the gradient densification threshold is set to 0.0002, the number of densification start iterations is 500, the number of densification end iterations is 15k, the densification iteration interval is 100, the number of training iterations is 20k, and the loss function uses a mixed loss of L1-Loss and SSIM for supervision. Training and testing are performed on the Mip-NeRF360, Tanks & Temples, and DeepBlending datasets. It should be noted that the modulation frequency vector learning rate of the wavelet visual primitive is set to 0.0025. Figure 3 The results of indoor and outdoor scenes in the dataset are compared ( Figure 3The method of the present invention demonstrates advantages in representing scenes with complex frequency components, such as the border of a portrait and a plant in a pot in an indoor scene, a truck windshield and a transmission tower in the distance of a train in an outdoor scene. The highlight reflections or complex textures in these areas are over-smoothed or fail to be reconstructed in the reconstruction of the four benchmark methods of Gaussian visual primitives, unbounded anti-aliased neural radiation fields, full-light voxel radiation fields and hash neural primitives. Only the wavelet visual primitives used in the present invention can reconstruct clear details in these areas.

[0055] For the six-dimensional dynamic neural radiation field reconstruction task, all comparison methods use default parameter settings. The wavelet visual primitive uses a three-dimensional wavelet primitive, guided by the deformation field algorithm. The deformation field uses a neural network with 8 hidden layers and 256 neurons in each layer to model the matching function between the parameter vector of the wavelet visual primitive and time t. The input of the network is the parameter vector of the wavelet primitive The output is the offset of the parameter vector with time t The wavelet vision primitive uses the same densification method and training parameter vector setting as the Gaussian vision primitive, that is, the primitive is initialized by the motion recovery structure algorithm, the number of pre-training iterations is 3k, the total number of training iterations is 20k, the gradient densification threshold is set to 0.0002, the number of densification start iterations is 500, the number of densification end iterations is 15k, the densification iteration interval is 100, the number of training iterations is 20k, and the loss function uses a mixed loss of L1-Loss and SSIM for supervision. Training and testing are performed on the virtual synthetic dataset D-NeRF and the real scene dataset NeRF-DS dataset. It should be noted that the modulation frequency vector learning rate of the wavelet vision primitive is set to 0.0025 here. Figure 4 The results are compared ( Figure 4 The privacy information involved is blurred in the image). The method of the present invention demonstrates the advantage of simultaneously representing high-frequency and low-frequency components in the scene, for example, the sleeves of the virtual person when jumping, the rolled-up pants when squatting and the tail of the dinosaur in the virtual synthetic data set, and the curtains and corners in the real data set. The two benchmark methods of Gaussian visual primitives and dynamic mirror implicit neural representation cannot simultaneously accurately reconstruct low-frequency and high-frequency components. The present invention can simultaneously capture and represent rich frequency components in dynamic scenes.

[0056] The present invention provides a visual signal representation device and method that can be used to represent microwavelet primitives. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A visual signal representation device capable of representing microwavelet primitives, characterized in that: It includes wavelet primitive generation module, differentiable rasterization module and physical process guidance module; The wavelet primitive generation module is used to initialize and parameterize the wavelet primitive; The differentiable rasterization module is used to rasterize the wavelet primitives, and the rasterization includes: mapping the wavelet primitives into pixel representations, and performing forward rendering and reverse gradient propagation; the input of the differentiable rasterization module is the wavelet primitives, and the output is the wavelet primitives in the screen space, and a depth sorting table of all grids in the screen space and the corresponding wavelet primitives; The physical process guiding module performs numerical processing of a preset physical process on the rasterized wavelet primitives to adaptively represent multi-dimensional visual signals; The input of the physical process guiding module is a depth sorting table of grids and corresponding wavelet primitives and wavelet primitives in screen space, and the output is a multi-dimensional visual signal value.

2. The device according to claim 1, characterized in that The wavelet primitive initialization method in the wavelet primitive generation module is random initialization or initialization by motion recovery structure algorithm. The parameterized expression formula of the wavelet primitive is: ， ， ， in, express The wavelet function expression in dimensional space, represents the transpose operator, express The position vector of a point in the dimensional space coordinate system, express The position vector of the wavelet primitive in the dimensional space coordinate system, express The modulation frequency vector of the wavelet primitive in the dimensional space coordinate system, express Gaussian distribution of the dimensional space coordinate system, express The size of the coordinate system in the dimensional space is The covariance matrix of and Respectively The size of the coordinate system in the dimensional space is The rotation matrix and scaling matrix of , e represents a natural constant.

3. The device according to claim 1, characterized in that The differentiable rasterization module uses general wavelet sputtering for rasterization, and the expression of the general wavelet sputtering is: ， After simplification, we get: ， in express dimensional space coordinate system to dimensional space coordinate system, Respectively The position vector of the point in the dimensional space coordinate system, the position vector of the wavelet primitive, the modulation frequency vector of the wavelet primitive, and the magnitude of The covariance matrix of represents the real number space, express A dimension to be mapped in a dimensional space coordinate system.

4. The device according to claim 3, characterized in that The differentiable rasterization module uses a fast rasterization method to perform a forward rendering process, including: dividing the screen into The grid, Represents the number of pixels occupied by the grid in the screen coordinate system, and calculates the equilateral rectangle of the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid; For each wavelet primitive, check whether the bounding rectangle of the wavelet primitive is consistent with the grid. The boundary areas of the two sets intersect if the area of ​​the intersecting area is If it is greater than 0, the grid is judged Intersect with the wavelet primitives; will satisfy The grid is added to set C to obtain the sorting table of wavelet primitives and grids; then the sorting table of wavelet primitives and grids is re-sorted.

5. The device according to claim 4, characterized in that The equilateral rectangle of the wavelet primitive in the middle of the screen and all The intersection area of ​​the grid is: ， ， ， ， Where C represents the set of all grids intersecting with the wavelet primitives, Indicates the screen space Line A grid of columns, Represents the area of ​​the rectangle where the wavelet primitive intersects the grid; ni represents the multiplication of n and i, and nj represents the multiplication of n and j. , They represent the length and width of the rectangle where the wavelet primitive and the grid intersect, r represents the side length of the equilateral rectangle circumscribing the wavelet primitive, They respectively represent the horizontal and vertical coordinates of the wavelet primitive in the screen coordinate system.

6. The device according to claim 4, characterized in that The reordering of the sorting table of wavelet primitives and grids specifically includes: the differentiable rasterization module first uses 32-bit numbers to encode the depth value of the wavelet primitive to obtain the serial number value wavelet_idx of the wavelet primitive; uses 32-bit numbers to encode all grids in the screen space to obtain the serial number tile_idx of the wavelet primitive and the intersecting grid; combines the serial number value wavelet_idx with the serial number tile_idx to obtain a 64-bit code in the form of (wavelet_idx, tile_idx), rearranges the 64-bit code, exchanges the coding serial numbers of the wavelet primitive and the grid, and obtains a 64-bit code in the form of (tile_idx, wavelet_idx); for different wavelet primitives in the grid with the same serial number, sorts them from small to large according to the last 32-bit coding value to obtain a depth sorting table of each grid and the corresponding wavelet primitive; and uses the depth sorting table of the grid and the corresponding wavelet primitive and the wavelet primitive in the screen space as inputs of the physical process guidance module.

7. A method for representing visual signals in microwavelet primitives implemented by the device according to any one of claims 1 to 6, characterized in that: include: Use minimizable wavelet primitives to explicitly represent visual signals, initialize and parameterize the wavelet primitives; For the wavelet primitive, a differentiable rasterization module is used to map the wavelet primitive to a screen space; A physical process guided module is used to perform numerical processing of preset physical processes on wavelet primitives in screen space to achieve adaptive multi-dimensional visual signal representation.

8. The method according to claim 7, characterized in that The visual signal representation method using microwavelet primitives is applied to two-dimensional image fitting tasks. For two-dimensional image fitting, the wavelet primitive is a two-dimensional wavelet primitive, the numerical processing method of the preset physical process is cumulative synthesis, and the output is an RGB pixel value.

9. The method according to claim 8, characterized in that The visual signal representation method of the micro-wavelet primitives is applied to the five-dimensional neural radiation field reconstruction task. For the reconstruction of the five-dimensional neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is alpha synthesis, and the output is the RGB pixel value of the image in a given observation direction.

10. The method according to claim 8, characterized in that The visual signal representation method using microwavelet primitives is applied to the task of reconstructing a six-dimensional dynamic neural radiation field. For the reconstruction of the six-dimensional dynamic neural radiation field, the wavelet primitive is a three-dimensional wavelet primitive, the numerical processing method of the preset physical process is a deformation field, and the output is the RGB pixel value of the image in a given observation direction at time t.

Images