Neural network hologram coding capability test method, device, equipment, medium and product
By creating test sets of noisy images and binary images with different frequencies, the holographic encoding capability of neural networks is evaluated. This solves the problem of the difficulty in achieving both reproduction quality and generation speed in existing technologies, and realizes comprehensive testing and evaluation of the information processing capability of neural networks in different frequency bands under extreme conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ARMOR ACADEMY OF CHINESE PEOPLES LIBERATION ARMY
- Filing Date
- 2025-02-08
- Publication Date
- 2026-05-15
AI Technical Summary
Existing neural network hologram encoding methods struggle to balance reproduction quality and generation speed, and lack effective testing methods to evaluate the encoding capabilities of neural networks.
By creating noise images and binary image test sets corresponding to low-frequency, mid-frequency, and high-frequency spectra, and adding them to the 3D scene model for rendering, the spectrum test set and binary image test set are obtained. These test sets are used to evaluate the holographic encoding capability of the neural network, and quantitative evaluation is performed using indicators such as PSNR and SSIM.
This study enables a comprehensive test of the holographic encoding capabilities of neural networks at different frequencies, providing a better understanding of the network's ability to process information in different frequency bands, offering targeted directions for model optimization, and evaluating the model's robustness under extreme conditions.
Smart Images
Figure CN122053809A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of three-dimensional imaging, and in particular to a method, apparatus, device, medium, and product for testing the coding capability of neural network holograms. Background Technology
[0002] Computer-generated holography (CGH) combines computer technology with optical holography to achieve functions that are difficult or impossible for optical holography, such as controlling the size of reconstructed objects or reconstructing objects that do not exist in reality. The primary device replacing the photosensitive material in optical holography is the spatial light modulator (SLM). However, current SLMs are mainly phase-type, capable of modulating only the phase of incident light, not its amplitude. Therefore, generating corresponding pure phase information from diffraction field information has become a major challenge in computational holography.
[0003] Traditional holography generation methods include iterative algorithms such as Gerchberg-Saxton (GS) and Wirtinger Holography (WH), one-step encoding algorithms such as Double-phase Holography (DPH) and Stochastic Gradient Descent (SGD). However, traditional methods often present a trade-off between reproduction quality and generation speed. The DPH algorithm offers the fastest generation speed, but its reproduction quality is often low. The WH algorithm, on the other hand, requires a long iteration time to generate holograms with high reproduction quality. In recent years, researchers have combined emerging neural networks with computational holography, achieving promising results by fitting the relationship between the diffraction field and the hologram through a massive number of nodes in the network. However, the ability of the neural network to encode holograms and whether the generated holograms meet requirements is crucial; therefore, testing the hologram encoding capabilities of neural networks is essential. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, device, medium, and product for testing the coding capability of neural networks for holograms, which can test the ability of neural networks to code holograms.
[0005] To achieve the above objectives, this application provides the following solution:
[0006] Firstly, this application provides a method for testing the coding capability of neural network holograms, including:
[0007] Create noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra.
[0008] Noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra are used as materials and added to a 3D scene model. The 3D scene model with added materials is then rendered to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set. The low-frequency spectrum test set includes intensity and depth images corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes intensity and depth images corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes intensity and depth images corresponding to the high-frequency spectrum.
[0009] Obtain a binary image test set; the binary image test set includes binary images of each test pattern in a preset test pattern set and depth images of each test pattern in the preset test pattern set; the binary images are obtained under extreme conditions;
[0010] The holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set.
[0011] Optionally, the creation of noise images corresponding to low-frequency spectra, mid-frequency spectra, and high-frequency spectra specifically includes:
[0012] According to formula H noise =2m to calculate the height of the noisy image in pixels;
[0013] According to formula W noise =H noise ×H÷W calculates the width in pixels of the noisy image;
[0014] The noise image is determined based on the number of pixels in height and the number of pixels in width of the noise image;
[0015] Where m is a real number with no practical meaning, H noise W represents the height of a noisy image in pixels. noise H represents the width in pixels of the noise image, H represents the preset length of the noise image, and W represents the preset width of the noise image. The m values corresponding to the low-frequency spectrum, the high-frequency spectrum, and the mid-frequency spectrum are all different.
[0016] Optionally, the process of constructing the three-dimensional scene model specifically includes:
[0017] Randomly generate the positions of objects in a 3D scene;
[0018] Randomly generate depth values for each object in a 3D scene;
[0019] A 3D scene model is obtained by geometrically modeling the position and depth values of each object in the 3D scene.
[0020] Optionally, a binary image test set is obtained, specifically including:
[0021] Under extreme conditions, obtain the binary images of each test pattern in the preset test pattern set;
[0022] Draw depth images of each test pattern in the preset test pattern set.
[0023] Optionally, the holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set, specifically including:
[0024] If the test requirement is to test the target neural network's ability to encode holograms corresponding to low-frequency spectra, then the low-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the low-frequency spectra.
[0025] The reconstructed 3D scene corresponding to the low-frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the low-frequency spectrum.
[0026] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum;
[0027] By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum with the intensity images in the low-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the low-frequency spectrum are obtained.
[0028] If the test requirement is to test the target neural network's ability to encode holograms corresponding to the intermediate frequency spectrum, then the intermediate frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the intermediate frequency spectrum;
[0029] The reconstructed three-dimensional scene corresponding to the intermediate frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the intermediate frequency spectrum.
[0030] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the intermediate frequency spectrum;
[0031] By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the mid-frequency spectrum with the intensity images in the mid-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the mid-frequency spectrum are obtained.
[0032] If the test requirement is to test the target neural network's ability to encode holograms corresponding to high-frequency spectra, then the high-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the high-frequency spectra.
[0033] The reconstructed 3D scene corresponding to the high frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the high frequency spectrum.
[0034] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the high-frequency spectrum;
[0035] By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the high frequency spectrum with the intensity images in the high frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the high frequency spectrum are obtained.
[0036] For any test pattern in the binary image test set, input the binary image and depth image of the test pattern into the target neural network to obtain the hologram corresponding to the test pattern;
[0037] The reconstructed 3D scene corresponding to the test pattern is obtained by reconstructing and reproducing the hologram corresponding to the test pattern.
[0038] Obtain the superimposed image of the reconstructed 3D scene corresponding to the test pattern;
[0039] By comparing the superimposed images of the reconstructed 3D scenes corresponding to each test pattern with the binary images of each test pattern, the test results of the target neural network's holographic encoding capability under extreme conditions are obtained.
[0040] Optionally, depth values for each object in the 3D scene are randomly generated, specifically including:
[0041] For any given object, randomly generate a floating-point number r;
[0042] Substitute the floating-point number r into the formula The depth value P(r) of the object is obtained.
[0043] Secondly, this application provides a neural network hologram coding capability testing device, comprising:
[0044] The noise image creation module is used to create noise images corresponding to low-frequency spectra, mid-frequency spectra, and high-frequency spectra.
[0045] The rendering module is used to add noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra as materials to a 3D scene model, and then render the 3D scene model with added materials to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set. The low-frequency spectrum test set includes an intensity image and a depth image corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes an intensity image and a depth image corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes an intensity image and a depth image corresponding to the high-frequency spectrum.
[0046] The binary image test set acquisition module is used to acquire a binary image test set; the binary image test set includes binary images of each test pattern in a preset test pattern set and depth images of each test pattern in the preset test pattern set; the binary images are acquired under extreme conditions;
[0047] The testing module is used to test the holographic encoding capability of the target neural network based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set.
[0048] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the neural network hologram encoding capability testing method described in any one of the above.
[0049] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the neural network hologram encoding capability testing method described above.
[0050] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the neural network hologram encoding capability testing method described above.
[0051] According to the specific embodiments provided in this application, this application has the following technical effects:
[0052] This application provides a method, apparatus, device, medium, and product for testing the holographic encoding capability of a neural network. The spectrum test set of this application includes low-frequency, mid-frequency, and high-frequency images to comprehensively test the network's performance capability at different frequencies. This allows for a better understanding of the neural network's ability to process information in different frequency bands during holographic encoding, thereby providing more targeted directions for model optimization. The binary image test set used includes binary images acquired under extreme conditions, which can evaluate the model's encoding capability under extreme conditions. In summary, this application can test the ability of a neural network to encode holograms. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1A flowchart of a neural network hologram encoding capability testing method provided in an embodiment of this application;
[0055] Figure 2 Noise images corresponding to different spectra provided in this application;
[0056] Figure 3 A schematic diagram of the 3D scene model with added materials provided in this application;
[0057] Figure 4 A schematic diagram of the spectrum test set provided in this application;
[0058] Figure 5 Intensity and depth images corresponding to different spectra provided in this application;
[0059] Figure 6 The depth distribution histogram of the depth image and the intensity distribution histogram of the intensity image are constructed using the method provided in this application;
[0060] Figure 7 This application provides a subset of binary images and depth images from the binary image test set.
[0061] Figure 8 This is a schematic diagram of a neural network hologram encoding capability testing method provided in an embodiment of this application;
[0062] Figure 9 A comparison chart of test results provided for one embodiment of this application;
[0063] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0065] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0066] In one exemplary embodiment, such as Figure 1 As shown, a method for testing the encoding capability of neural network holograms is provided, including the following steps:
[0067] Step 201: Create noise images corresponding to the low-frequency spectrum, the mid-frequency spectrum, and the high-frequency spectrum.
[0068] Step 202: Add the noise images corresponding to the low-frequency spectrum, mid-frequency spectrum, and high-frequency spectrum as materials to the 3D scene model, and render the 3D scene model with added materials to obtain the low-frequency spectrum test set, mid-frequency spectrum test set, and high-frequency spectrum test set. The low-frequency spectrum test set includes: intensity images and depth images corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes intensity images and depth images corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes intensity images and depth images corresponding to the high-frequency spectrum.
[0069] Step 203: Obtain a binary image test set; the binary image test set includes the binary image of each test pattern in the preset test pattern set and the depth image of each test pattern in the preset test pattern set. For example... Figure 7 As shown, Figure 7 Part (b) is Figure 7 The depth image of part (a) Figure 7 The middle (d) part is Figure 7 The depth image of part (c) Figure 7 The middle (f) part is Figure 7 The depth image of the middle (e) section. Figure 7 The middle (h) part is Figure 7 The depth image of part (g); the intensity distribution of the color blocks in the binary image does not conform to the distribution pattern of the real scene, and the distribution and changes of the color blocks are more extreme, and it was obtained under extreme conditions. The extreme conditions are: some extreme scenes or specific visual requirements. The extreme scenes are high-contrast scenes, and the specific visual requirements are that areas without content are black or transparent, and areas with content are high-contrast color blocks.
[0070] Step 204: Test the holographic encoding capability of the target neural network using the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set. If the test results are unsatisfactory, the training process can be tracked to improve or retrain the target neural network.
[0071] By performing steps 201 to 204 above, the holographic encoding capability of the neural network can be tested.
[0072] In an exemplary embodiment, the creation of noise images corresponding to low-frequency spectra, mid-frequency spectra, and high-frequency spectra specifically includes:
[0073] According to formula H noise =2 mCalculate the height in pixels of the noisy image.
[0074] According to formula W noise =H noise ×H÷W calculates the width in pixels of the noisy image.
[0075] The noisy image is determined based on the number of pixels in its height and the number of pixels in its width, such as... Figure 2 As shown, Figure 2 Parts (a), (b), and (c) represent noise images for m=2, m=6, and m=10, respectively. For example, H... noise =2 2 =4. To obtain a noise texture image of size 3840×2160, we need to get W by multiplying 4×3840÷2160. noise The noise image is then directly magnified to a specified height, i.e., 3840×2160, using nearest neighbor interpolation.
[0076] Where m is a real number with no practical meaning, H noise W represents the height of a noisy image in pixels. noise This represents the width in pixels of the noise image. H represents the preset length of the noise image, and W represents the preset width of the noise image. The values of m for low-frequency, high-frequency, and mid-frequency frequencies are all different. It is recommended to set m to 1-2 for low-frequency noise textures, 5-6 for mid-frequency noise textures, and 9-10 for high-frequency noise textures. For example... Figure 6 As shown, the pixel distribution statistics of the three datasets (NYU, MIT-CHG-4K, and GM-4K) across 10 depth layers can be obtained through depth distribution histograms. Figure 6 As shown in part (a), the pixel depth distribution statistics are more uniform. The intensity image spectral distribution of the four datasets (GM-4K(WF), GM-4K(MF), GM-4K(LF), GM-4K(HF)) can be obtained through the intensity distribution histogram. Figure 6 As shown in part (b), after performing a Fourier transform, the intensity value is obtained according to the radius. It can be found that by controlling the scale of the noise texture, different ratios of high-frequency and low-frequency components in the dataset can be achieved.
[0077] In an exemplary embodiment, before adding the noise images corresponding to the low-frequency spectrum, the mid-frequency spectrum, and the high-frequency spectrum as materials to the 3D scene model, the method further includes:
[0078] Construct a 3D scene model.
[0079] In an exemplary embodiment, the process of constructing the 3D scene model specifically includes:
[0080] Randomly generate the positions of objects in a 3D scene.
[0081] Randomly generate depth values for each object in a 3D scene.
[0082] A 3D scene model is obtained by geometrically modeling the position and depth values of each object in the 3D scene.
[0083] In an exemplary embodiment, let a series of variables z0, z1, ..., z N Let z represent the depth plane within the camera's line of sight, z0 represent the depth plane closest to the orthogonal camera, and N represent the total number of depth layers. When a continuous geometric scene is transformed into a multi-depth scene, the depth plane located at (z0, z0, z0) is the depth plane within the camera's line of sight. n-1 , z n The geometric pixels of the depth plane are projected and compressed into the (n-1)th depth plane z. n-1 To achieve uniform depth distribution, theoretically, geometric projection pixels are needed for each depth. Statistically, they have equal proportions, that is:
[0084]
[0085] Since the geometry at different locations may be occluded, it is necessary to calculate the probability of passing through the corresponding occlusion, which is distributed in z0, z1, ..., z N-1 The probability of the geometry representing depth can be expressed as a harmonic series. Further calculation yields the probability density function P(n) (PDF), used to determine the depth position (z-value) of objects in the scene. This function is defined as:
[0086]
[0087] Where P(n) is the probability density of the proportion of pixels in geometry at different depth layers in a multi-depth scene, representing the proportion of projected pixels in the nth depth plane to the total number of pixels, and N is the required number of depth layers. It is a constant after N is determined.
[0088] Furthermore, through cumulative integration, the cumulative distribution function C(n) (CDF) can be expressed as:
[0089]
[0090] Where i is a real number with no practical meaning, and since the CDF function is a discrete function, the inverse function C of the CDF can be obtained by fitting values. -1The function (r) can be used to obtain the random geometric depth distribution that conforms to the P(x) distribution by taking a random floating-point number r between 0 and 1 as input. The fitted function is expressed as follows:
[0091]
[0092] Thus, by controlling the size and number of models, a uniform depth distribution across the 3D scene models can be achieved. Therefore, in an exemplary embodiment, randomly generating the depth values of each object in the 3D scene specifically includes:
[0093] For any object, randomly generate a floating-point number r.
[0094] Substituting the floating-point number r into formula (4) yields the depth value P(r) of the object.
[0095] In practical applications, cube, triangular pyramid, and ellipsoid models are added at random positions and random rotation angles. The specific depth position z of each model is calculated by formula (4), and the x and y positions are randomly obtained. The random range is the rendering range of the camera. The range used in this embodiment is: x∈[-1.92, 1.92], y∈[-1.08, 1.08], which is selected according to the actual situation. The scaling ratio of the model can also be set according to actual needs. The scaling sizes of the cube, tetrahedron, and sphere used in this embodiment are 1.61199, 3, and 1, respectively. This ratio can make the volume of the three basically the same, better ensuring the uniformity of the depth distribution. When assigning values, it is recommended that the model size should not be too large, exceeding half of the rendering size, nor too small, which would lead to the need for too many models and reduce rendering efficiency.
[0096] In an exemplary embodiment, noise images corresponding to the low-frequency spectrum, the mid-frequency spectrum, and the high-frequency spectrum are used as materials and added to a 3D scene model, respectively. The 3D scene model with added materials is then rendered to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set, specifically including:
[0097] In Blender, create a new image object and assign the generated noise image to it. This creates a new material object, which is then connected to a diffuse BSDF node via node editing, and finally linked to a material output node. This material can then be applied to the 3D scene model.
[0098] The computer enters the main rendering loop, and in each loop executes the following steps:
[0099] Remove unused objects and data blocks. At the beginning of each loop, the computer removes all unused objects in the Blender scene, including data blocks such as materials and textures, to ensure that the resources in the scene are up-to-date and valid.
[0100] Add the previously built 3D scene model to Blender.
[0101] Apply textures. Apply the previously created noise materials to the model according to the needs of low-frequency, mid-frequency, and high-frequency testing, such as... Figure 3 As shown.
[0102] Set up a daylight source. Add a daylight source to the scene and randomly set its energy value. The energy value used in this application is randomly obtained between [5, 20]. Too small or too large an energy value will result in more low-frequency components and reduce the test capability.
[0103] Add a camera. The computer adds a camera object to the scene and sets it to an orthographic camera with an orthographic ratio of 3.84, which is the width of the rendered image. At a 4K scale, this results in a height of 2.16. The camera's position is set to (0, 0, 1) by default to ensure it is not occluded or clipped by the model.
[0104] Set the rendering parameters and execute the rendering. Finally, the computer sets the rendering resolution to 3840*2160 and uses nodes to render the image. The image is output in RGB mode.
[0105] Rendering and saving (using the compositing node). After rendering is complete, the computer saves the image using the compositing node settings. The specific steps are as follows:
[0106] 1. Set up the rendering layer node:
[0107] The rendering layer node receives the rendered image data and uses it as input for subsequent nodes.
[0108] 2. Set the mapping range node:
[0109] The Mapping Range Node receives fog field data and maps its brightness values to adjust the image's contrast and brightness. The minimum value is set to 0 and the maximum value to 10, while the maximum value is set to 255.
[0110] 3. Set the file output node:
[0111] Two file output nodes are set up to store different data, namely the adjusted image (intensity image) and the depth image, such as... Figure 4 and Figure 5 As shown, where Figure 5Part (a) represents the intensity image corresponding to the low-frequency spectrum, part (b) represents the depth image corresponding to the low-frequency spectrum, part (c) represents the intensity image corresponding to the mid-frequency spectrum, part (d) represents the depth image corresponding to the mid-frequency spectrum, part (e) represents the intensity image corresponding to the high-frequency spectrum, and part (f) represents the depth image corresponding to the high-frequency spectrum.
[0112] In one exemplary embodiment, obtaining a binary image test set specifically includes:
[0113] Under extreme conditions, obtain binary images of each test pattern in the preset test pattern set; obtain binary images of some test patterns through manual collection.
[0114] Draw depth images of each test pattern in the preset test pattern set.
[0115] In an exemplary embodiment, the holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set, and prior to this, the following steps are included:
[0116] Performing depth transformation and preprocessing (such as normalization and data augmentation) on the images in the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set helps improve the stability and generalization ability of the model. Figure 8 As shown in section (a).
[0117] In an exemplary embodiment, the holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set, specifically including:
[0118] If the test requirement is to test the target neural network's ability to encode holograms corresponding to low-frequency spectra, then the low-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the low-frequency spectra, such as... Figure 8 As shown in section (b), the target neural network encodes these input data into holograms based on the trained network architecture (such as U-Net++). This encoding process essentially converts information from a three-dimensional scene into a two-dimensional hologram, which can be reconstructed and reproduced under specific optical systems.
[0119] Reconstructing and reproducing a 3D scene from the hologram corresponding to the low-frequency spectrum yields a reconstructed 3D scene corresponding to the low-frequency spectrum, such as... Figure 8As shown in section (c). After encoding, the hologram is reconstructed using optical or numerical methods to generate the corresponding 3D reconstructed scene. During reconstruction, the depth information of the scene is obtained based on the information in the hologram, and this depth information corresponds to the depth map in the original test data. The depth pixels in the reconstructed scene can be extracted using a specific algorithm, and then compared with the input data.
[0120] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum. Take pictures of the reconstructed 3D scene at different focal points to obtain multiple intensity images. Then, extract the clear positions from each image and stitch them together to obtain the focal superimposed image of the reconstructed 3D scene. The focal superimposed image is a clear image with a large depth of field that is synthesized from multiple images with different focal points.
[0121] By comparing the superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum with the intensity images in the low-frequency spectrum test set, the test results of the target neural network's holographic encoding capability corresponding to the low-frequency spectrum are obtained. Figure 8 As shown in section (d), to quantitatively evaluate the encoding capability of the neural network, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were used as the main test metrics. These metrics were calculated by comparing the difference between the intensity image of the test set and the focus image of the reconstructed 3D scene. PSNR primarily measures the absolute error between the focus image of the reconstructed 3D scene and the original image, while SSIM focuses on the similarity of image structure and details.
[0122] If the test requirement is to test the target neural network's ability to encode holograms corresponding to the intermediate frequency spectrum, then the intermediate frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the intermediate frequency spectrum.
[0123] The reconstructed three-dimensional scene corresponding to the intermediate frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the intermediate frequency spectrum.
[0124] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the mid-frequency spectrum.
[0125] By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the mid-frequency spectrum with the intensity images in the mid-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the mid-frequency spectrum are obtained.
[0126] If the test requirement is to test the target neural network's ability to encode holograms corresponding to high-frequency spectra, then the high-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the high-frequency spectra.
[0127] The reconstructed 3D scene corresponding to the high frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the high frequency spectrum.
[0128] Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the high frequency spectrum.
[0129] By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the high-frequency spectrum with the intensity images in the high-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the high-frequency spectrum are obtained.
[0130] For any test pattern in the binary image test set, the binary image and depth image of the test pattern are input into the target neural network to obtain the hologram corresponding to the test pattern.
[0131] The reconstructed 3D scene corresponding to the test pattern is obtained by reconstructing and reproducing the hologram corresponding to the test pattern.
[0132] Obtain the superimposed image of the reconstructed 3D scene corresponding to the test pattern.
[0133] By comparing the superimposed images of the reconstructed 3D scenes corresponding to each test pattern with the binary images of each test pattern, the test results of the target neural network's holographic encoding capability under extreme conditions are obtained.
[0134] This application provides a new approach to the widespread application of real-time, high-quality holographic displays by constructing a test scheme with spectral and binary image test sets as important standards for evaluating the holographic encoding capabilities of neural networks. The low-frequency, mid-frequency, and high-frequency spectral test sets can evaluate the performance of neural networks when encoding different spectral information. Low frequencies contain relatively smooth, slowly changing images, mid-frequency frequencies contain more detailed textures, and high frequencies contain very complex, rapidly changing details. This design allows for comprehensive testing of the network's performance under different spectral distributions, providing a better understanding of the neural network's ability to process information in different frequency bands during holographic encoding, thus offering more targeted directions for model optimization.
[0135] The artificially designed binary image test set for special scenes contains artificially designed datasets of binary images and depth maps. It effectively simulates extreme scenarios or specific visual requirements. These images contain high-contrast patterns or special geometric structures, allowing for thorough testing of model performance under extreme conditions, such as encoding and reconstruction capabilities in high-contrast scenes. This test set is particularly suitable for evaluating the robustness of models when handling extreme images. A comprehensive testing scheme can provide an effective standard for evaluating the performance of future holographic coding neural networks, enabling the objective quantification of the performance of different models in various scenarios, thereby promoting technological progress in this field.
[0136] This application also provides an embodiment to demonstrate the effectiveness of the testing method provided in the above embodiment, specifically as follows:
[0137] 6 datasets ( Figure 9 The U-Net++ model trained on the training set (shown in the image) encodes 7 test set scenarios. Figure 9 The PSNR and SSIM values of the holographic numerical reconstruction performance (error bars using 95% confidence intervals) of the test set (each test set contains 20 pairs of 4K RGB-D images) are averaged across the 20 pairs of RGB-D images.
[0138] Figure 9 This presentation showcases the statistical results of PSNR and SSI M for numerical reconstruction of models trained on six datasets across seven test sets. Each dataset underwent 50 training epochs, with each epoch containing 500 pairs of 4K RGB-D images. Each test set contains 20 pairs of 4K RGB-D images. It can be seen that:
[0139] The MIT-CGH-4K and four GM-4K test sets exhibit strong homogeneity due to the need to control the randomness of batch rendering generation in order to manage scene distribution. This is reflected in the relatively concentrated confidence intervals in the tests. In contrast, the data based on real-world modeling scenes and binary image scenes requires more manual collection, resulting in greater data variability and irregular depth and intensity distributions, which manifests as larger confidence intervals in the tests.
[0140] Models trained with MIT-CGH-4K, GM-4K (wide-frequency), and GM-4K (mid-frequency) all performed well across various test sets, demonstrating strong generalization ability. Models trained with GM-4K (low-frequency) and GM-4K (high-frequency) excelled on their respective low-frequency and high-frequency test sets, but performed poorly on the high-frequency and low-frequency test sets of their counterparts. The model trained with NYU performed almost identically to other models on the low-frequency test set, but performed poorly on other test sets.
[0141] The binary pattern scenes test set consists of 20 pairs of target intensity images at different resolutions and artificially set depth images. The performance of models trained on each dataset fluctuates significantly on this test set, with substantial differences in numerical values. It can be considered that binary scenes readily reveal the encoding capabilities of different models. The test results show that the model trained with the proposed noisy texture GM-4K exhibits superior reconstruction performance. The experimentally reconstructed binary 3D scenes demonstrate the high generalization ability of this application. Furthermore, the signal intensity is uniform, and the background is noise-free, making it highly suitable for the design of head-up displays (HUDs) and diffractive optical elements (DOEs).
[0142] Based on the same inventive concept, this application also provides a neural network hologram coding capability testing device for implementing the aforementioned neural network hologram coding capability testing method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the neural network hologram coding capability testing device provided below can be found in the limitations of the neural network hologram coding capability testing method described above, and will not be repeated here.
[0143] In one exemplary embodiment, a neural network hologram encoding capability testing apparatus is provided, comprising:
[0144] The noise image creation module is used to create noise images corresponding to low-frequency spectra, mid-frequency spectra, and high-frequency spectra.
[0145] The rendering module is used to add noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra as materials to a 3D scene model, and then render the 3D scene model with added materials to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set. The low-frequency spectrum test set includes an intensity image and a depth image corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes an intensity image and a depth image corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes an intensity image and a depth image corresponding to the high-frequency spectrum.
[0146] The binary image test set acquisition module is used to acquire a binary image test set; the binary image test set includes binary images of each test pattern in the preset test pattern set and depth images of each test pattern in the preset test pattern set; the binary images are acquired under extreme conditions.
[0147] The testing module is used to test the holographic encoding capability of the target neural network based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set.
[0148] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores neural network hologram encoding capability test data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a neural network hologram encoding capability testing method.
[0149] Those skilled in the art will understand that Figure 10 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.
[0150] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.
[0151] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.
[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0154] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0156] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for testing the coding capability of neural network holograms, characterized in that, The method for testing the neural network hologram encoding capability includes: Create noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra. Noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra are used as materials and added to a 3D scene model. The 3D scene model with added materials is then rendered to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set. The low-frequency spectrum test set includes intensity and depth images corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes intensity and depth images corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes intensity and depth images corresponding to the high-frequency spectrum. Obtain a binary image test set; the binary image test set includes binary images of each test pattern in a preset test pattern set and depth images of each test pattern in the preset test pattern set; the binary images are obtained under extreme conditions; The holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set.
2. The method for testing the coding capability of neural network holograms according to claim 1, characterized in that, The creation of noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra specifically includes: According to formula H noise =2 m Calculate the height pixel count of the noisy image; According to formula W noise =H noise ×H÷W calculates the width in pixels of the noisy image; The noise image is determined based on the number of pixels in height and the number of pixels in width of the noise image; Where m is a real number with no practical meaning, H noise W represents the height of a noisy image in pixels. noise H represents the width in pixels of the noise image, H represents the preset length of the noise image, and W represents the preset width of the noise image. The m values corresponding to the low-frequency spectrum, the high-frequency spectrum, and the mid-frequency spectrum are all different.
3. The method for testing the coding capability of neural network holograms according to claim 1, characterized in that, The construction process of the 3D scene model specifically includes: Randomly generate the positions of objects in a 3D scene; Randomly generate depth values for each object in a 3D scene; A 3D scene model is obtained by geometrically modeling the position and depth values of each object in the 3D scene.
4. The method for testing the coding capability of neural network holograms according to claim 1, characterized in that, Obtain a binary image test set, specifically including: Under extreme conditions, obtain the binary images of each test pattern in the preset test pattern set; Draw depth images of each test pattern in the preset test pattern set.
5. The method for testing the coding capability of neural network holograms according to claim 1, characterized in that, The holographic encoding capability of the target neural network is tested based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set, specifically including: If the test requirement is to test the target neural network's ability to encode holograms corresponding to low-frequency spectra, then the low-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the low-frequency spectra. The reconstructed 3D scene corresponding to the low-frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the low-frequency spectrum. Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum; By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the low-frequency spectrum with the intensity images in the low-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the low-frequency spectrum are obtained. If the test requirement is to test the target neural network's ability to encode holograms corresponding to the intermediate frequency spectrum, then the intermediate frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the intermediate frequency spectrum; The reconstructed three-dimensional scene corresponding to the intermediate frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the intermediate frequency spectrum. Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the intermediate frequency spectrum; By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the mid-frequency spectrum with the intensity images in the mid-frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the mid-frequency spectrum are obtained. If the test requirement is to test the target neural network's ability to encode holograms corresponding to high-frequency spectra, then the high-frequency spectrum test set is input into the target neural network to obtain the holograms corresponding to the high-frequency spectra. The reconstructed 3D scene corresponding to the high frequency spectrum is obtained by reconstructing and reproducing the hologram corresponding to the high frequency spectrum. Obtain the focal superimposed image of the reconstructed 3D scene corresponding to the high-frequency spectrum; By comparing the focal superimposed image of the reconstructed 3D scene corresponding to the high frequency spectrum with the intensity images in the high frequency spectrum test set, the test results of the target neural network's ability to encode holograms corresponding to the high frequency spectrum are obtained. For any test pattern in the binary image test set, input the binary image and depth image of the test pattern into the target neural network to obtain the hologram corresponding to the test pattern; The reconstructed 3D scene corresponding to the test pattern is obtained by reconstructing and reproducing the hologram corresponding to the test pattern. Obtain the superimposed image of the reconstructed 3D scene corresponding to the test pattern; By comparing the superimposed images of the reconstructed 3D scenes corresponding to each test pattern with the binary images of each test pattern, the test results of the target neural network's holographic encoding capability under extreme conditions are obtained.
6. The method for testing the coding capability of neural network holograms according to claim 3, characterized in that, Randomly generate depth values for each object in a 3D scene, specifically including: For any given object, randomly generate a floating-point number r; Substitute the floating-point number r into the formula The depth value P(r) of the object is obtained.
7. A neural network hologram coding capability testing device, characterized in that, The neural network hologram coding capability testing device includes: The noise image creation module is used to create noise images corresponding to low-frequency spectra, mid-frequency spectra, and high-frequency spectra. The rendering module is used to add noise images corresponding to low-frequency, mid-frequency, and high-frequency spectra as materials to a 3D scene model, and then render the 3D scene model with added materials to obtain a low-frequency spectrum test set, a mid-frequency spectrum test set, and a high-frequency spectrum test set. The low-frequency spectrum test set includes an intensity image and a depth image corresponding to the low-frequency spectrum; the mid-frequency spectrum test set includes an intensity image and a depth image corresponding to the mid-frequency spectrum; and the high-frequency spectrum test set includes an intensity image and a depth image corresponding to the high-frequency spectrum. The binary image test set acquisition module is used to acquire a binary image test set; the binary image test set includes binary images of each test pattern in a preset test pattern set and depth images of each test pattern in the preset test pattern set; the binary images are acquired under extreme conditions; The testing module is used to test the holographic encoding capability of the target neural network based on the low-frequency spectrum test set, the mid-frequency spectrum test set, the high-frequency spectrum test set, and the binary image test set.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the neural network hologram coding capability testing method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the neural network hologram encoding capability testing method according to any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the neural network hologram encoding capability testing method according to any one of claims 1-6.