A holographic communication method based on rescaling

Abstract

The present invention discloses a holographic communication method based on rescaling, comprising the following steps: 1. constructing and training a complex convolutional neural network, and downsampling an input holographic image based on the trained complex convolutional neural network to obtain a reduced holographic image; 2. constructing a rate allocation model, performing rate allocation on the real and imaginary parts of the reduced holographic image, and then compressing and transmitting the reduced holographic image; and 3. decompressing the real and imaginary parts of the received rate-allocated holographic image, and upsampling the image based on the trained complex convolutional neural network to obtain a holographic image of the original input size. The present invention achieves rapid transmission of holographic images, reduces distortion of holographic images during transmission, has high computational efficiency, and can adapt to changes in network channel rates.

Description

Technical Field

[0001] The present invention belongs to the field of information communication technology, and in particular relates to a holographic communication method based on rescaling. Background Art

[0002] With the rapid development of industry and people's constant pursuit of a better life, traditional 2D images can no longer meet the needs of human production and life. Ordinary 2D images are obtained by recording the intensity distribution of diffusely reflected light from the subject on a "negative" through the principle of lens imaging. Ordinary images can only provide a certain angle of the scene; even wide-angle lenses cannot effectively and comprehensively represent the scene. At the same time, holograms have emerged as a type of digital image that is more expressive and richer in information than ordinary images.

[0003] A hologram is a recording of the interference pattern of light waves from an object. It boasts three-dimensional, full-viewing angles, and full color. It utilizes the principles of laser interference and diffraction to record the intensity distribution and phase information of diffusely reflected light from an object. The resulting hologram, based on this two-pronged information, offers a more detailed, richer, and more comprehensive representation of the object than a typical image. With the advancement of computers, computer-generated holograms (CGHs) have gradually become mainstream.

[0004] Compared to traditional planar images, the storage and transmission of holograms also require higher bandwidth and storage space. This is because computer-generated holograms need to record a large amount of information, resulting in large amounts of data and high storage and transmission costs. During the reconstruction process, computer-generated holograms require complex calculations and digital processing, which is computationally complex. The reconstructed image quality of the generated hologram is poor, with problems such as zero-order diffraction, speckle noise, and color aberration affecting the image quality. With technological development, people have increasingly higher requirements for the real-time transmission and reconstruction quality of holographic data. How to save the transmission cost of holograms while minimizing the impact on the reconstruction quality of holograms is currently a pressing issue. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a holographic communication method based on rescaling, which supports holographic images of various formats, realizes fast transmission of holographic images, and has high rate-distortion performance.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A holographic communication method based on rescaling, the method comprising the following steps:

[0008] Step 1: Build and train a complex convolutional neural network. Based on the trained complex convolutional neural network, downsample the input holographic image to obtain a reduced holographic image.

[0009] Step 2: Construct a bit rate allocation model. Based on the transmission network rate limit, perform unequal bit rate allocation on the real and imaginary parts of the reduced holographic image using the Lagrangian relaxation method. Compress the reduced holographic image according to the allocated bit rate and transmit it.

[0010] Step 3: Decompress the real and imaginary parts of the received holographic image after code rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the size of the original input.

[0011] Furthermore, the step 1 includes: the complex convolutional neural network includes a downsampling module, a differentiable compression algorithm module, and an upsampling module connected in sequence.

[0012] Furthermore, the weights of the complex convolutional neural network are updated using the gradient descent method until the loss value reaches convergence, and a trained complex convolutional neural network is obtained; wherein the loss function L is defined as:

[0013] ,

[0014] Where, Represents multi-scale structural similarity and Combination of losses; represents the loss function used to evaluate the fidelity of the high-resolution hologram after scaling restoration, It means that the angular spectrum method (ASM) is used to simulate the diffraction process, and the restored high-resolution hologram is reconstructed on the target plane. The mean square error loss calculated according to the attention mask is Represents the weight factor.

[0015] Furthermore, the downsampling module includes a first complex convolution layer, a second complex convolution layer, an inverse pixel rearrangement layer, a first complex residual layer, a second complex residual layer, a third complex residual layer, a third complex convolution layer, and a fourth complex convolution layer connected in sequence, wherein the number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3, respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the inverse pixel rearrangement layer performs a downsampling operation, and during downsampling, the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original; the number of convolution kernels in the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0016] Furthermore, the differentiable compression algorithm module includes a YUV conversion layer, a UV downsampling layer, a DCT transformation layer, a quantization layer, a DCT inverse transformation layer, a UV upsampling layer, and an RGB conversion layer connected in sequence, wherein the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT transformation layer performs discrete cosine transform on the three channels of Y, U, and V respectively, the quantization layer rounds the transformation result, the DCT transformation layer performs discrete cosine inverse transform on the three channels of Y, U, and V respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image.

[0017] Furthermore, the upsampling module includes a fifth complex convolution layer, a fourth complex residual layer, a fifth complex residual layer, a sixth complex residual layer, a sixth complex convolution layer, a seventh complex convolution layer, a pixel rearrangement layer, and an eighth complex convolution layer, which are sequentially connected. The number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, and 3, respectively. The convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the pixel rearrangement layer performs an upsampling operation, and during upsampling, the width and height of the holographic image become twice the original, and the number of channels is reduced to a quarter of the original; the number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0018] Furthermore, step 2 includes estimating the bit rate and distortion of the real image and imaginary image respectively using information entropy theory and Pasval's theorem under different compression qualities, and generating a bit rate table and a distortion table. According to the transmission network rate limit, the bit rate and holographic reconstruction amplitude distortion are balanced through the bit rate allocation model and the lookup algorithm to allocate the optimal bit rate to the real and imaginary images.

[0019] The beneficial effects of the present invention are:

[0020] The present invention constructs a complex convolutional neural network composed of an upsampling module, a differentiable compression algorithm module, and an upsampling module. It has good feature extraction performance for holographic images and considers the correlation between the real and imaginary parts during the feature enhancement process, overcoming the problem that existing rescaling methods have good effects on ordinary images but poor effects on holographic images.

[0021] The present invention can convert the holographic image, perform downsampling, compress it into a specified format and transmit it to the client according to the client request and the real-time network channel rate, thereby realizing the rapid transmission of the holographic image, reducing the distortion of the holographic image during transmission, having high computational efficiency, and being able to adapt to changes in the network channel rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a holographic communication method based on rescaling according to the present invention;

[0023] Figure 2 This is a network principle diagram of a holographic communication method based on rescaling according to the present invention;

[0024] Figure 3 This is the structure diagram of the complex convolutional neural network;

[0025] Figure 4 This is a schematic diagram of the bit rate allocation model;

[0026] Figure 5 A comparison diagram of the original image and the restored image transmitted based on the method of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings.

[0028] like Figure 1-Figure 2 As shown, the present invention is a holographic communication method based on rescaling. First, a complex convolutional neural network is constructed to downsample the holographic image on the server side to obtain a reduced holographic image. Then, according to the network transmission rate limit, the bit rate allocation module is used to allocate and compress the real and imaginary parts of the reduced holographic image. The real and imaginary parts of the reduced holographic image are compressed at different bit rates, and the results are transmitted to the client. Specifically, under different compression qualities, the information entropy theory and Pasval's theorem are used to estimate the bit rate and distortion of the real image and the imaginary image, and a bit rate table and a distortion table are generated. The bit rate allocation model and the table lookup algorithm are used to balance the bit rate and the holographic reconstruction amplitude distortion, and a better bit rate is allocated to the real and imaginary parts. The real and imaginary parts of the reduced holographic image are decompressed on the client, and the reduced holographic image is upsampled to obtain a holographic image of the original size. The structure of the upsampling module is similar to that of the downsampling module, but there are some significant differences. Specifically, the pixel arrangement layer replaces the inverse pixel arrangement layer in the upsampling stage, and the order of some layers and modules has been changed. This makes the downsampling module and the upsampling module have a symmetrical structure, which helps to restore the high-resolution hologram from the low-resolution hologram.

[0029] The present invention specifically comprises the following steps:

[0030] Step 1, construct a complex convolutional neural network;

[0031] The complex convolutional neural network of the present invention is composed of a downsampling module, a differentiable compression algorithm module, and an upsampling module connected in sequence, wherein the downsampling module is used to downsample the holographic image, referring to Figure 3In an embodiment of the present invention, there are eight layers connected sequentially, and their structure is: first complex convolution layer → second complex convolution layer → inverse pixel permutation layer → first complex residual layer → second complex residual layer → third complex residual layer → third complex convolution layer → fourth complex convolution layer.

[0032] The parameters of each layer are set as follows: the number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs convolution operation and Leaky ReLu activation operation; the inverse pixel rearrangement layer performs a downsampling operation, during which the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original; the number of convolution kernels in the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0033] This method can achieve the downsampling effect without convolution operation by utilizing the inverse pixel rearrangement layer, which reduces the number of parameters while speeding up the image compression. The key elements of the original holographic image can be extracted by using multiple complex convolution layers and complex residual layers for image restoration in the subsequent upsampling module.

[0034] The differentiable compression algorithm module is used to implement end-to-end compressed sensing training for complex convolutional neural networks. Figure 3 In the example of the present invention, there are seven layers connected in sequence, and the structure is as follows: YUV conversion layer → UV downsampling layer → DCT conversion layer → quantization layer → DCT inverse conversion layer → UV upsampling layer → RGB conversion layer;

[0035] Among them, the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT transformation layer performs discrete cosine transform on the Y, U, and V channels respectively, the quantization layer rounds the transformation results, the DCT transformation layer performs inverse discrete cosine transform on the Y, U, and V channels respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image.

[0036] This method simulates the differentiable form of the compression algorithm and connects the up and down sampling modules so that the complex neural network can form a whole and perform end-to-end training. Moreover, through the quantization loss in the compression process, the complex neural network can better restore the compressed image.

[0037] The upsampling module is used to restore the image after downsampling and compression distortion. In an embodiment of the present invention, there are 8 layers connected in sequence, and their structure is as follows: fifth complex convolution layer → fourth complex residual layer → fifth complex residual layer → sixth complex residual layer → sixth complex convolution layer → seventh complex convolution layer → pixel rearrangement layer → eighth complex convolution layer.

[0038] The parameters of each layer are set as follows: the number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, and 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs convolution operation and Leaky ReLu activation operation; the pixel rearrangement layer performs an upsampling operation, and during upsampling, the width and height of the holographic image are doubled, and the number of channels is reduced to one-quarter of the original; the number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0039] This method can achieve the upsampling effect without convolution operation by utilizing the pixel rearrangement layer, which reduces the number of parameters and speeds up the image recovery. It can generate high-frequency details in the original holographic image by utilizing multiple complex convolution layers and complex residual layers, thereby restoring the original holographic image.

[0040] Finally, the downsampling module, the differentiable compression algorithm module, and the upsampling module are connected in sequence to form the complex convolutional neural network in the present invention.

[0041] The input volume function and output volume function of each convolutional layer of the present invention are as follows:

[0042] ,

[0043] ,

[0044] Except for the first complex convolutional layer whose input is the original holographic image, each layer of the network will obtain the input feature map from the previous layer of the network, where 、 and is the height, width and number of output channels of the feature map output by the previous layer of network; except that the output of the eighth complex convolutional layer is the restored holographic image, each layer of network will output a feature map, where 、 and The height, width and number of output channels of the feature map output by this layer of network.

[0045] Furthermore, training the neural network involves:

[0046] Input the training set into the complex convolutional neural network, and use the gradient descent method to update the weights of the complex convolutional neural network until the loss value is reduced to convergence, and a trained complex convolutional neural network is obtained;

[0047] The present invention defines the loss function L as:

[0048] ,

[0049] in, Represents multi-scale structural similarity ( )and Combination of losses; Represents the loss function used to evaluate the fidelity of the restored high-resolution hologram, that is, the mean square error between the restored hologram and the original hologram; in order to achieve a good visual perception effect of the hologram on the target plane, use Losses guide this process, It means that the angular spectrum method (ASM) is used to simulate the diffraction process, the recovered high-resolution hologram is reconstructed on the target plane, and then the mean square error loss is calculated based on the attention mask, where Represents a weight factor used to balance different losses, which is set to 0.04 in this invention.

[0050] Step 2: Build a rate allocation model:

[0051] like Figure 4 As shown, the rate allocation model of the present invention is derived from the relationship between the amplitude image of the holographic image and the real image and imaginary image of the holographic image, wherein the rate allocation model is a non-uniform rate allocation implemented based on the Lagrangian relaxation method; rate allocation is a key aspect of data flow because it determines how many bits are allocated to different parts of the stream by complying with the aggregate rate constraint, and the rate-distortion prediction module of the present invention implements the prediction of rate and distortion based on the information entropy theorem and Parseval's theorem respectively.

[0052] In the hologram stream, since the reconstructed hologram amplitude will eventually be viewed by the user, its distortion reflects the visual quality of the reconstructed hologram. In the present invention, since the real image and imaginary image distortion have different effects on the reconstructed amplitude distortion, different bit rates need to be allocated to the real and imaginary data to transmit the hologram. In this embodiment, the transmission network rate limit is set to ,in and are the bit rates of the real and imaginary image data respectively.

[0053] Use a function To approximately reconstruct the hologram distortion Image distortion with real and imaginary parts and The relationship between: , the derivation process is as follows:

[0054] Assume that the real part is , the imaginary part is , and The errors caused by compression are and , the amplitude error of the reconstructed hologram introduced by compression is , we can get: .

[0055] After Taylor expansion, by retaining the first-order terms, we have:

[0056] .

[0057] The amplitude distortion of the reconstructed hologram is expressed as , so it is approximately:

[0058] .

[0059] in, yes and The average value of the multiplication is close to 0 and can be omitted. Therefore:

[0060] .

[0061] in, Represents the phase of the holographic image;

[0062] Since the errors in the original phase and the real and imaginary parts are approximately independent, the amplitude distortion of the reconstructed hologram becomes:

[0063] .

[0064] Arranging the above formulas, we can get:

[0065] .

[0066] This method approximates the complex distortion relationship into a linear relationship through the Taylor formula, so that it can be used as a submodule in the complex convolutional neural network, improving the speed and efficiency of bit rate allocation.

[0067] Step 3: Decompress the real and imaginary parts of the received holographic image after code rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the size of the original input.

[0068] Furthermore, this method is applied to transmit the target holographic image. The target holographic image to be transmitted is sequentially input into the trained complex convolutional neural network, and the holographic image is output after the transmission compression and other processes. The peak signal-to-noise ratio (PSNR) and structural similarity metric (SSIM) are used to judge the similarity between the output image and the original image, and the rate-distortion performance is used to judge its compression efficiency. Figure 5 As shown, the large image in the first row represents the original hologram, and three local holographic images are displayed for comparing image quality; the second row represents a local magnified image of the reconstructed image of the holographic image after transmission recovery; the third row represents a local magnified image of the reconstructed image of the original holographic image; comparing the various details in the local magnified image, such as letters and doll outlines, it can be seen that they are very close to the original image. The use of this method can well compress the holographic image, and has high rate-distortion performance. It can reduce the distortion of the image under limited bandwidth, and the restored image is consistent with the original image.

[0069] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A holographic communication method based on rescaling, characterized in that: The method comprises: Step 1: construct and train a complex convolutional neural network. Based on the trained complex convolutional neural network, downsample the input holographic image to obtain a reduced holographic image. The complex convolutional neural network includes a downsampling module, a differentiable compression algorithm module, and an upsampling module connected in sequence. The downsampling module includes a first complex convolution layer, a second complex convolution layer, an inverse pixel rearrangement layer, a first complex residual layer, a second complex residual layer, a third complex residual layer, a third complex convolution layer, and a fourth complex convolution layer, which are sequentially connected. The number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3, respectively. The convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation. The inverse pixel rearrangement layer performs a downsampling operation. During downsampling, the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original. The number of convolution kernels in the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure. The differentiable compression algorithm module includes a YUV conversion layer, a UV downsampling layer, a DCT conversion layer, a quantization layer, a DCT inverse conversion layer, a UV upsampling layer, and an RGB conversion layer connected in sequence, wherein the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT conversion layer performs discrete cosine transform on the three channels of Y, U, and V respectively, the quantization layer rounds the transformation result, the DCT conversion layer performs discrete cosine inverse transform on the three channels of Y, U, and V respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image; The upsampling module includes a fifth complex convolution layer, a fourth complex residual layer, a fifth complex residual layer, a sixth complex residual layer, a sixth complex convolution layer, a seventh complex convolution layer, a pixel rearrangement layer, and an eighth complex convolution layer, which are sequentially connected. The number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, and 3, respectively. The convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation. The pixel rearrangement layer performs an upsampling operation. During upsampling, the width and height of the holographic image are doubled, and the number of channels is reduced to a quarter of the original. The number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure. Step 2: Construct a bit rate allocation model. Based on the transmission network rate limit, perform unequal bit rate allocation on the real and imaginary parts of the reduced holographic image using the Lagrangian relaxation method. Compress the reduced holographic image according to the allocated bit rate and transmit it. Step 3: Decompress the real and imaginary parts of the received holographic image after code rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the original input size.

2. The holographic communication method based on rescaling according to claim 1, characterized in that: Use the gradient descent method to update the weights of the complex convolutional neural network until the loss lose The value is reduced to convergence, and a trained complex convolutional neural network is obtained.

3. The holographic communication method based on rescaling according to claim 1, characterized in that: The step 2 includes estimating the bit rate and distortion of the real image and the imaginary image respectively using information entropy theory and Pasval's theorem under different compression qualities, and generating a bit rate table and a distortion table. According to the transmission network rate limit, the bit rate and holographic reconstruction amplitude distortion are balanced through a bit rate allocation model and a lookup table algorithm to allocate the optimal bit rate to the real and imaginary images.

Images