Holographic communication method based on rescaling

Abstract

The invention discloses a holographic communication method based on rescaling, and the method comprises the steps: 1, constructing a complex convolutional neural network, training the complex convolutional neural network, carrying out the down-sampling of an input holographic image based on the trained complex convolutional neural network, and obtaining a reduced holographic image; 2, constructing a code rate distribution model, performing code rate distribution on a real part and an imaginary part of the reduced holographic image, and then performing compression and transmission; and step 3, decompressing the real part and the imaginary part of the received holographic image subjected to code rate allocation, and performing up-sampling based on the trained complex convolutional neural network to obtain the holographic image with the original input size. According to the method, the rapid transmission of the holographic image is realized, the distortion of the holographic image during transmission is reduced, the operation efficiency is high, and meanwhile, the method can adapt to the change of the network channel rate.

Description

Technical Field

[0001] The present invention belongs to the field of information communication technology, and in particular relates to a holographic communication method based on rescaling. Background Art

[0002] With the rapid development of industry and people's constant pursuit of a better life, traditional 2D images can no longer meet the needs of human production and life. Ordinary 2D images are obtained by recording the intensity distribution of diffuse reflected light of the subject on the "negative" through the principle of lens imaging. Ordinary images can only provide a certain angle of the scene. Even a wide-angle lens cannot effectively and comprehensively express the scene. At the same time, holograms have emerged as a type of digital image that is more expressive and richer in information than ordinary images.

[0003] A hologram is an interference pattern that records the light waves of an object. It has the characteristics of three-dimensional, full-viewing angle, and full color. It uses the interference and diffraction principles of lasers to record the intensity distribution and phase information of the diffusely reflected light of the object being photographed. The hologram obtained based on these two types of information is more detailed, richer, and more comprehensive than the general image of the object being photographed. With the development of computers, computer-generated holograms (CGH) have gradually become mainstream.

[0004] Compared with traditional planar images, the storage and transmission of holograms also require higher bandwidth and storage space, because computer-generated holograms need to record a large amount of information, the data volume is large, and the storage and transmission costs are high. During the reconstruction process, computer-generated holograms require complex calculations and digital processing, the computational complexity is high, and the reconstructed image quality of the generated hologram is poor, with problems such as zero-order diffraction, speckle noise, and color aberration, which affect the image quality. With the development of technology, people have higher and higher requirements for the real-time transmission and reconstruction quality of hologram data. How to save the transmission cost of holograms and minimize the impact on the reconstruction quality of holograms is a problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a holographic communication method based on rescaling, which supports holographic images in various formats, realizes fast transmission of holographic images, and has high rate-distortion performance.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A holographic communication method based on rescaling, the method comprising the following steps:

[0008] Step 1: construct and train a complex convolutional neural network, and downsample the input holographic image based on the trained complex convolutional neural network to obtain a reduced holographic image;

[0009] Step 2: construct a code rate allocation model, and perform unequal code rate allocation on the real and imaginary parts of the reduced holographic image based on the Lagrangian relaxation method according to the transmission network rate limit, and compress the reduced holographic image according to the allocated code rate for transmission;

[0010] Step 3: Decompress the real and imaginary parts of the received holographic image after the bit rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the size of the original input.

[0011] Furthermore, the step 1 includes: the complex convolutional neural network includes a downsampling module, a differentiable compression algorithm module, and an upsampling module connected in sequence.

[0012] Furthermore, the weights of the complex convolutional neural network are updated using the gradient descent method until the loss value is reduced to convergence, and a trained complex convolutional neural network is obtained; wherein the loss function L is defined as:

[0013] ,

[0014] In the formula, Represents multi-scale structural similarity and Combination of losses; represents the loss function used to evaluate the fidelity of the high-resolution hologram after scaling restoration, It means that the angular spectrum method (ASM) is used to simulate the diffraction process, and the restored high-resolution hologram is reconstructed on the target plane. According to the mean square error loss calculated by the attention mask, Represents the weight factor.

[0015] Furthermore, the downsampling module includes a first complex convolution layer, a second complex convolution layer, an inverse pixel rearrangement layer, a first complex residual layer, a second complex residual layer, a third complex residual layer, a third complex convolution layer, and a fourth complex convolution layer connected in sequence, wherein the number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3, respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the inverse pixel rearrangement layer performs a downsampling operation, and during downsampling, the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original; the number of convolution kernels of the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0016] Furthermore, the differentiable compression algorithm module includes a YUV conversion layer, a UV downsampling layer, a DCT transformation layer, a quantization layer, a DCT inverse transformation layer, a UV upsampling layer, and an RGB conversion layer connected in sequence, wherein the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT transformation layer performs discrete cosine transform on the three channels of Y, U, and V respectively, the quantization layer performs rounding operation on the transformation result, the DCT transformation layer performs discrete cosine inverse transform on the three channels of Y, U, and V respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image.

[0017] Furthermore, the upsampling module includes a fifth complex convolution layer, a fourth complex residual layer, a fifth complex residual layer, a sixth complex residual layer, a sixth complex convolution layer, a seventh complex convolution layer, a pixel rearrangement layer, and an eighth complex convolution layer, which are sequentially connected, wherein the number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the pixel rearrangement layer performs an upsampling operation, and during the upsampling, the width and height of the holographic image become twice as large as before, and the number of channels is reduced to one quarter of the original; the number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0018] Furthermore, the step 2 includes estimating the bit rate and distortion of the real image and the imaginary image under different compression qualities, respectively, using information entropy theory and Pasval's theorem, and generating a bit rate table and a distortion table. According to the transmission network rate limit, the bit rate and holographic reconstruction amplitude distortion are balanced through a bit rate allocation model and a lookup algorithm to allocate the optimal bit rate to the real and imaginary images.

[0019] The beneficial effects of the present invention are:

[0020] The present invention constructs a complex convolutional neural network composed of an upsampling module, a differentiable compression algorithm module and an upsampling module, which has good feature extraction performance for holographic images and considers the correlation between the real part and the imaginary part in the feature enhancement process, thus overcoming the problem that the existing rescaling method has good effect on ordinary images but poor effect on holographic images;

[0021] The present invention can convert the holographic image and perform down-sampling according to the client request and the real-time network channel rate, and compress it into a specified format for transmission to the client, thereby realizing the rapid transmission of the holographic image, reducing the distortion of the holographic image during transmission, having high computational efficiency, and being able to adapt to changes in the network channel rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a holographic communication method based on rescaling according to the present invention;

[0023] Figure 2 A network principle diagram of a holographic communication method based on rescaling according to the present invention;

[0024] Figure 3 It is a structural diagram of a complex convolutional neural network;

[0025] Figure 4 This is a schematic diagram of the code rate allocation model;

[0026] Figure 5 A comparison diagram of the original image and the restored image transmitted based on the method of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings

[0028] like Figure 1-Figure 2 As shown, the present invention is a holographic communication method based on rescaling, firstly, a complex convolutional neural network is constructed to downsample the holographic image on the server side to obtain a reduced holographic image, and then according to the network transmission rate limit, the bit rate allocation module is used to allocate and compress the real and imaginary parts of the reduced holographic image, and the real and imaginary parts of the reduced holographic image are compressed at different bit rates, and the result is transmitted to the client. Specifically, under different compression qualities, the information entropy theory and Pasval's theorem are used to estimate the bit rate and distortion of the real image and the imaginary image, and a bit rate table and a distortion table are generated. The bit rate allocation model and the table lookup algorithm are used to balance the bit rate and the holographic reconstruction amplitude distortion, and a better bit rate is allocated to the real and imaginary parts; the real and imaginary parts of the reduced holographic image are decompressed on the client, and the reduced holographic image is upsampled to obtain a holographic image of the original size. The structure of the upsampling module is similar to that of the downsampling module, but there are some significant differences. Specifically, the pixel arrangement layer replaces the inverse pixel arrangement layer in the upsampling stage, and the order of some layers and modules has been changed. This makes the downsampling module and the upsampling module have a symmetrical structure, which helps to restore the high-resolution hologram from the low-resolution hologram.

[0029] The present invention specifically comprises the following steps:

[0030] Step 1, construct a complex convolutional neural network;

[0031] The complex convolutional neural network of the present invention is composed of a downsampling module, a differentiable compression algorithm module, and an upsampling module connected in sequence, wherein: the downsampling module is used to downsample the holographic image, referring to Figure 3In an embodiment of the present invention, there are eight layers connected sequentially, and the structure is: first complex convolution layer→second complex convolution layer→inverse pixel rearrangement layer→first complex residual layer→second complex residual layer→third complex residual layer→third complex convolution layer→fourth complex convolution layer.

[0032] The parameters of each layer are set as follows: the number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs convolution operation and Leaky ReLu activation operation; the inverse pixel rearrangement layer performs a downsampling operation, during which the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original; the number of convolution kernels in the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0033] This method can achieve the downsampling effect without convolution operation by utilizing the inverse pixel rearrangement layer, thereby reducing the number of parameters and speeding up the image compression. Multiple complex convolution layers and complex residual layers can be used to extract key elements in the original holographic image for image restoration in the subsequent upsampling module.

[0034] The differentiable compression algorithm module is used to implement end-to-end compressed sensing training for complex convolutional neural networks. Figure 3 In the example of the present invention, there are seven layers connected in sequence, and the structure is: YUV conversion layer → UV downsampling layer → DCT conversion layer → quantization layer → DCT inverse transformation layer → UV upsampling layer → RGB conversion layer;

[0035] Among them, the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT transformation layer performs discrete cosine transform on the Y, U, and V channels respectively, the quantization layer rounds the transformation result, the DCT transformation layer performs inverse discrete cosine transform on the Y, U, and V channels respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image.

[0036] This method simulates the differentiable form of the compression algorithm and connects the up and down sampling modules so that the complex neural network can form a whole for end-to-end training. Moreover, through the quantization loss in the compression process, the complex neural network can better restore the compressed image.

[0037] The upsampling module is used to restore the image after downsampling and compression distortion. In an embodiment of the present invention, there are 8 layers connected in sequence, and their structures are: fifth complex convolution layer → fourth complex residual layer → fifth complex residual layer → sixth complex residual layer → sixth complex convolution layer → seventh complex convolution layer → pixel rearrangement layer → eighth complex convolution layer.

[0038] The parameters of each layer are set as follows: the number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs convolution operation and Leaky ReLu activation operation; the pixel rearrangement layer performs an upsampling operation, during which the width and height of the holographic image are doubled and the number of channels is reduced to one quarter of the original; the number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

[0039] This method can achieve the upsampling effect without convolution operation by utilizing the pixel rearrangement layer, thereby reducing the number of parameters and speeding up the image recovery. Multiple complex convolution layers and complex residual layers can be used to generate high-frequency details in the original holographic image, thereby restoring the original holographic image.

[0040] Finally, the downsampling module, the differentiable compression algorithm module, and the upsampling module are connected in sequence to form the complex convolutional neural network in the present invention.

[0041] The input volume function and output volume function of each convolutional layer of the present invention are as follows:

[0042] ,

[0043] ,

[0044] Except for the first complex convolutional layer whose input is the original holographic image, each layer of the network will get the input feature map from the previous layer of the network, where , and is the height, width and number of output channels of the feature map output by the previous layer of network; except that the output of the eighth complex convolutional layer is the restored holographic image, each layer of network will output a feature map, where , and The height, width, and number of output channels of the feature map output by this layer of the network.

[0045] Furthermore, training the neural network includes:

[0046] Input the training set into the complex convolutional neural network, and use the gradient descent method to update the weights of the complex convolutional neural network until the loss value is reduced to convergence, and obtain a trained complex convolutional neural network;

[0047] The present invention defines the loss function L as:

[0048] ,

[0049] in, Represents multi-scale structural similarity ( )and Combination of losses; represents the loss function used to evaluate the fidelity of the restored high-resolution hologram, that is, the mean square error between the restored hologram and the original hologram; in order to achieve a good visual perception effect of the hologram on the target plane, use Losses guide this process, It means that the angular spectrum method (ASM) is used to simulate the diffraction process, the restored high-resolution hologram is reconstructed on the target plane, and then the mean square error loss is calculated according to the attention mask, where Represents a weight factor, which is used to balance different losses and is set to 0.04 in the present invention.

[0050] Step 2: Build a rate allocation model:

[0051] like Figure 4 As shown, the rate allocation model of the present invention is derived from the relationship between the amplitude image of the holographic image and the real image and imaginary image of the holographic image, wherein the rate allocation model is a non-equal rate allocation based on the Lagrangian relaxation method; rate allocation is a key aspect of data flow because it determines how many bits are allocated to different parts of the stream by complying with the aggregate rate constraint, and the rate-distortion prediction module of the present invention realizes the prediction of rate and distortion based on the information entropy theorem and Parseval's theorem respectively.

[0052] In the hologram stream, since the reconstructed hologram amplitude will eventually be viewed by the user, its distortion reflects the visual quality of the reconstructed hologram; in the present invention, since the real image and imaginary image distortion have different effects on the reconstructed amplitude distortion, different bit rates need to be allocated to the real and imaginary data to transmit the hologram. In this embodiment, the transmission network rate limit is set to ,in and are the bit rates of the real and imaginary image data respectively.

[0053] Using a function To approximate the reconstruction of the hologram distortion Image distortion with real and imaginary parts and The relationship between: , the derivation process is as follows:

[0054] Assume that the real part is , the imaginary part is , and The errors caused by compression are and , the amplitude error of the reconstructed hologram introduced by compression is , we can get: .

[0055] After Taylor expansion, by retaining the first-order terms, we have:

[0056] .

[0057] The amplitude distortion of the reconstructed hologram is expressed as , so it is approximately:

[0058] .

[0059] in, yes and The average value of the multiplication is close to 0 and can be omitted. Therefore:

[0060] .

[0061] in, Represents the phase of the holographic image;

[0062] Since the errors in the original phase and the real and imaginary parts are approximately independent, the amplitude distortion of the reconstructed hologram becomes:

[0063] .

[0064] Arranging the above formulas yields:

[0065] .

[0066] This method approximates the complex distortion relationship into a linear relationship through the Taylor formula, so that it can be used as a sub-module in the complex convolutional neural network, improving the speed and efficiency of bit rate allocation.

[0067] Step 3: Decompress the real and imaginary parts of the received holographic image after the bit rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the size of the original input.

[0068] Furthermore, the method is applied to transmit the target holographic image, and the target holographic image to be transmitted is sequentially input into the trained complex convolutional neural network, and the holographic image restored after the transmission compression process is output. The peak signal-to-noise ratio (PSNR) and structural similarity metric (SSIM) are used to judge the similarity between the output image and the original image, and the rate-distortion performance is used to judge its compression efficiency. Figure 5 As shown, the large image in the first row represents the original hologram, and three local holographic images are displayed for comparing image quality; the second row represents a local enlarged image of the reconstructed image of the holographic image after transmission recovery; the third row represents a local enlarged image of the reconstructed image of the original holographic image; comparing the various details in the local enlarged image, such as letters and doll outlines, it can be seen that it is very close to the original image. The use of this method can compress the holographic image well, and the rate-distortion performance is high. The distortion of the image can be reduced under limited bandwidth, and the restored image is consistent with the original image.

[0069] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A holographic communication method based on rescaling, characterized in that: The method comprises: Step 1: construct and train a complex convolutional neural network, and downsample the input holographic image based on the trained complex convolutional neural network to obtain a reduced holographic image; Step 2: construct a code rate allocation model, and perform unequal code rate allocation on the real and imaginary parts of the reduced holographic image based on the Lagrangian relaxation method according to the transmission network rate limit, and compress the reduced holographic image according to the allocated code rate for transmission; Step 3: Decompress the real and imaginary parts of the received holographic image after the bit rate allocation, and upsample it based on the trained complex convolutional neural network to obtain a holographic image of the original input size.

2. A holographic communication method based on rescaling according to claim 1, characterized in that: The step 1 includes: the complex convolutional neural network includes a downsampling module, a differentiable compression algorithm module, and an upsampling module which are connected in sequence.

3. The holographic communication method based on rescaling according to claim 1, characterized in that: Use the gradient descent method to update the weights of the complex convolutional neural network until the loss value drops to convergence, and obtain the trained complex convolutional neural network; where the loss function L is defined as: ， In the formula, Represents multi-scale structural similarity and Combination of losses; represents the loss function used to evaluate the fidelity of the high-resolution hologram after scaling restoration, It means that the angular spectrum method is used to simulate the diffraction process, and the restored high-resolution hologram is reconstructed on the target plane. According to the mean square error loss calculated by the attention mask, Represents the weight factor.

4. The holographic communication method based on rescaling according to claim 2, characterized in that: The downsampling module includes a first complex convolution layer, a second complex convolution layer, an inverse pixel rearrangement layer, a first complex residual layer, a second complex residual layer, a third complex residual layer, a third complex convolution layer, and a fourth complex convolution layer, which are sequentially connected, wherein the number of convolution kernels in the first to fourth complex convolution layers is set to 8, 16, 256, and 3, respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the inverse pixel rearrangement layer performs a downsampling operation, and during downsampling, the width and height of the holographic image are reduced to half of the original, and the number of channels is increased to four times the original; the number of convolution kernels in the first to third complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

5. The holographic communication method based on rescaling according to claim 2, characterized in that: The differentiable compression algorithm module includes a YUV conversion layer, a UV downsampling layer, a DCT conversion layer, a quantization layer, a DCT inverse conversion layer, a UV upsampling layer, and an RGB conversion layer which are sequentially connected, wherein the YUV conversion layer converts the RGB format image output by the fourth complex convolution into a YUV image, the UV downsampling layer downsamples the image of the UV channel, the DCT conversion layer performs discrete cosine transform on the three channels of Y, U, and V respectively, the quantization layer performs an integer operation on the transformation result, the DCT conversion layer performs discrete cosine inverse transform on the three channels of Y, U, and V respectively, the UV upsampling layer upsamples the image of the UV channel, and the RGB conversion layer converts the YUV format image into an RGB image.

6. The holographic communication method based on rescaling according to claim 2, characterized in that: The upsampling module includes a fifth complex convolution layer, a fourth complex residual layer, a fifth complex residual layer, a sixth complex residual layer, a sixth complex convolution layer, a seventh complex convolution layer, a pixel rearrangement layer, and an eighth complex convolution layer, which are sequentially connected, wherein the number of convolution kernels in the fifth to eighth complex convolution layers is set to 64, 64, 256, 3 respectively, the convolution kernel size is set to 3×3, the step size is set to 1, and each convolution layer performs a convolution operation and a Leaky ReLu activation operation; the pixel rearrangement layer performs an upsampling operation, and during the upsampling, the width and height of the holographic image become twice as large as before, and the number of channels is reduced to one quarter of the original; the number of convolution kernels in the fourth to sixth complex residual layers is set to 64, the step size is set to 1, and each complex residual layer performs a convolution operation and then a residual structure.

7. The holographic communication method based on rescaling according to claim 1, characterized in that: The step 2 includes estimating the bit rate and distortion of the real image and the imaginary image respectively using information entropy theory and Pasval's theorem under different compression qualities, and generating a bit rate table and a distortion table. According to the transmission network rate limit, the bit rate and holographic reconstruction amplitude distortion are balanced through a bit rate allocation model and a lookup algorithm to allocate the optimal bit rate to the real and imaginary images.

Images