Object light decoding device, object light inverse quantization device, object light correction device, object light decoding method, and program

Abstract

To apply object light required for holographic three-dimensional image display to existing video encoding standards, and to efficiently encode and transmit the object light, and then to display it correctly on a receiving device.SOLUTION: An object light decoding device 1 according to the present invention includes an image decoding device 10 that receives an input bit stream of encoded object light and outputs integer-converted object light, an object light inverse quantization device that applies linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by decimal points, and an object light correction device 30 that corrects the object light data to match the pixel spacing of a reproduction optical system.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object light decoding device, an object light inverse quantization device, an object light correction device, an object light decoding method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technique related to video image coding. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Versatile Video Coding, Version 1, Rec. ITU-T H.266, ISO / IEC FDIS 23090-3, July 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] The data covered by Non-Patent Document 1 is primarily images intended to be displayed on a device (hereinafter referred to as a display) that displays images and videos by emitting light from three-color (RGB) pixels, such as a liquid crystal display.

[0005] On the other hand, three-dimensional image display, a method known as holography, requires data on the amplitude and phase of the object light, and because the format is different, there is a problem that it cannot be encoded by simply applying conventional image encoding techniques.

[0006] In addition, for the optical amplitude and phase data, it is necessary to convert the r interval to match the reproduction optical system (device).

[0007] Here, among the data for holographic display, the "amplitude and phase distribution of light from an object independent of the reproduction optical system (device)" that is the subject of the present invention will be referred to as "object light" in this embodiment.

[0008] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide an object light decoding device, an object light inverse quantization device, an object light correction device, an object light decoding method, and a program that can apply the object light required for holographic three-dimensional image display to existing video encoding standards, and that can efficiently encode and transmit the object light, and then correctly display it on a receiving device. [Means for solving the problem]

[0009] A first feature of the present invention is that it comprises an object light decoding device comprising an image decoding device that receives an input of a bit stream of encoded object light and outputs integer-converted object light data; an object light inverse quantization device that applies linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by decimal points; and an object light correction device that corrects the object light data described by decimal points to match the pixel spacing of a reproduction optical system.

[0010] A second feature of the present invention is an object light inverse quantization device that converts integer-converted object light data output by an image decoding device into object light data described by a decimal point in accordance with a flag described in metadata included in a bit stream of the encoded object light.

[0011] A third aspect of the present invention is an object light correction device which refers to a sampling interval when calculating the object light described in metadata included in a bit stream of encoded object light, and refers to a pixel interval of a display device provided in a receiving device, and resamples and corrects object light data so that the sampling interval matches the pixel interval.

[0012] A fourth feature of the present invention is summarized as comprising the steps of: receiving a bit stream of encoded object light as an input, and outputting integer-converted object light data; applying linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by decimal points; and correcting the object light data described by decimal points to match the pixel spacing of a reproduction optical system.

[0013] A fifth feature of the present invention is a program for causing a computer to function as an object light decoding device, the object light decoding device having a function of receiving a bit stream of encoded object light as input and outputting integer-converted object light data, a function of applying linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by decimal points, and a function of correcting the object light data described by decimal points to match the pixel spacing of a reproduction optical system. Effect of the Invention

[0014] According to the present invention, it is possible to provide an object light decoding device, an object light inverse quantization device, an object light correction device, an object light decoding method, and a program that can apply the object light required for holographic stereoscopic image display to existing video encoding standards, and efficiently encode and transmit the object light, and then correctly display it on a receiving device. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of functional blocks of an object light decoding device 1 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of metadata used in an embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of functional blocks of the object light quantization device 20 according to an embodiment. [Figure 4] FIG. 4 shows an example of the operation of the object optical decoding device 1 according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of differences in data types according to data_type in one embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a mapping table defined by the mapping_table included in the metadata in one embodiment. [Figure 7] FIG. 7 is a diagram for explaining an example of a method of interpolating the sampling interval by the object beam corrector 30 according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0017] First Embodiment Hereinafter, an object optical decoding device 1 according to an embodiment of the present invention will be described with reference to FIG.

[0018] As shown in FIG. 1, an object light decoding device 1 according to this embodiment includes an image decoding device 10, an object light inverse quantization device 20, and an object light correction device 30.

[0019] (Image Decoding Device 10) The image decoding device 10 is configured to receive as input a bit stream (encoded bit stream) obtained by encoding an object light, and to output integer-converted object light data (n bit int).

[0020] The image decoding device 10 is configured to generate integer-converted object light data (n bit int) by decoding a bitstream (encoded bitstream) with metadata attached as SEI (Supplemental Enhancement Information) or VUI (Video Usability Information) supported in Non-Patent Document 1, etc.

[0021] The coded bit stream input to the image decoding device 10 is as follows.

[0022] The object light data used in holographic reconstruction is amplitude and phase data distributed two-dimensionally. For example, a two-dimensional array is assumed in which the amplitude and phase data for each RGB color channel (or the real and imaginary part data when the amplitude and phase data are converted to values ​​on a complex plane) are recorded as floating-point or fixed-point variables for each sampling point.

[0023] Such object light data can be handled in the same way as general image data by quantizing it (integerization, imaging).

[0024] Therefore, in this embodiment, the object light data quantized as described above is encoded, and then a bit stream (encoded bit stream) to which the metadata shown in FIG. 2 is added is assumed as input.

[0025] In other words, it can be said that the image decoding device 10 is configured to receive an encoded bitstream and decode image data.

[0026] Specifically, the image decoding device 10 obtains quantized object light data by performing an inverse DCT transform and decoding based on a prediction residual.

[0027] The format of the object light data obtained at this time coincides with the format of the object light data input to an image encoding device (not shown).

[0028] For example, if an image encoding device generates and encodes image data (object light data) having integer values ​​from 0 to 255 for each color channel from an object light, the object light data (decoded quantized object light signal) obtained by image decoding device 10 will also have the same format.

[0029] Here, for the object light data, the real part data and imaginary part data obtained by converting the above-mentioned amplitude data and phase data into values ​​on a complex plane may be treated as two-dimensional arrays of floating point numbers, and hereinafter, unless otherwise specified to limit the data to the above-mentioned amplitude data and phase data, the data may be treated as complex numbers consisting of such real part data and imaginary part data.

[0030] The bit stream may be transmitted via a transmission path, or may be stored in a storage medium and then provided to the image decoding device 10 from an image encoding device (not shown).

[0031] (Object light inverse quantization device 20) The object light inverse quantization device 20 is configured to apply linear or nonlinear inverse quantization to both the amplitude and phase of the object light data (n bit int) output from the image decoding device 10, thereby outputting object light data (float) described by a decimal point.

[0032] Object beam dequantization device 20 is configured to dequantize the quantized (integer) object beam data, thereby converting it into an amplitude-phase format required for holographic reconstruction.

[0033] Specifically, as shown in FIG. 3, object light inverse quantization device 20 includes object light data discrimination section 21, inverse quantization mode selection section 22, and mapping section .

[0034] The object light data discrimination unit 21 is configured to determine whether or not the encoded bit stream input to the image decoding device 10 is an encoded bit stream of object light.

[0035] When the object light data discrimination unit 21 determines that the encoded bit stream is an encoded object light, the inverse quantization mode selection unit 22 is configured to determine what type of quantization was applied to the object light data, which was floating-point data, and to select an inverse quantization mode corresponding to the determination result.

[0036] The mapping unit 23 is configured to perform mapping of the object light data (n bit int) to the object light data (float) based on the inverse quantization mode selected by the inverse quantization mode selection unit 22 .

[0037] (Object light correction device 30) The object beam corrector 30 is configured to correct the object beam data (float) output from the object beam inverse quantizer 20 in accordance with the pixel interval of the reproduction optical system.

[0038] The specific function of the object beam corrector 30 will be described later.

[0039] (Operation of the object optical decoding device 1) An example of the operation of the object optical decoding device 1 according to this embodiment will be described below with reference to FIGS.

[0040] As shown in FIG. 4, in step S101, the image decoding device 10 decodes integer-converted object light data (n bit int) from the encoded bit stream acquired from the object light encoding device.

[0041] In steps S102 to S105, the object light inverse quantization device 20 converts the integer-converted object light data (n bit int) into object light data (float) described in decimal points such as floating point.

[0042] That is, the object light inverse quantization device 20 uses metadata in the encoded bitstream processed by the image decoding device 10 to perform mapping to values ​​close to the original object light data.

[0043] Specifically, in step S102, the object beam dequantization device 20 refers to object_wave_flag in the metadata to determine whether the encoded bit stream is an encoded bit stream of the object beam.

[0044] If it is determined that the encoded bit stream is not an encoded object light (object_wave_flag=0), the object light inverse quantization device 20 assumes that it is another data format and performs processing according to the procedure specified in non-patent document 1, etc., and therefore this operation ends.

[0045] If it is determined that the encoded bit stream is an encoded object light (object_wave_flag=1), in steps S103 and S104, the object light inverse quantization device 20 converts the object light data (n bit int) generated by the image decoding device 10 into amplitude-phase format data so that it can be used for holographic stereoscopic image display.

[0046] That is, in step S103, the object light dequantization device 20 refers to data_type in the metadata to determine what type of data the object light data (n bit int) decoded from the encoded bit stream consists of.

[0047] Specifically, as shown in FIG. 5, the object light inverse quantization device 20 determines whether the object light data (n bit int) corresponds to amplitude data (data_type=0), phase data (data_type=1), real part data (data_type=2), imaginary part data (data_type=3), combined amplitude / phase data (data_type=4), or combined real / imaginary part data (data_type=5) depending on the value of data_type.

[0048] Here, actual holographic reconstruction requires a set of amplitude data and phase data (or real part data and imaginary part data), but such a determination is made because it is also assumed that the amplitude data and phase data are transmitted in separate streams.

[0049] One effect of treating the amplitude data and phase data as separate streams is that it is possible to apply processing specific to each of the amplitude data and phase data individually, but on the other hand, it is necessary to transmit each of the amplitude data and phase data individually during playback, which is expected to reduce transmission efficiency.

[0050] In contrast, by treating the amplitude data and phase data as combined amplitude-phase data, it becomes possible to transmit the bit stream in a single session, and the total bit rate can be expected to be reduced.

[0051] Furthermore, when data_type is combined amplitude and phase data (4) or combined real and imaginary part data (5), i.e., when one bit stream contains a set of amplitude data and phase data (real part data and imaginary part data), the object beam inverse quantization device 20 needs to determine which bit is the amplitude (real part), and makes such a determination using the parameter amp_real_bit_param.

[0052] If it is determined that the data_type is real part data (2), imaginary part data (3), or combined real and imaginary part data (5), the object beam inverse quantization device 20 performs a conversion process to data in amplitude-phase format.

[0053] In step S105, the object beam inverse quantization device 20 determines what type of quantization has been applied to the object beam data described in floating point, and selects an inverse quantization mode corresponding to the determination result.

[0054] Here, when mapping_type, which indicates the inverse quantization mode (mapping method), indicates linear inverse quantization (mapping_type=0), the object light inverse quantization device 20 equally divides the range of object light data values ​​based on object_wave_max, which indicates the maximum value of the object light data included in the metadata, and object_wave_min, which indicates the minimum value of the object light data, and the number of bits, and maps the representative value included in each division as the value of the quantized sampling point.

[0055] Examples of the representative value include the average value, maximum value, and minimum value of the division section, and such a representative value is specified by representative_value_type.

[0056] On the other hand, when the mapping_type indicating the inverse quantization mode (mapping method) indicates nonlinear inverse quantization (mapping_type=1), the object light inverse quantization device 20 similarly performs mapping to the value of the object light data based on the mapping_table indicating the mapping table (see Figure 6) included in the metadata and the transfer_characteristics in the VUI.

[0057] Specifically, in step S105, the object light inverse quantization device 20 uses a mapping table to convert the pixel values ​​of the object light data converted into integers into mapping values ​​corresponding to the pixel values, that is, convert from integers to floating points.

[0058] Similarly, when using a transfer function of transfer_characteristics, the object light inverse quantization device 20 may map each pixel value as an input to a function output. Note that in this case, the output range is compressed or expanded to the range between the maximum value of the object light data described in object_wave_max and the minimum value of the object light data described in object_wave_min.

[0059] In step S106, the object light correction device 30 corrects the sampling interval of the object light data with reference to the metadata in accordance with the characteristics of the receiving device (display device).

[0060] Here, the object beam data (float) obtained by object beam inverse quantization apparatus 20 is data that does not depend on the reproduction optical system of the receiving device, and generally cannot be reproduced correctly as a desired holography as it is.

[0061] Therefore, reproduction is possible by correcting the object light data (float) by the object light correction device 30 so as to match it with the reproduction optical system of the receiving device.

[0062] In holographic reconstruction, the physical distance between adjacent pixels on the display device is an important parameter that affects the quality of the reconstructed image.

[0063] For this reason, when calculating object light data, the physical distance between sampling points is usually set so that a desired holographic reconstruction image can be obtained in a virtual 3D space on a computer.

[0064] Any mismatch between the sampling interval set here and the pixel interval of the display device (display) will cause degradation of the final reconstructed image.

[0065] On the premise that the distance interval of the sampling points when the object light data is calculated is given as pixel_pitch in the metadata of the encoded bit stream, the object light correction device 30 corrects the sampling interval of the object light data.

[0066] FIG. 7 shows an example of a method for correcting the sampling interval.

[0067] In the example of FIG. 7(b), it is considered that a hologram obtained from object light data calculated with 8K resolution / pixel pitch=4 nm is displayed on a display device with 8K resolution / pixel pitch=2 nm.

[0068] In this case, if the hologram were to be displayed as is, the 8K resolution / pixel pitch would be 2nm, which could result in a reproduced image that is different from the intended image.

[0069] Therefore, the object light data is corrected in advance by the object light corrector 30 so that the sampling interval is 2 nm, and then displayed.

[0070] That is, in the above example, the object beam correction device 30 places new sampling points midway between the sampling points, performs interpolation so that the sampling interval is 2 nm, and then displays the corrected object beam data, thereby solving the problem during reproduction.

[0071] For such upsampling processing, existing interpolation techniques such as bicubic, bilinear, nearest neighbor, etc. can be widely applied.

[0072] Alternatively, the object beam corrector 30 may perform interpolation so as to increase the difference between adjacent pixels. In such a case, the object beam corrector 30 may perform interpolation so as to maximize the total variation of the entire or partial region of the object beam data.

[0073] Similarly, if the pixel pitch of the display device is large compared to the sampling interval when calculating the object light data shown in Figure 7(a), the object light correction device 30 may downsample the object light data to match the pixel pitch of the display device, and then display the object light data on the display device.

[0074] Due to the above-mentioned interpolation process, when displayed on a display device, there is a possibility that a difference will occur between the number of pixels of the object light data and the number of pixels of the display device.

[0075] When the number of pixels of the object light data is large, the display device may perform display using only a portion of the object light data, as shown in FIG. 7(c).

[0076] In such a case, the area to be displayed on the display device may be the center of the object light data, or a position directly facing the object may be selected based on the position of the object previously assigned as metadata, or a position where the representative RGB value of the image that actually reproduces the object light data and is captured by another imaging system may be selected.

[0077] Furthermore, when the number of pixels of the object light data is small, all of the object light data may be displayed using a partial area of ​​the display device (which results in an unused area being generated in part of the display device).

[0078] In such a case, the method of selecting the display position of the object light data may be to simply display it at the center of the display device, or to display it at a position closest to the user's predetermined viewing position or eye position.

[0079] According to the above-described embodiment, the object light required for holographic stereoscopic image display can be applied to existing video encoding standards, and the object light can be efficiently encoded and transmitted, and then correctly displayed on a receiving device.

[0080] The above-described image decoding device 10, object light inverse quantization device 20, and object light correction device 30 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0081] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0082] 1...Object light decoding device 10...Image decoding device 20…Object light inverse quantization device 21...Object light data discrimination unit 22... Inverse quantization mode selection unit 23…Mapping section 30…Object light correction device

Claims

1. An object optical decoding device, comprising: an image decoding device that receives an encoded object light bit stream as an input and outputs integer-converted object light data; an object light inverse quantization device that applies linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by a decimal point; and an object beam correction device that corrects the object beam data described by the decimal point to match the pixel interval of a reproduction optical system.

2. An object beam inverse quantization apparatus, comprising: An object light inverse quantization device that converts integer-converted object light data output by an image decoding device into object light data described by a decimal point in accordance with a flag described in metadata included in the bit stream of the encoded object light.

3. determining whether the object light data decoded from the bit stream corresponds to amplitude data, phase data, real part data, imaginary part data, combined amplitude / phase data, or combined real / imaginary part data according to parameters described in the metadata; 3. The object light inverse quantization apparatus according to claim 2, wherein the object light data is converted into object light data described by the decimal point based on the result of the determination.

4. 4. The object light inverse quantization device according to claim 3, wherein when the object light data is the real part data, the imaginary part data, or the combined real and imaginary part data, the object light data is converted into data in an amplitude-phase format.

5. 3. The object light inverse quantization device according to claim 2, wherein linear inverse quantization or nonlinear inverse quantization is applied in accordance with a quantization mode described in the metadata to perform conversion to object light data described by the decimal point.

6. 6. The object light inverse quantization device according to claim 5, wherein, at sampling points of the integer-converted object light data, the linear inverse quantization is applied to a portion of the data, and the nonlinear quantization is applied to the remaining data, according to the quantization mode.

7. 3. The object light inverse quantization device according to claim 2, further comprising: a pixel value of the integer-converted object light data being mapped to object light data described by floating point numbers according to a transfer function corresponding to the description in the metadata.

8. 3. The object light inverse quantization device according to claim 2, wherein pixel values ​​of the object light data converted into integers are mapped to object light data described by floating point numbers according to a mapping table described in the metadata.

9. An object beam corrector, comprising: Refer to the sampling interval for calculating the object light, which is described in the metadata included in the bit stream of the encoded object light; Refer to the pixel spacing of the display device provided in the receiving device, an object beam correction device that resamples and corrects the object beam data so that the sampling interval and the pixel interval match;

10. receiving an encoded object beam bit stream as input and outputting integerized object beam data; applying linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light to output object light data described by a decimal point; and correcting the object beam data described by the decimal point to match the pixel spacing of a reproduction optical system.

11. A program for causing a computer to function as an object optical decoding device, comprising: The object optical decoding device comprises: a function for receiving an encoded object beam bit stream as input and outputting integer-coded object beam data; a function of applying linear or nonlinear inverse quantization to each of the amplitude and phase of the integer-converted object light, thereby outputting object light data described by a decimal point; and a function of correcting the object light data described by the decimal point to match the pixel interval of the reproduction optical system.

Images