Multi-viewpoint image encoding device and program thereof, and multi-viewpoint image decoding device and program thereof

Abstract

To provide a multi-viewpoint image encoder capable of changing the compression rate of a multi-viewpoint image by distance.SOLUTION: A multi-viewpoint image encoder 1 includes: viewpoint image layering means 11 for generating a viewpoint image by layer for each preset depth interval, for each of viewpoint images, from the viewpoint images constituting a multi-viewpoint image, based on a distance image corresponding to the multi-viewpoint image;image encoding means 13 by layer for encoding a plurality of the viewpoint images by layer having the same depth interval based on preset code amount control parameters for each depth interval, and generating an image coded stream by layer for each depth interval; and multiplexing means 15 for multiplexing the image coded stream by layer for each depth interval and generating a coded stream.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a multi-viewpoint image encoding device and a program therefor, and a multi-viewpoint image decoding device and a program therefor. [Background technology]

[0002] Currently, a method is being considered for transmitting multi-viewpoint images displayed on a ray reproduction type display together with a distance image (depth map) indicating distance information from the shooting position (viewpoint position). International standards define coding methods such as 3D-AVC and 3D-HEVC that code multi-viewpoint images and distance images and generate arbitrary viewpoint images on the decoding side (see Non-Patent Documents 1 and 2).

[0003] In such a conventional multi-viewpoint image coding method, a technique has been proposed in which the difference between the viewpoint images is coded and transmitted in order to reduce the amount of code. Specifically, in the conventional encoding method, as shown in Fig. 14(a), it is assumed that an object O is photographed from a viewpoint position a and another viewpoint position b. At this time, as shown in Fig. 14(b), a viewpoint image Ga at the viewpoint position a and a viewpoint image Gb at the viewpoint position b are respectively photographed. In addition, the conventional encoding method creates a difference image Gc shown in FIG. 14(c), which is the difference between the viewpoint image Ga and the viewpoint image Gb, and then encodes and transmits the viewpoint image Ga (or viewpoint image Gb) and the difference image Gc, thereby reducing the amount of code. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Shimizu, "Trends in international standardization of 3D video coding using depth maps," IPSJ Technical Report, Vol. 2013-AVM-82, No. 11, pp. 1-6 (2013). [Non-Patent Document 2] Senoo, Y. Yamamoto, Oi, and Kurita, "Standardization Activities for MPEG Multi-viewpoint Video Coding," National Institute of Information and Communications Technology Quarterly Report, Vol. 56, Nos. 1 / 2, pp. 79-90 (2010). Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional encoding methods have the following problems. In a ray reproduction type 3D display device, the display resolution changes depending on the distance from the display. Specifically, in a ray reproduction type 3D display device, the spatial frequency is highest near the lens array, and the spatial frequency decreases as you move forward or backward from the lens array. Therefore, if conventional encoding techniques are applied directly to multi-viewpoint images, there is a problem in that if degradation of the reproduced image occurs due to encoding compression, the degradation occurs throughout the reproduced image, and the degradation becomes noticeable in the stereoscopic image near the high-resolution lens array. Furthermore, in the conventional method of reducing the amount of code by transmitting a difference image, the difference image is generated without referring to distance information, so images of different depths are mixed in the difference image, and the amount of code cannot be changed according to distance, resulting in the same problems as described above.

[0006] The present invention has been made in consideration of such problems, and has an objective of providing a multi-viewpoint image encoding device and a program thereof, as well as a multi-viewpoint image decoding device and a program thereof, which are capable of changing the compression rate of multi-viewpoint images according to distance by referring to distance information. [Means for solving the problem]

[0007] In order to solve the above problem, the multi-viewpoint image encoding device of the present invention is a multi-viewpoint image encoding device that encodes multi-viewpoint images, and is configured to include a viewpoint image layering means, a layer-by-layer image encoding means, and a multiplexing means.

[0008] In such a configuration, the multi-viewpoint image coding device generates layered viewpoint images for each preset depth section from viewpoint images constituting the multi-viewpoint image based on a distance image corresponding to the multi-viewpoint image by the viewpoint image layering means. The layered viewpoint images are images in which only the images of the corresponding depth sections are extracted from the viewpoint images. As a result, each of the multiple viewpoint images is separated for each depth section. Then, the multi-viewpoint image encoding device uses a layer-by-layer image encoding means to encode multiple layer-by-layer viewpoint images having the same depth range based on a code amount control parameter that is preset for each depth range, thereby generating a layer-by-layer image encoding stream for each depth range. In this way, the layer-specific image encoding means performs encoding for each layer-specific viewpoint image in the same depth section, and can set individual parameters for controlling the amount of code for each depth section. Then, the multi-viewpoint image coding apparatus multiplexes the layer-specific image coding streams for each depth section using a multiplexing means to generate a coding stream. This allows the multi-viewpoint image encoding device to encode the multi-viewpoint images by changing the compression rate for each depth section. The multi-viewpoint image encoding device can be operated by a program that causes a computer to function as each of the means described above.

[0009] In addition, in order to solve the above problem, the multi-viewpoint image decoding device of the present invention is a multi-viewpoint image decoding device that decodes an encoded stream of the multi-viewpoint image generated by multiplexing layer-by-layer image encoding streams for each depth zone, in which the viewpoint images constituting the multi-viewpoint image are encoded for each of a plurality of layer-by-layer viewpoint images having the same depth zone, and is configured to include a multiplexing separation means, a layer-by-layer image decoding means, and a viewpoint image synthesis means.

[0010] In such a configuration, the multi-viewpoint image decoding device separates the coded stream into a plurality of layer-specific image coded streams for each depth section by the demultiplexing means. Then, the multi-viewpoint image decoding device decodes the layer-by-layer image encoded stream into a plurality of layer-by-layer viewpoint images in the same depth section by the layer-by-layer image decoding means. Then, the multi-viewpoint image decoding device generates a viewpoint image by synthesizing, by the viewpoint image synthesis means, layer-specific viewpoint images corresponding to the same viewpoint among the layer-specific viewpoint images. This allows the multi-viewpoint image decoding device to decode multi-viewpoint images that have been encoded with different compression rates for each depth section. The multi-viewpoint image decoding device can be operated by a program that causes a computer to function as each of the means described above. Effect of the Invention

[0011] The present invention provides the following excellent effects. According to the present invention, a multi-viewpoint image can be encoded / decoded by changing the compression ratio for each depth section. As a result, when encoding a multi-viewpoint image displayed on a ray reproduction type display, the present invention can suppress deterioration of image quality in the depth region, which has high perceptual sensitivity, and can encode the image with high compression efficiency. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the overall configuration of a multi-viewpoint image encoding device according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing the internal configuration of a viewpoint image layering means in FIG. 1. FIG. [Diagram 3] FIG. 1 is an explanatory diagram for explaining a method for generating a multi-view image. [Figure 4] FIG. 1 is an explanatory diagram for explaining distance images, in which (a) shows viewpoint images constituting a multi-viewpoint image, and (b) shows a distance image corresponding to (a). [Diagram 5] FIG. 11 is an explanatory diagram for explaining a threshold value of a depth section. [Figure 6] 13(a) to 13(c) are diagrams showing mask data for each depth section. [Figure 7]13(a) to 13(c) are diagrams showing layer-specific viewpoint images for each depth section. [Figure 8] 1 is a diagram showing spatial frequency characteristics in which the horizontal axis represents distance from the position of the lens array as a reference position, and the vertical axis represents spatial frequency. [Figure 9] FIG. 2 is an explanatory diagram for explaining an encoding method of the encoding means of FIG. [Figure 10] 4 is a flowchart showing an operation of the multi-viewpoint image encoding device according to the embodiment of the present invention. [Figure 11] 1 is a block diagram showing an overall configuration of a multi-view image decoding device according to an embodiment of the present invention. [Figure 12] FIG. 12 is an explanatory diagram for explaining a decoding method of the decoding means in FIG. [Figure 13] 5 is a flowchart showing the operation of the multi-view image decoding device according to the embodiment of the present invention. [Figure 14] FIG. 1 is an explanatory diagram for explaining a conventional encoding method, in which (a) is a diagram showing the relationship between a subject and a viewpoint position, (b) is a diagram showing images captured at each viewpoint position, and (c) is a diagram showing a difference image between the viewpoint images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <Configuration of a multi-viewpoint image encoding device> The configuration of a multi-view image encoding device 1 according to an embodiment of the present invention will be described with reference to FIG.

[0014] The multi-viewpoint image encoding device 1 encodes multi-viewpoint images. The multi-viewpoint image encoding device 1 changes compression efficiency for each depth section that divides the depth specified in a distance image (distance information) of a subject. First, the images (multi-viewpoint image P, distance image D) input to the multi-viewpoint image encoding device 1 will be described, and then the configuration of the multi-viewpoint image encoding device 1 will be described.

[0015] The multi-viewpoint image P is an image composed of images from a plurality of viewpoint positions. The multi-viewpoint image P does not need to be a real-life image, and may be, for example, a CG image in which a subject is virtually photographed by a virtual camera. The multi-viewpoint image encoding device 1 encodes the multi-viewpoint image P as a viewpoint image P. 1 ,P 2 ,…,P n Enter the information for each

[0016] The distance image (distance information) D is an image that represents the distance from the viewpoint position to the subject in the subject space where the subject is photographed, with pixel values ​​assigned to a range from a minimum depth value to a maximum depth value that is predetermined for each pixel. 1 ,P 2 ,…,P n The corresponding distance image (viewpoint distance image) D 1 ,D 2 ,…,D n It is composed of: It should be noted that the resolution of the multi-viewpoint image P and the distance image D do not necessarily need to be the same, and it is sufficient that the pixels of the multi-viewpoint image P and the pixels of the distance image D are associated with each other in advance.

[0017] Here, examples of a multi-viewpoint image P and a distance image D will be described with reference to Fig. 3 and Fig. 4. As shown in Fig. 3, a multi-viewpoint image P and a distance image D are captured by a plurality of cameras C (here, nine cameras are used as an example). 1 ,C 2 ,…,C 9 A multi-view camera C is arranged in a two-dimensional space, and objects O (here, O 1 ,O 2 ,O 3 In Figure 3, camera C 1 ,C 2 ,…,C 9 However, the cameras are not limited to being parallel in the shooting directions. For example, each camera may be arranged to face a specific subject.

[0018] As shown in FIG. 4(a), one of the viewpoint images P constituting the multi-view image P is 1is the image of one camera of the multi-view camera C (e.g., C 1 ) are images taken by individual cameras C 1 ,C 2 ,…,C 9 The image captured by the camera is the same as that shown in Figure 4(a), except for the viewpoint position.

[0019] As shown in FIG. 4(b), the viewpoint image P 1 The corresponding range image D 1 is a method for matching images captured by adjacent cameras, and calculating pixel values ​​corresponding to the disparity according to the distance between corresponding pixels in the viewpoint image P 1 Here, the distance image D 1 The closer to the viewpoint, the whiter the object is, and the farther from the viewpoint, the blacker the object is. In this way, a method for generating a distance image from images captured by multiple cameras can use a known method disclosed in, for example, JP 2019-184308 A. Note that, although the distance image is generated from images captured by the cameras here, it may also be acquired by a distance image sensor that measures distance from the round trip time of a projected laser.

[0020] Returning to FIG. 1, the configuration of the multi-viewpoint image encoding device 1 will be described. As shown in Figure 1, the multi-viewpoint image encoding device 1 includes a threshold setting means 10, a viewpoint image layering means 11, a code amount control information setting means 12, a layer-by-layer image encoding means 13, a distance image encoding means 14, and a multiplexing means 15.

[0021] The threshold setting means 10 sets a threshold for hierarchically dividing (layering) the depth of the distance image in the viewpoint image layering means 11. The threshold is a depth value indicating the boundary of a depth section obtained by dividing the distance image D by depth. This threshold setting means 10 sets a predetermined viewpoint position as a minimum depth value, a farthest position to be displayed as a maximum depth value, and sets a depth value that is the boundary of the section therebetween. The threshold setting means 10 sets (m-1) thresholds zp1 ,zp 2 ,…,zp m-1 is input from outside. The depth sections do not necessarily have to be equally spaced, and may be set unevenly for each section for which it is desired to vary the compression efficiency. The number m of depth sections may be at least 2 or more.

[0022] As a result, as shown in FIG. 5, the maximum depth value zfar is reduced to the threshold value zp 1 Section L up to 1 (section length zlen 1 ), threshold zp 1 From the threshold zp 2 Section L up to 2 (section length zlen 2 ), threshold zp 2 From the threshold zp 3 Section L up to 3 (section length zlen 3 ), threshold zp 3 From the threshold zp 4 Section L up to 4 (section length zlen 4 ), threshold zp 4 to the minimum depth value znear 5 (section length zlen 5 5 shows an example in which the number of depth sections m is set to "5" and the minimum depth value and the maximum depth value are equally divided into 5. For example, when the multi-viewpoint images to be encoded are used as an integral type element image group, it is preferable to set this threshold value so as to separate the depth before and after the depth near the lens array, which makes it possible to change the compression efficiency at least between the image near the lens array and the other images. The setting of the threshold when the multi-viewpoint image P is used as a group of elemental images in the integral method will be described later. The threshold value setting means 10 sets the input threshold value (zp 1 ,…,zp m-1 ) to the viewpoint image layering means 11.

[0023] The viewpoint image layering means 11 generates layered viewpoint images for each predetermined depth range from the viewpoint images constituting the multi-viewpoint image based on a distance image corresponding to the multi-viewpoint image. Here, the viewpoint image layering means 11 includes a plurality of layering means 110 for each viewpoint image. That is, the viewpoint image layering means 11 includes n layering means 110 (110) which is the same number as the number n of the viewpoint images. 1 ,110 2 ,…,110 n )

[0024] The layering means 110 extracts images corresponding to each depth section (layer) from the viewpoint images constituting the multi-viewpoint image, and layers the viewpoint images. Layering means 110 1 is one of the viewpoint images P that composes the multi-view image P. 1 The viewpoint image P of the distance image D 1 The corresponding range image D 1 The layering is performed based on the depth range (layer), and the layer-by-layer viewpoint image P 1 L 1 ,P 1 L 2 ,…,P 1 L m Generate. Layering means 110 1 is the generated layered viewpoint image P 1 L 1 ,P 1 L 2 ,…,P 1 L m to the layer-by-layer image encoding means 13.

[0025] Layering means 110 2 ~110 n is a viewpoint image P that constitutes a multi-view image P. 2 ~P n The viewpoint image P 2 ~P n The corresponding range image D 2 ~D n The layering means 110 performs the processes other than the layering based on1 is the same as: Here, the internal configuration of the layering means 110 will be described with reference to Fig. 2 (and Fig. 1 as appropriate). Fig. 2 is a block diagram showing the internal configuration of the viewpoint image layering means 11. In this case, the layering means 110 1 The layering means 110 will be described as an example. 2 ~110 n The same applies to.

[0026] Layering means 110 1 The image processing apparatus includes a region dividing means 111 and a region image generating means 112. The region division means 111 divides the input viewpoint image P 1 The present invention generates area information that divides the area. The area division means 111 divides a distance image (a viewpoint distance image) D 1 The set of pixels is used as region information indicating a region corresponding to the depth section. Here, the region division means 111 generates region information for each depth section as mask data. Specifically, the region division means 111 divides the region information for each depth section into a distance image D 1 Mask data is generated by setting the pixel value of a pixel included in the depth section to "1" and other pixel values ​​to "0." Note that the threshold value that separates the two sections is included in one of the sections.

[0027] For example, the object O shown in FIG. 1 ,O 2 ,O 3 The distance image for the distance θ is shown in Fig. 4(b). 1 As shown in FIG. 1 ,O 2 ,O 3 But in section L 2 ,L 3 ,L 4 Assume that the distances are in the respective depth ranges. In this case, the area division means 111 divides the area L 1 For the distance image D1 At the threshold zp 1 As described above, mask data (not shown) is generated by setting pixel values ​​equal to or less than the maximum depth value zfar to "1" and other pixel values ​​to "0."

[0028] In addition, the area division means 111 divides the area L 2 For the distance image D 1 At the threshold zp 2 Above threshold zp 1 The pixel values ​​less than 1 are set to "1" and the other pixel values ​​are set to "0", and the mask data M 2 Generate. In addition, the area division means 111 divides the area L 3 For the distance image D 1 At the threshold zp 3 Above threshold zp 2 The pixel values ​​less than 1 are set to "1" and the other pixel values ​​are set to "0", and the mask data M 3 Generate.

[0029] In addition, the area division means 111 divides the area L 4 For the distance image D 1 At the threshold zp 4 Above threshold zp 3 The pixel values ​​less than 1 are set to "1" and the other pixel values ​​are set to "0", and the mask data M 4 Generate. In addition, the area division means 111 divides the area L 5 For the distance image D 1 In this case, the minimum depth value znear is greater than or equal to the threshold zp 4 Mask data (not shown) is generated by setting the pixel values ​​of pixels in a depth range less than this to "1" and other pixel values ​​to "0."

[0030] As a result, the region division means 111 divides the distance image D 1 The region can be divided into layers of depth. The area division means 111 outputs the generated area information (mask data) to the area image generation means 112.

[0031] The area image generating means 112 generates layer-specific viewpoint images from the viewpoint images based on the area information generated by the area dividing means 111 for each depth section. Here, the region image generating means 112 is divided into a plurality of region image generating means 112 corresponding to the number m of the depth sections for each depth section. 1 ,112 2 ,…,112 m It is composed of:

[0032] Area image generation means 112 1 is the viewpoint image P 1 Among these, the section L divided by the area dividing means 111 1 The image of the area is layered as a viewpoint image P 1 L 1 The result is: Area image generation means 112 2 is the viewpoint image P 1 Among these, the section L divided by the area dividing means 111 2 The image of the area is layered as a viewpoint image P 1 L 2 The result is: Similarly, the area image generating means 112 m is the viewpoint image P 1 Among these, the section L divided by the area dividing means 111 m The image of the area is layered as a viewpoint image P 1 L m The result is:

[0033] Area image generation means 112 1 ,112 2 ,…,112 m is the viewpoint image P 1 The same processing is performed in that a layer-specific viewpoint image corresponding to an area in a predetermined depth range is generated from the image. Specifically, the area image generating means 112 1 ,112 2 ,…,112 m are the viewpoint images P 1 By multiplying the mask data, which is the area information output from the area division means 111, by the depth, the viewpoint image P1 Layered viewpoint image P 1 L 1 ,P 1 L 2 ,…,P 1 L m Generate.

[0034] For example, the viewpoint image P shown in FIG. 1 and distance image D 1 In this case, as shown in FIG. 1 ,O 2 ,O 3 But in section L 2 ,L 3 ,L 4 Assume that the distances are in the respective depth ranges. In this case, the area image generating means 112 2 The layered viewpoint image P 1 L 2 , Section L 3 The layered viewpoint image P shown in FIG. 1 L 3 , Section L 4 The layered viewpoint image P shown in FIG. 1 L 4 The viewpoint image P 1 It is generated by extracting from the section L 1 and interval L 5 The layer-specific viewpoint images are omitted in the illustration. Returning to FIG. 1, the description of the configuration of the multi-viewpoint image encoding device 1 will be continued.

[0035] The code amount control information setting means 12 sets, in the layer-by-layer image encoding means 13, parameters (code amount control parameters) for controlling the code amount of an image for each depth section (layer). The code amount control parameter is information that specifies a quantization step that controls the code amount when encoding an image. For example, a quantization parameter (QP) used in an encoding method such as HEVC can be used as the parameter. When the value of this quantization parameter is set to a large value, the quantization step becomes large and the code amount becomes small. On the other hand, when the value of the quantization parameter is set to a small value, the quantization step becomes small and the code amount becomes large. Here, a quantization parameter is used as the code amount control parameter, but a quantization step corresponding to the quantization parameter may also be used.

[0036] The code amount control information setting means 12 inputs an individual quantization parameter QP from the outside for each depth section. The number of quantization parameters QP is the same as the number m of sections divided by the threshold set by the threshold setting means 10. That is, the code amount control information setting means 12 inputs a quantization parameter (QP 1 ,QP 2 ,…,QP m ) is input from outside. This quantization parameter makes it possible to control to which depth section of a multi-view image a large amount of code is to be allocated. The code amount control information setting means 12 sets the quantization parameter (QP 1 ,…,QP m ) to the layer-by-layer image encoding means 13.

[0037] Here, with reference to Fig. 8, a description will be given of quantization parameters when a group of integral type elemental images is used as a multi-viewpoint image to be encoded by the multi-viewpoint image encoding device 1. Fig. 8 shows spatial frequency characteristics with the horizontal axis representing the distance from the position of the lens array as a reference position O, and the vertical axis representing spatial frequency (cycle / rad=cpr). Note that the lens array is a two-dimensional arrangement of lenses (elemental lenses) corresponding to each elemental image in an integral type stereoscopic image display device. Here, the position of the observer M is defined as znear, the minimum depth value of the nearest position, zfar, and zp 1 ,…,zp 4 The minimum depth value znear and the maximum depth value zfar are divided into five intervals (interval L 1 ~L 5 ) is the threshold value that divides the image into the range zfar and zfar. Note that the maximum depth value zfar is assumed to be finite, and an image at infinity is assumed to exist at the position zfar.

[0038] As shown in FIG. 8, in the integral method, the spatial frequency becomes highest at a reference position O which is the lens array surface. Therefore, the quantization parameter input to the multi-viewpoint image encoding device 1 is set to have a smaller value in the section closer to the reference position O. In the example of FIG. 3 By setting the quantization parameter value of L to be smaller than that of other sections, the section L including the lens array is 3 A large amount of code can be allocated to Returning to FIG. 1, the description of the configuration of the multi-viewpoint image encoding device 1 will be continued.

[0039] The layered image encoding means 13 encodes a plurality of layered viewpoint images having the same depth range based on a code amount control parameter preset for each depth range to generate a layered image encoded stream for each depth range. The layered image encoding means 13 includes a plurality of encoding means 130 for each depth range (layer).

[0040] The encoding means 130 encodes a plurality of layer-specific viewpoint images corresponding to the same depth section based on the code amount control parameters (quantization parameters) set by the code amount control information setting means 12 . Specifically, the encoding means 130 performs an orthogonal transform on the layer-specific viewpoint images for each block of a predetermined size, and converts the images into frequency components. This orthogonal transform is, for example, a discrete cosine transform (DCT). The encoding means 130 then quantizes the transform coefficients, which are frequency components, with a quantization step specified by a quantization parameter set by the code amount control information setting means 12. The encoding means 130 then performs variable-length coding on the quantized transform coefficients (quantization coefficients) to generate stream data (layer-specific image coding stream). Note that the encoding means 130 adds the quantization parameter to the layer-specific image coding stream as auxiliary information. The encoding means 130 may use any encoding method as long as the encoding method is capable of controlling the amount of code. It is preferable that the encoding means 130 uses a multi-viewpoint image encoding method that encodes a plurality of layered viewpoint images as images that are continuous in time series. This is because the amount of code can be reduced by using the multi-viewpoint image encoding method, since a plurality of layered viewpoint images corresponding to the same depth section are highly correlated. For example, the encoding means 130 uses an encoding method such as HEVC (MV-HEVC).

[0041] Here, the encoding means 130 includes m encoding means 130, the number of which is equal to the number of sections (number of layers) m in the depth section. 1 ,130 2 ,…,130 m It is composed of: Encoding means 130 1 is a set of layered viewpoint images generated by the viewpoint image layering means 11, which are arranged in the same depth range (L 1 ) layered viewpoint image P 1 L 1 ,P 2 L 1 ,…,P n L 1 and a layered image coding stream PL 1 Generate. Encoding means 130 2 is a set of layered viewpoint images generated by the viewpoint image layering means 11, which are arranged in the same depth range (L 2 ) layered viewpoint image P 1 L2 ,P 2 L 2 ,…,P n L 2 and a layered image coding stream PL 2 Here, as shown in FIG. 9, the encoding means 130 (130 2 ) is a set of similar layered viewpoint images P 1 L 2 ,P 2 L 2 ,…,P n L 2 As a result, the encoding means 130 (130 2 ) is a layered image coding stream PL 2 can be generated.

[0042] Similarly, the encoding means 130 m is a set of layered viewpoint images generated by the viewpoint image layering means 11, which are arranged in the same depth range (L m ) layered viewpoint image P 1 L m ,P 2 L m ,…,P n L m and a layered image coding stream PL m Generate. This allows the layer-by-layer image encoding means 13 to perform encoding on the multi-viewpoint image P by controlling the amount of code for each depth section (layer). The layered image encoding means 13 encodes the generated layered image encoded stream PL 1 ,PL 2 ,…,PL m is output to the multiplexing means 15.

[0043] The distance image encoding means 14 encodes the distance image D. The distance image encoding means 14 encodes the viewpoint images P 1 ,P 2 ,…,P n The corresponding range image D1 ,D 2 ,…,D n are encoded as successive images in a time series. The distance image encoding means 14 may use any encoding method, but may use, for example, the same encoding method as the encoding means 130, such as HEVC. Distance image encoding means 14 outputs stream data resulting from encoding the distance image (distance image encoded stream) to multiplexing means 15.

[0044] The multiplexing means 15 multiplexes the plurality of layered image encoded streams generated by the layered image encoding means 13 and the distance image encoded stream generated by the distance image encoding means 14 . For example, the multiplexing means 15 may be a predetermined encoding means 130. 1 ,…,130 m The multiplexing means 15 concatenates the generated layered image coded streams in the order of the first to third layers, or in the order in which the coding is completed. Furthermore, the multiplexing means 15 concatenates the distance image coded stream before or after the concatenated layered image coded stream.

[0045] Furthermore, the multiplexing means 15 generates stream configuration information as header information, such as the stream configuration (e.g., the order of stream arrangement, the number of depth sections), the size of the viewpoint image and distance image (number of horizontal pixels, number of vertical pixels), and the data length of each of the layered image encoding stream and the distance image encoding stream.

[0046] Then, the multiplexing means 15 adds header information to the stream obtained by concatenating the layered image coded stream and the distance image coded stream, thereby generating the coded stream S. Note that, when the multi-view images are moving images, the multiplexing means 15 continuously codes the multi-view images to generate the coded stream S. The multiplexing means 15 stores the data (encoded stream S) obtained by encoding the multi-viewpoint images in a storage device (not shown) or outputs the data to the outside via a network.

[0047] With the above-described configuration, the multi-viewpoint image encoding device 1 can encode multi-viewpoint images with different encoding efficiency for each depth section. This allows the multi-viewpoint image encoding device 1 to suppress degradation of image quality at a desired depth more than degradation of image quality at other depths, thereby generating encoded data with improved compression efficiency. The multi-viewpoint image encoding device 1 can be operated by a program (multi-viewpoint image encoding program) for causing a computer to function as each of the means described above.

[0048] <Operation of the multi-viewpoint image encoding device> Next, the operation of the multi-viewpoint image encoding device 1 according to the embodiment of the present invention will be described with reference to FIG. 10 (for the configuration, refer to FIG. 1 and FIG. 2 as appropriate).

[0049] In step S1, the threshold setting means 10 sets a threshold for hierarchically dividing (layering) the depth of the distance image. Here, the threshold setting means 10 sets a predetermined viewpoint position as a minimum depth value, a farthest position to be displayed as a maximum depth value, and inputs a depth value that is a boundary of the depth section between them from outside. In step S2, the code amount control information setting means 12 sets a code amount control parameter for each depth section specified by the threshold value set in step S1. Here, the code amount control information setting means 12 inputs a quantization parameter (QP) from outside as the code amount control parameter. The order of steps S1 and S2 may be reversed.

[0050] In step S3, the layering means 110 of the viewpoint image layering means 11 generates region information in which the region of the distance image is divided for each depth section specified by the threshold value set in step S1 by the region dividing means 111. Here, the region dividing means 111 generates, for each depth section, a set of pixels having a corresponding depth value in the distance image as region information (mask data) indicating the region corresponding to the depth section.

[0051] In step S4, the layering means 110 generates layered viewpoint images from the viewpoint images based on the region information generated in step S3 for each depth section by the region image generation means 112. Here, the layering means 110 generates layered viewpoint images from the viewpoint images based on the region information generated in step S3 for each depth section by the region image generation means 112. 1 ,…,112 m By multiplying the viewpoint image by the mask data for each depth section, a layer-specific viewpoint image for each depth section is generated.

[0052] In step S5, the layer-specific image encoding means 13 encodes the layer-specific viewpoint images for each depth section generated in step S4 as chronologically consecutive images for the same depth section (layer) based on the quantization parameters for each depth section set in step S2. Here, the layer-by-layer image encoding means 13 includes a plurality of encoding means 130 1 ,…,130 m A layered image coding stream is generated for each depth section (layer) by the above.

[0053] In step S6, the distance image encoding means 14 encodes a plurality of distance images corresponding to a plurality of viewpoint images constituting a multi-viewpoint image to generate an encoded distance image stream. In step S7, the multiplexing means 15 multiplexes the plurality of layered image coded streams generated in step S5 and the distance image coded stream generated in step S6 to generate a coded stream.

[0054] Here, if the next image (multi-view image, distance image) is further input (Yes in step S8), the multi-view image encoding device 1 returns to step S3 to continue the operation. On the other hand, if the input has ended (No in step S8), the multi-viewpoint image encoding device 1 ends the operation. Through the above operations, the multi-viewpoint image encoding device 1 can encode multi-viewpoint images with different encoding efficiency for each depth section.

[0055] <Configuration of a multi-viewpoint image decoding device> Next, with reference to FIG. 11, a configuration of a multi-view image decoding device 2 according to an embodiment of the present invention will be described. The multi-viewpoint image decoding device 2 decodes the encoded stream generated by the multi-viewpoint image encoding device 1 (FIG. 1). In other words, the multi-viewpoint image decoding device 2 decodes an encoded stream generated by multiplexing layer-specific image encoding streams for each depth range, in which the viewpoint images constituting a multi-viewpoint image are encoded for each of multiple layer-specific viewpoint images having the same depth range. As shown in FIG. 11, the multi-viewpoint image decoding device 2 includes a demultiplexing means 20, a layer-specific image decoding means 21, a view image synthesis means 22, and a distance image decoding means .

[0056] The demultiplexing means 20 separates the input coded stream into a plurality of layer-specific image coded streams for each depth section, and a distance image coded stream. The demultiplexing means 20 refers to header information of the coded stream, and separates and extracts a plurality of layered image coded streams and a distance image coded stream from the coded stream. The demultiplexing means 20 demultiplexes the layer-by-layer image coding stream PL 1 ,PL 2 ,…,PL m to the layer-by-layer image decoding means 21. Moreover, demultiplexing means 20 outputs the demultiplexed encoded distance image stream to distance image decoding means 23 .

[0057] The layer-by-layer image decoding means 21 decodes the plurality of layer-by-layer image encoded streams separated by the demultiplexing means 20. The layer-by-layer image decoding means 211 includes decoding means 210 for a predetermined number of depth sections.

[0058] The decoding means 210 decodes each layered image coding stream and generates layered viewpoint images for the viewpoint images in the same depth section. The decoding means 210 performs decoding processing using a decoding method corresponding to the coding method of the coding means 130 described in FIG. Specifically, the decoding means 210 performs variable-length decoding on the layered image coding stream to generate quantization coefficients. Then, the decoding means 210 multiplies the quantization coefficients by a size of a quantization step specified by a quantization parameter attached to the layered image coding stream as attached information, and performs inverse orthogonal transform (inverse DCT) to sequentially decode layered viewpoint images. Here, the layer-by-layer image decoding means 21 is made up of m pieces of decoding means 210, the number of which is the same as the number of sections (the number of layers) m in the depth section. 1 ,210 2 ,…,210 m It is composed of:

[0059] Decryption means 210 1 is a layered image coding stream separated by the demultiplexing means 20, the depth interval L 1 Layered image coding stream PL 1 , multiple layered viewpoint images P 1 L 1 ,P 2 L 1 ,…,P n L 1 Decrypt it to: Decryption means 210 2 is a layered image coding stream separated by the demultiplexing means 20, the depth interval L 2 Layered image coding stream PL 2 , multiple layered viewpoint images P 1 L 2 ,P 2 L 2 ,…,P n L 2 As shown in FIG. 2 ) is a layered image coding stream PL 2 Then, the layered viewpoint images P 1 L2 ,P 2 L 2 ,…,P n L 2 are decoded in sequence.

[0060] Similarly, the decoding means 210 m is a layered image coding stream separated by the demultiplexing means 20, the depth interval L m Layered image coding stream PL m , multiple layered viewpoint images P 1 L m ,P 2 L m ,…,P n L m Decrypt it to: Decryption means 210 (210 1 ,…,210 5 ) output the layer-specific viewpoint images that they have generated to the viewpoint image synthesis means 22.

[0061] The viewpoint image synthesis means 22 synthesizes layer-specific viewpoint images corresponding to the same viewpoint among the layer-specific viewpoint images decoded by the layer-specific image decoding means 21, to generate a viewpoint image. Here, the viewpoint image synthesis means 22 includes a plurality of synthesis means 220 corresponding to the number of viewpoints. That is, the viewpoint image synthesis means 22 includes n synthesis means 220 (220) which is the same number as the number n of viewpoint images. 1 ,220 2 ,…,220 n )

[0062] Synthesis means 220 1 is a layer-by-layer viewpoint image P of the same viewpoint among the layer-by-layer viewpoint images decoded by the layer-by-layer image decoding means 21. 1 L 1 ,P 1 L 2 ,…,P 1 L m By synthesizing these, the viewpoint image P 1 Generate. Synthesis means 220 2is a layer-by-layer viewpoint image P of the same viewpoint among the layer-by-layer viewpoint images decoded by the layer-by-layer image decoding means 21. 2 L 1 ,P 2 L 2 ,…,P 2 L m By synthesizing these, the viewpoint image P 2 Generate. Synthesis means 220 n is a layer-by-layer viewpoint image P of the same viewpoint among the layer-by-layer viewpoint images decoded by the layer-by-layer image decoding means 21. n L 1 ,P n L 2 ,…,P n L m By synthesizing these, the viewpoint image P n Generate. The synthesizing means 220 generates a viewpoint image by adding layered viewpoint images of the same viewpoint.

[0063] The distance image decoding means 23 decodes the distance image encoded stream separated by the demultiplexing means 20. The distance image decoding means 23 performs decoding processing using a decoding method corresponding to the encoding method of the distance image encoding means 14 described in FIG.

[0064] With the above-described configuration, the multi-viewpoint image decoding device 2 can decode an encoded stream in which the multi-viewpoint image encoding device 1 encodes the multi-viewpoint image with different quantization parameters for each depth section. This allows the multi-viewpoint image decoding device 2 to increase the resolution of a stereoscopic image reproduced at a depth to which a large amount of code is allocated. The multi-viewpoint image decoding device 2 can be operated by a program (multi-viewpoint image decoding program) for causing a computer to function as each of the means described above.

[0065] <Operation of the multi-viewpoint image decoding device> Next, the operation of the multi-view image decoding device 2 according to the embodiment of the present invention will be described with reference to FIG. 13 (for the configuration, refer to FIG. 11 as appropriate).

[0066] In step S10, the demultiplexing means 20 separates the coded stream into a plurality of layered image coded streams and a distance image coded stream. In step S11, the layer-specific image decoding means 21 decodes the multiple layer-specific image encoding streams separated in step S10 to generate layer-specific viewpoint images for each depth section. Here, the layer-specific image decoding means 21 decodes the multiple decoding means 210. 1 ,…,210 m Each layered image coding stream is decoded by In step S12, the viewpoint image synthesis means 22 synthesizes the layered viewpoint images corresponding to the same viewpoint among the plurality of layered viewpoint images generated in step S11 to generate a viewpoint image. Here, the viewpoint image synthesis means 220 synthesizes the layered viewpoint images corresponding to the same viewpoint among the plurality of layered viewpoint images generated in step S11 to generate a viewpoint image. 1 ,…,220 n By adding the layer-specific viewpoint images corresponding to the same viewpoint, individual viewpoint images are generated, whereby the viewpoint images constituting the pre-encoding multi-view image are reproduced.

[0067] In step S13, the distance image decoding means 23 decodes the encoded distance image stream separated in step S10, thereby reproducing the pre-encoded distance image. Here, if an encoded stream is further input (No in step S14), the multi-viewpoint image decoding device 2 returns to step S10 and continues the operation. On the other hand, if the input of the encoded stream has finished (Yes in step S14), the multi-viewpoint image decoding device 2 ends the operation. Through the above operations, the multi-viewpoint image decoding device 2 can decode multi-viewpoint images that have been coded by changing the code amount for each depth section.

[0068] <Modification> The above describes the configuration and operation of the multi-viewpoint image encoding device 1 and the configuration and operation of the multi-viewpoint image decoding device 2 according to an embodiment of the present invention, but the present invention is not limited to this embodiment.

[0069] (Variation 1) Here, the threshold setting means 10 of the multi-viewpoint image encoding device 1 sets the threshold by being specified from the outside. However, these thresholds may be preset in an internal memory or the like of the multi-viewpoint image encoding device 1. In this case, the multi-viewpoint image encoding device 1 may omit the threshold setting means 10 from its configuration. In this case, the region dividing means 111 may divide the regions of the distance image by the preset thresholds.

[0070] (Variation 2) Here, the viewpoint image layering means 11 of the multi-viewpoint image encoding device 1 is a plurality of layering means 110 (110 1 ,110 2 ,…,110 n ) are configured to operate in parallel. Also, the layer-by-layer image encoding means 13 is configured to operate in parallel by a plurality of encoding means 130 (130 1 ,130 2 ,…,130 m ) and are configured to operate in parallel. However, the layering means 110 and the encoding means 130 do not necessarily have to be configured to operate in parallel, and may be configured as a single unit and operate in sequence.

[0071] In addition, here, the layer-by-layer image decoding means 21 of the multi-viewpoint image decoding device 2 is divided into a plurality of decoding means 210 (210 1 ,210 2 ,…,210 m ) and are configured to operate in parallel. In addition, the viewpoint image synthesis means 22 of the multi-viewpoint image decoding device 2 is a plurality of synthesis means 220 (220 1 ,220 2 ,…,220 n ) and are configured to operate in parallel. However, the decoding means 210 and the synthesis means 220 do not necessarily have to be configured to operate in parallel, and may be configured as a single unit and operate in sequence.

[0072] (Variation 3) Here, the distance image encoding means 14 of the multi-viewpoint image encoding device 1 encodes the viewpoint images P 1 ,P 2 ,…,P n The corresponding range image D 1 ,D 2 ,…,D n The above was coded as a series of images in a time series. However, the distance image encoding means 14 encodes the distance image D 1 ,D 2 ,…,D n The distance image D constructed as above may be encoded as a single image. In that case, the distance image decoding means 23 of the corresponding multi-viewpoint image decoding device 2 may decode the encoded distance image stream separated by the demultiplexing means 20 as one image.

[0073] (Variation 4) Here, the multi-viewpoint image encoding device 1 includes a distance image encoding means 14, and the multi-viewpoint image decoding device 2 includes a distance image decoding means 23. However, the distance image encoding means 14 may be configured as a distance image encoding device separated from the multi-viewpoint image encoding device 1. Also, the distance image decoding means 23 may be configured as a distance image decoding device separated from the multi-viewpoint image decoding device 2. [Explanation of symbols]

[0074] 1. Multi-view image coding device 10. Threshold setting means 11 Viewpoint image layering method 110 Layering Methods 111 Area segmentation means 112 Area image generation means 12. Code amount control information setting means 13 Layered image coding method 130 Encoding means 14 Distance image encoding means 15 Multiplexing means 2. Multi-view image decoding device 20 Demultiplexing means 21 Layer-by-layer image decoding means 210 Decoding means 22 Viewpoint image synthesis means 220 Synthesis means 23 Distance image decoding means

Claims

1. A multi-viewpoint image encoding device that encodes a multi-viewpoint image, a viewpoint image layering means for generating layered viewpoint images for each preset depth section from viewpoint images constituting the multi-viewpoint image based on a distance image corresponding to the multi-viewpoint image; a layer-by-layer image encoding means for encoding the plurality of layer-by-layer viewpoint images having the same depth range based on a code amount control parameter preset for each depth range to generate a layer-by-layer image encoded stream for each depth range; a multiplexing means for multiplexing the layered image coded stream for each depth section to generate a coded stream; A multi-viewpoint image encoding device comprising:

2. The multi-viewpoint image encoding device according to claim 1, characterized in that the layered image encoding means encodes the multiple layered viewpoint images having the same depth range as consecutive images in a time series using a multi-viewpoint image encoding method.

3. The multi-viewpoint image is a group of integral type element images, The multi-viewpoint image encoding device according to claim 1 or claim 2, characterized in that the code amount control parameter is a value set so as to allocate a larger code amount to an image in a depth section that includes the position of the integral type lens array than to other depth sections.

4. The viewpoint image layering means a region dividing means for generating mask data for dividing a region of the viewpoint image for each of the depth sections; a region image generating means for generating the layer-specific viewpoint image by multiplying the viewpoint image by the mask data for each depth section; The multi-viewpoint image encoding device according to claim 1 , further comprising:

5. A multi-viewpoint image encoding program for causing a computer to function as the multi-viewpoint image encoding device according to any one of claims 1 to 4.

6. A multi-viewpoint image decoding device that decodes an encoded stream generated by multiplexing layered image encoding streams for each depth section, the layered image encoding streams being generated by encoding a plurality of layered viewpoint images having the same depth section, comprising: a demultiplexing means for separating the coded stream into a plurality of layer-specific coded image streams for each depth section; A layered image decoding means for decoding the layered image coding stream into a plurality of layered viewpoint images in the same depth section; a viewpoint image synthesis means for synthesizing the layer-specific viewpoint images corresponding to the same viewpoint among the layer-specific viewpoint images to generate the viewpoint image; A multi-viewpoint image decoding device comprising:

7. The layered image coding stream is a stream coded by a multi-viewpoint image coding method that codes a plurality of layered viewpoint images having the same depth section as consecutive images in a time series, The multi-view image decoding device according to claim 6 , wherein the layer-by-layer image decoding means decodes the layer-by-layer image coding stream by a decoding method corresponding to the multi-view image coding method.

8. A multi-viewpoint image decoding program for causing a computer to function as the multi-viewpoint image decoding device according to claim 6 or 7.

Images