Image filtering device and program thereof, and image encoding device and program thereof
The image filtering and encoding devices utilize depth information to segment and adjust compression parameters for each depth section, enhancing encoding efficiency and maintaining image quality in multi-view displays.
Patent Information
- Application Number
- JP2021017923
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-08
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2041-02-08
AI Technical Summary
Conventional methods for encoding multi-view images and depth maps for 3D image display suffer from insufficient compression efficiency, particularly when displaying 3D images with the naked eye, as they do not effectively utilize distance information to optimize encoding.
An image filtering device and encoding device that processes images based on depth information, segmenting images into multiple depth sections and applying different compression efficiencies by adjusting parameters such as resolution and brightness gradation for each section, thereby generating and integrating images with varying compression efficiencies.
Enhances compression efficiency by reducing the code amount while maintaining image quality, especially in depth areas with high perceptual sensitivity, thus improving the encoding process for multi-view image displays.
Smart Images

Figure 0007583633000006 
Figure 0007583633000007 
Figure 0007583633000008
Abstract
Description
[Technical field]
[0001] The present invention relates to an image filtering device and a program therefor, and an image coding device and a program therefor. [Background technology]
[0002] Currently, a method is being considered for transmitting multi-viewpoint images displayed on a ray reproduction type display together with a distance image (depth map) indicating distance information from the shooting position (viewpoint position). International standards define coding methods such as 3D-AVC and 3D-HEVC that code multi-viewpoint images and distance images and generate arbitrary viewpoint images on the decoding side (see Non-Patent Documents 1 and 2). In encoding such multi-viewpoint images, improving the encoding efficiency is an issue.
[0003] In recent years, a method has been proposed for improving the coding efficiency by performing image processing to emphasize edges before coding. For example, Patent Document 1 proposes a prefilter that emphasizes edges by weighting edge regions with large changes in a moving image. Also, in order to improve the coding efficiency, a hierarchical coding method has been proposed in which coding parameters are changed stepwise and applied. For example, Patent Document 2 proposes a method of coding an image according to the characteristics of a residual signal, which is a motion vector for each macroblock. Patent Document 3 proposes a method of hierarchically coding a moving image with scalability and decoding the necessary layer at the time of decoding. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2005-191865 A [Patent Document 2] JP 2004-350072 A [Patent Document 3] JP 2007-266748 A [Non-patent literature]
[0005] [Non-Patent Document 1] Shimizu, "Trends in international standardization of 3D video coding using depth maps," IPSJ Technical Report, Vol. 2013-AVM-82, No. 11, pp. 1-6 (2013). [Non-Patent Document 2] Senoo, Y. Yamamoto, Oi, and Kurita, "Standardization Activities for MPEG Multi-viewpoint Video Coding," National Institute of Information and Communications Technology Quarterly Report, Vol. 56, Nos. 1 / 2, pp. 79-90 (2010). Summary of the Invention [Problem to be solved by the invention]
[0006] Conventional techniques perform compression encoding by referring to information such as motion vectors and edges of moving images. However, to display a naked-eye 3D image, a multi-view image composed of many images is required, and conventional methods have the problem of insufficient compression efficiency. The present invention has been made in consideration of such problems, and has an object to provide an image filtering device and a program thereof, as well as an image encoding device and a program thereof, that are capable of improving compression efficiency compared to conventional methods by referring to distance information. [Means for solving the problem]
[0007] In order to solve the above problems, the image filtering device of the present invention is an image filtering device that converts a subject image into an image whose compression efficiency can be changed according to depth based on distance information of the subject, and is configured to include a depth-specific image generation means, a depth-specific filtering means, and a region integration means.
[0008] In such a configuration, the image filtering device generates a plurality of images by extracting images corresponding to the depth ranges specified by the distance information from the subject image for each depth range set in advance by the image-by-depth generating means, whereby the image-by-depth generating means generates images separated for each depth range from the subject image.
[0009] The image filtering device uses the depth-specific filtering means to filter the gradation more roughly as the degree of compression efficiency preset for each depth section increases. As a result, the depth-specific filtering means generates a plurality of depth-specific filtered images with different degrees of compression efficiency. For example, this filtering can be performed by changing the resolution or changing the luminance gradation. Then, the image filtering device uses the region integrating means to integrate the plurality of depth-specific filtered images into one image. This allows the image filtering device to generate an image from the subject image, the compression efficiency of which can be changed according to the depth.
[0010] In addition, in order to solve the above problem, the image filtering device of the present invention is an image filtering device that converts a subject image into an image whose compression efficiency can be changed according to depth based on distance information of the subject, and is configured to include a depth-specific filtering means, a depth-specific image generation means, and a region integration means.
[0011] In such a configuration, the image filtering device generates a plurality of filtered images from the subject image by filtering the gradation more roughly as the degree of compression efficiency preset for each depth section specified by the distance information increases. In this way, the depth filtering means generates a plurality of filtered images with different degrees of compression efficiency. For example, this filtering can be performed by changing the resolution or changing the luminance gradation.
[0012] The image filtering device extracts images of depths corresponding to the depth sections from the plurality of filtered images generated for each depth section by the image-by-depth generating means, and generates a plurality of filtered images by depth. In this way, the image-by-depth generating means generates images separated for each depth section from the plurality of filtered images. Then, the image filtering device uses the region integrating means to integrate the plurality of depth-specific filtered images into one image. This allows the image filtering device to generate an image from the subject image, the compression efficiency of which can be changed according to the depth. The image filtering device can be operated by a program that causes a computer to function as each of the above-mentioned means.
[0013] In addition, in order to solve the above-mentioned problems, the image encoding device of the present invention is an image encoding device that encodes a subject image by changing the compression efficiency according to depth based on distance information of the subject, and is configured to include a depth-specific image generation means, a depth-specific filtering means, a region integration means, and an encoding means.
[0014] In such a configuration, the image encoding device generates a plurality of images by extracting, for each preset depth section, an image corresponding to the depth section specified by the distance information from the subject image, and generates a plurality of images by depth using the image by depth generating means. The image encoding device then generates a plurality of depth-filtered images from the plurality of depth-filtered images by filtering the gradations more roughly as the degree of compression efficiency preset for each depth section increases using the depth-filtering means. Then, the image encoding device uses the region merging means to merge the multiple depth-specific filtered images into one image. Furthermore, the image coding apparatus codes the image integrated by the region integration means by the coding means. This allows the image encoding device to encode the subject image with different compression efficiencies depending on the depth.
[0015] In addition, in order to solve the above-mentioned problems, the image encoding device of the present invention is an image encoding device that encodes a subject image by changing the compression efficiency according to depth based on distance information of the subject, and is configured to include a depth-specific filtering means, a depth-specific image generation means, a region integration means, and an encoding means.
[0016] In such a configuration, the image encoding device generates multiple filtered images from the subject image by using a depth-specific filtering means to filter the gradation more coarsely, the greater the degree of compression efficiency that is preset for each depth range specified by the distance information. Then, the image encoding device causes the depth-specific image generating means to extract an image of a depth corresponding to the depth section from the plurality of filtered images generated for each depth section, and generates a plurality of depth-specific filtered images. Then, the image encoding device uses the region merging means to merge the multiple depth-specific filtered images into one image. Furthermore, the image coding apparatus codes the image integrated by the region integration means by the coding means. This allows the image encoding device to encode the subject image with different compression efficiencies depending on the depth. The image coding apparatus can be operated by a program that causes a computer to function as each of the means described above. Effect of the Invention
[0017] The present invention provides the following excellent effects. According to the present invention, it is possible to convert an object image into an image with different compression efficiency for each depth section. As a result, when encoding a multi-viewpoint image displayed on a ray reproduction type display, the present invention can suppress deterioration of image quality in the depth region with high perceptual sensitivity and can encode the image with high compression efficiency. [Brief description of the drawings]
[0018] [Figure 1]FIG. 1 is a block diagram showing the configuration of an image encoding device including an image filtering means (device) according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is an explanatory diagram for explaining a method for generating a multi-viewpoint image. [Diagram 3] FIG. 2 is an explanatory diagram for explaining distance images, where (a) shows a subject image, and (b) shows a distance image corresponding to (a). [Figure 4] FIG. 11 is an explanatory diagram for explaining a threshold value of a depth section. [Diagram 5] 13(a) to 13(c) are diagrams showing mask data for each depth section. [Figure 6] 13(a) to 13(c) are diagrams showing depth-specific images for each depth section. [Figure 7] 11 is a graph showing spatial frequency characteristics of the integral method in which the depth section is divided by a threshold value. FIG. [Figure 8] 4 is a flowchart showing the operation of the image encoding device according to the first embodiment of the present invention. [Figure 9] FIG. 11 is a block diagram showing the configuration of an image encoding device including image filtering means (device) according to a second embodiment of the present invention. [Figure 10] 10 is a flowchart showing the operation of an image encoding device according to a second embodiment of the present invention. [Figure 11] FIG. 13 is a block diagram showing the configuration of an image filtering means (device) according to a modified example of the present invention. [Figure 12] FIG. 11 is an explanatory diagram for explaining a correction technique of a filtering correction means. [Figure 13] 1 is a configuration diagram showing an example of an image coding device according to an embodiment of the present invention, which is configured by an image filtering device and a coding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <First embodiment: Configuration of an image encoding device> The configuration of an image encoding device 1 according to a first embodiment of the present invention will be described with reference to FIG.
[0020] The image coding device 1 codes an image of a subject by changing the compression efficiency for each depth based on a distance image (distance information) of the subject. The image coding device 1 changes the compression efficiency for each depth section that divides the depth specified in the distance image. First, the images (object image, distance image) input to the image coding device 1 will be described, and then the configuration of the image coding device 1 will be described.
[0021] The subject image is an individual viewpoint image of a multi-view image obtained by photographing a subject. Note that the subject image does not need to be a real image, and may be, for example, a CG image obtained by virtually photographing a subject with a virtual camera. The image encoding device 1 inputs the multi-view image for each viewpoint image.
[0022] A distance image (distance information) is an image that represents the distance from the viewpoint position in the subject space where the subject is photographed, using pixel values that are assigned to each pixel in a range from a minimum depth value to a maximum depth value that is predetermined. This distance image corresponds to each viewpoint image of a multi-viewpoint image. It should be noted that the resolution of the subject image and the distance image do not necessarily need to be the same, and it is sufficient if the pixels of the subject image and the pixels of the distance image are associated in advance.
[0023] Here, examples of a subject image and a distance image will be described with reference to Fig. 2 and Fig. 3. As shown in Fig. 2, assume that a multi-view camera C in which a plurality of cameras (here, nine cameras as an example) C1, C2, ..., C9 are two-dimensionally arranged captures subjects O (here, O1, O2, O3) at different depth positions in the subject space. Note that Fig. 2 shows an example in which the cameras C1, C2, ..., C9 are arranged so that their shooting directions are parallel, but this is not limited to being parallel. For example, each camera may be arranged so as to face the direction of a specific subject. As shown in Fig. 3(a), a subject image G is an image captured by one camera (e.g., C1) of the multi-viewpoint camera C. Note that the images captured by each of the cameras C1, C2, ..., C9 are the same as those in Fig. 3(a) except for the viewpoint positions. As shown in Fig. 3(b), distance image D is created by matching images captured by adjacent cameras and assigning pixel values corresponding to the parallax according to the distance between corresponding pixels to pixel positions in subject image G. Here, distance image D displays areas closer to the viewpoint position as white and areas farther from the viewpoint position as black.
[0024] In this way, a method for generating a distance image from images captured by multiple cameras can use a known method disclosed in, for example, JP 2019-184308 A. Note that, although the distance image is generated from images captured by the cameras here, it may also be acquired by a distance image sensor that measures distance from the round trip time of a projected laser.
[0025] Returning to FIG. 1, the configuration of the image encoding device 1 will be described. As shown in FIG. 1, the image encoding device 1 includes an image filtering means 10 and an encoding means 20. The image filtering means 10 converts the subject image into an image whose compression efficiency can be changed according to depth, based on a distance image (distance information) of the subject. As shown in FIG. 1, the image filtering means 10 includes a threshold setting means 11 , a depth-specific image generating means 12 , a filter parameter setting means 13 , a depth-specific filtering means 14 , and a region integrating means 15 .
[0026] The threshold setting means 11 sets thresholds for dividing the depth of a distance image hierarchically in the depth-classified image generating means 12. The thresholds are depth values that indicate boundaries of depth sections obtained by dividing the distance image by depth. This threshold setting means 11 sets a predetermined viewpoint position as a minimum depth value, a farthest position to be displayed as a maximum depth value, and sets a depth value that is the boundary of the section therebetween. The threshold setting means 11 sets (N-1) thresholds zp1, zp2, ..., zp N-1 Here, the number of depth sections N is set to "5", and the values obtained by equally dividing the minimum depth value and the maximum depth value by 5 are set to the threshold values zp1, ..., zp4.
[0027] The depth sections do not necessarily have to be equally spaced, and may be set unevenly for each section for which it is desired to vary the compression efficiency. In addition, although the number of depth sections is described as "5" here, it is sufficient that the number is at least "2" or more. This allows the distance image to be divided into sections L1 (section length zlen1) from the maximum depth value zfar to the threshold zp1, L2 (section length zlen2) from threshold zp1 to threshold zp2, L3 (section length zlen3) from threshold zp2 to threshold zp3, L4 (section length zlen4) from threshold zp3 to threshold zp4, and L5 (section length zlen5) from threshold zp4 to the minimum depth value znear, as shown in Figure 4.
[0028] For example, when the object image (multi-viewpoint image) to be encoded is used as an integral type element image group, it is preferable to set this threshold value so as to separate the image at the depth near the lens array. This makes it possible to change the compression efficiency at least between the image near the lens array and the other images. A specific example of setting the threshold value when the multi-viewpoint images are used as an integral type elemental image group will be described later. The threshold setting means 11 outputs the input thresholds (zp1, . . . , zp4) to the depth-specific image generating means 12.
[0029] The depth-specific image generating means 12 extracts an image corresponding to a depth section specified in the distance image from the subject image for each depth section set in advance, and generates a plurality of depth-specific images. Here, the depth-specific image generating means 12 includes a region dividing means 120 and a region image generating means 121.
[0030] The area division means 120 generates area information for dividing the area of the input subject image for each depth section specified by the threshold value set by the threshold value setting means 11 . The region division means 120 regards, for each depth section, a set of pixels of the distance image having a corresponding depth value as region information indicating a region corresponding to the depth section. Here, the region division means 120 generates region information for each depth section as mask data. Specifically, the region division means 120 generates mask data for each depth section by setting the pixel values of pixels in the distance image that are included in the depth section to "1" and other pixel values to "0." Note that the threshold value dividing the two sections is included in one of the sections.
[0031] For example, suppose that the distance image for subjects O1, O2, and O3 shown in Figure 2 is distance image D shown in Figure 3(b), and subjects O1, O2, and O3 are present in the depth sections L2, L3, and L4, respectively, as shown in Figure 4. In this case, for section L1, region partitioning means 120 generates mask data (not shown) by setting pixel values in distance image D that are equal to or greater than threshold zp1 and equal to or less than maximum depth value zfar to "1" and other pixel values to "0." For section L2, region partitioning means 120 sets pixel values in distance image D that are equal to or greater than threshold zp2 and less than threshold zp1 to "1" and other pixel values to "0," generating mask data M1 shown in FIG. 5(a). For section L3, region partitioning means 120 sets pixel values in distance image D that are equal to or greater than threshold zp3 and less than threshold zp2 to "1" and other pixel values to "0," generating mask data M2 shown in FIG. 5(b).
[0032] For section L4, region partitioning means 120 sets pixel values in distance image D that are equal to or greater than threshold zp4 and less than threshold zp3 to "1" and other pixel values to "0," generating mask data M3 shown in FIG. 5(c). In addition, for section L5, the region division means 120 generates mask data (not shown) by setting the pixel values of pixels in the depth section in the distance image D that is greater than or equal to the minimum depth value znear and less than the threshold value zp4 to “1” and the other pixel values to “0”. This enables the area division means 120 to divide the area of the distance image into depth sections specified by the threshold value. The area division means 120 outputs the generated area information (mask data) to the area image generation means 121.
[0033] The region image generating means 121 generates depth-specific images from the subject image based on the region information generated by the region dividing means 120 for each depth section. Here, the region image generating means 121 is composed of a plurality of region image generating means 1211, 1212, ..., 1215 for each depth section, the number of which corresponds to the number of depth sections.
[0034] The region image generating means 1211 generates an image of the region of the section L1 divided by the region dividing means 120 from the subject image as an image by depth. The region image generating means 1212 generates an image of the region of the section L2 divided by the region dividing means 120 from the subject image as an image by depth. Similarly, the area image generating means 1213, 1214, and 1215 generate images of areas in sections L3, L4, and L5 divided by the area dividing means 120 in the captured image as images classified by depth.
[0035] The area image generating means 1211, 1212, ..., 1215 perform the same process in that they generate depth-specific images corresponding to areas in a predetermined depth range from the subject image. Specifically, the region image generating means 121 multiplies the subject image by mask data, which is region information output from the region dividing means 120, for each depth, to generate a depth-specific image for each depth section.
[0036] For example, in the object image G and distance image D shown in FIG. 3, it is assumed that objects O1, O2, and O3 exist in the depth sections L2, L3, and L4, respectively, as shown in FIG. In this case, the region image generating means 121 generates a depth-specific image G1 shown in Fig. 6(a) corresponding to the section L2, a depth-specific image G2 shown in Fig. 6(b) corresponding to the section L3, and a depth-specific image G3 shown in Fig. 6(c) corresponding to the section L3. Note that the depth-specific images for the sections L1 and L5 are not shown. The depth-specific image generating means 12 outputs the generated depth-specific images for each depth section to the depth-specific filtering means 14 .
[0037] The filter parameter setting means 13 sets filter parameters for filtering the depth-specific images in the depth-specific filtering means 14 . The filter parameters are parameters that indicate the degree of compression efficiency for each depth section, and are resolution, brightness gradation, etc. For example, the resolution is a parameter that changes the compression efficiency because a high resolution increases the amount of code and a low resolution decreases the amount of code. In addition, the brightness gradation is a parameter that changes the compression efficiency because the amount of code increases if it is high and decreases if it is low. Of the brightness (Y) and chrominance (U, V), it is preferable to control only the brightness because changing the chrominance causes color shifts.
[0038] Specifically, when the resolution is used as a filter parameter, the depth section in which the highest resolution is desired is set to "1", and thereafter, the resolution ratios are set in order of decreasing resolution, such as "1 / 2", "1 / 4", ... For example, if the resolution of the subject image (depth-specific image) is 1,920 pixels horizontally by 1,080 pixels vertically, the filter parameter "1 / 2" means that the subject image (depth-specific image) is filtered to an image equivalent to 960 pixels horizontally by 540 pixels vertically.
[0039] Furthermore, when the luminance gradation is used as a filter parameter, the depth section in which the most gradation is desired is set to "1", and the gradation ratios are set thereafter in order of decreasing gradation, such as "1 / 2", "1 / 4", ... For example, if the brightness gradation of the subject image (depth-specific image) is 256 levels (8-bit gradation in codeword length), the filter parameter "1 / 2" means that the brightness gradation of the subject image (depth-specific image) is filtered to 128 levels (equivalent to 7-bit gradation in codeword length). The filter parameter setting means 13 inputs filter parameters from the outside for each depth section, and outputs them to the depth-specific filtering means 14 .
[0040] The depth-based filtering means 14 generates a plurality of depth-based filtered images from a plurality of depth-based images by filtering the gradation more roughly as the degree of compression efficiency preset for each depth section increases. Here, the depth-based filtering means 14 includes a plurality of filtering means 140.
[0041] The filtering means 140 filters the depth-classified images based on the parameters (filter parameters) set by the filter parameter setting means 13 . Here, the filtering means 140 is made up of a plurality of filtering means 1401, 1402, . . . , 1405 according to the number of depth sections.
[0042] The filtering means 1401 filters the depth-classified images generated by the area image generating means 1211 of the depth-classified image generating means 12 so that the gradation becomes coarser as the degree of compression efficiency increases. The filtering means 1402 filters the depth-specific images generated by the area image generating means 1212 of the depth-specific image generating means 12 . Similarly, filtering means 1403 , 1404 , and 1405 filter the depth-specific images generated by the area image generating means 1213 , 1214 , and 1215 of the depth-specific image generating means 12 , respectively.
[0043] The filtering means 140 (1401, ..., 1405) performs the same processing in that it filters the depth-classified images using a predetermined filtering method. The predetermined filtering method is resolution conversion or luminance gradation conversion. Each filtering method will be described below.
[0044] (resolution conversion) When resolution conversion is used as the filtering method, the filtering means 140 performs resolution conversion on the depth-specific images in accordance with the filter parameters set by the filter parameter setting means 13 . For example, when the filter parameter is "1 / 2", the filtering means 140 converts the resolution of the depth-specific image to 1 / 2 in both the horizontal and vertical directions. Specifically, the filtering means 140 calculates the average value of pixel values for every 2 horizontal pixels by 2 vertical pixels of the depth-specific image, and sets the same average value to four pixels (2 horizontal pixels by 2 vertical pixels) at the same pixel position after filtering. In this way, when reducing the resolution to 1 / n in the horizontal and vertical directions, the filtering means 140 calculates the average value of pixel values for each n×n pixel of the depth-classified image, and sets this as the pixel value of the n×n pixel at the same pixel position after filtering. Note that for an area on the periphery of the depth-classified image that is less than n×n pixels, the filtering means 140 calculates the average value of pixel values within that area, and sets this as the pixel value at the same pixel position after filtering.
[0045] Furthermore, filtering means 140 excludes pixels in the depth-specific images for which no pixel value is set (value "0") from the pixels to be averaged, and calculates the average value only for pixels for which pixel values are set. Filtering means 140 then sets the calculated average value at the same pixel position after filtering as the pixels for which pixel values are set in the depth-specific images, and sets "0" at the same pixel position after filtering as the pixels for which no pixel value is set. However, when the filter parameter is "1", the filtering means 140 does not perform filtering on the depth-classified images. In this way, the filtering means 140 converts the resolution while keeping the number of pixels the same before and after filtering.
[0046] (Luminance gradation conversion) When luminance tone conversion is used as the filtering method, the filtering means 140 performs luminance tone conversion on the depth-specific images in accordance with the filter parameters set by the filter parameter setting means 13 . For example, when the filter parameter is "1 / 2", the filtering means 140 converts the luminance gradation of the depth-specific image to 1 / 2. Specifically, when the luminance gradation of the depth-specific image is 256 gradations (8-bit codeword length), the filtering means 140 converts the luminance value to 128 gradations (corresponding to 7-bit codeword length). In this way, when the luminance gradation is reduced to 1 / n, the filtering means 140 calculates the converted luminance I' by I'=floor(I / n)×n+(n / 2), where I is the luminance of the depth-specific image. Note that floor(x) is a floor function that calculates the integer part of x. This allows the luminance value to be set in stages according to the gradation.
[0047] Returning to FIG. 1, the description of the configuration of the image encoding device 1 will be continued. The filtering means 140 (1401, . . . , 1405) outputs images filtered for each depth section (depth-specific filtered images) to the region integrating means 15.
[0048] The region integrating means 15 integrates the depth-specific filtered images generated by the depth-specific filtering means 14 and reconstructs them into one image. The region integration means 15 extracts effective pixels for each depth from the depth-specific filtered images, and generates a single integrated image as a whole. Note that, in the depth-specific filtered images, pixels whose pixel value is other than "0" are effective pixels. By configuring the image filtering means 10 in this manner, the image filtering means 10 filters the subject image by depth, and generates an image with different resolution or brightness gradation by depth. This enables the image filtering means 10 to generate an image in which the amount of code can be reduced according to the depth. The image filtering means 10 (region integration means 15 ) outputs the generated image (integrated image) to the encoding means 20 .
[0049] The encoding means 20 encodes the image (integrated image) generated by the image filtering means 10. A general encoding device may be used as the encoding means 20. For example, the encoding means 20 may be an encoding device using an encoding method such as HEVC. The encoding means 20 stores data obtained by encoding the integrated image (encoded data) in a storage device (not shown) or outputs the data to the outside via a network.
[0050] By configuring the image coding device 1 as described above, the image coding device 1 can code the object image by changing the coding efficiency for each depth section. This enables the image coding device 1 to suppress degradation of image quality at a desired depth more than degradation of image quality at other depths, thereby generating coded data with improved compression efficiency. The image encoding device 1 can be operated by a program (image encoding program) for causing a computer to function as each of the means described above.
[0051] [Specific examples of integral type filter parameters] Next, with reference to Fig. 7, a specific example of filter parameters when an integral type element image group is used as a subject image (multi-viewpoint image) to be encoded by the image encoding device 1 will be described. Fig. 7 shows spatial frequency characteristics with the horizontal axis representing the distance from the position of the lens array to the reference position O, and the vertical axis representing the spatial frequency (cycle / rad=cpr). Note that the lens array is a two-dimensional arrangement of lenses (element lenses) corresponding to each element image in an integral type stereoscopic display device.
[0052] Here, the position of the observer M is defined as the minimum depth value znear, the maximum depth value zfar, and zp1, ..., zp8 are thresholds that divide the minimum depth value znear and the maximum depth value zfar into nine equal intervals (intervals L1 to L9). Note that the maximum depth value zfar is finite, and an image at infinity is assumed to exist at the position of zfar.
[0053] As shown in FIG. 7, in the integral method, the spatial frequency becomes highest at a reference position O which is the lens array surface. Therefore, the filter parameters input to the image encoding device 1 are set so that the closer to the reference position O, the higher the resolution or the more luminance gradations are provided. Conversely, the farther from the reference position O, the lower the resolution or the fewer luminance gradations are provided. For example, if the object image and distance image are 1920 pixels horizontally and 1080 pixels vertically, and the resolution is 8-bit gradation, the pixel values of the distance image, which are depth values, will be in the range of [0, 255]. Specific examples of depth intervals (depth value ranges) and resolutions (ratios) in this case are shown in Table 1 below.
[0054] [Table 1]
[0055] For example, in section L5 including the lens array surface in Fig. 7, the resolution is set to the original image (1920 x 1080 pixels) (ratio "1"). Also, in sections L1 and L9, which are the furthest from the lens array surface, the resolution is set to 30 x 17 pixels (ratio "1 / 64").
[0056] Moreover, specific examples of depth intervals (depth value ranges) and luminance gradations (ratios) are shown in Table 2 below.
[0057] [Table 2]
[0058] For example, in section L5 including the lens array surface in Fig. 7, the luminance gradation is left as it is (ratio "1") in the original image (256 gradations). Also, in sections L1 and L9, which are the furthest from the lens array surface, the luminance gradation is set to 16 gradations (ratio "1 / 16"). As described above, by setting the threshold value of the depth section based on the position of the lens array, it is possible to suppress deterioration in image quality in the vicinity of the lens array and reduce the amount of code.
[0059] <First embodiment: Operation of image encoding device> Next, the operation of the image encoding device 1 according to the first embodiment of the present invention will be described with reference to FIG. 8 (for the configuration, refer to FIG. 1 as appropriate).
[0060] In step S1, the threshold setting means 11 sets a threshold for the depth section of the subject image. Here, the threshold setting means 11 sets a predetermined viewpoint position as a minimum depth value, a farthest position to be displayed as a maximum depth value, and inputs a depth value that is a boundary of the depth section between them from outside.
[0061] In step S2, the filter parameter setting means 13 sets a filter parameter for each depth section identified by the threshold value set in step S1. Here, the filter parameter setting means 13 inputs, as the filter parameter, a parameter that specifies the resolution or brightness gradation for each depth section from outside. The order of steps S1 and S2 may be reversed.
[0062] In step S3, the depth-specific image generating means 12 generates region information by dividing the region of the distance image into depth sections specified by the threshold value set in step S1 by the region dividing means 120. Here, the region dividing means 120 generates, for each depth section, a set of pixels having corresponding depth values in the distance image as region information (mask data) indicating the region corresponding to the depth section.
[0063] In step S4, the depth-specific image generating means 12 generates depth-specific images by extracting, for each depth section, a subject image corresponding to the region information generated in step S3 using the region image generating means 121. Here, the depth-specific image generating means 12 generates depth-specific images for each depth section by multiplying the subject image by mask data for each depth section using a plurality of region image generating means 1211, ..., 1215.
[0064] In step S5, the depth-specific filtering means 14 filters the depth-specific images for each depth section generated in step S4, based on the filter parameters set in step S2. Here, the depth-based filtering means 14 performs predetermined filtering (resolution conversion or luminance tone conversion) on the depth-based images using a plurality of filtering means 1401, . . . , 1405, to generate depth-based filtered images.
[0065] In step S6, the region integration means 15 integrates the depth-specific filtered images for each depth section generated in step S5, and reconstructs the images into one integrated image. In step S7, the encoding means 20 encodes the integrated image generated in step S6. Here, if the next image (subject image, distance image) is further input (Yes in step S8), the image encoding device 1 returns to step S3 to continue the operation. On the other hand, if the input has ended (No in step S8), the image encoding device 1 ends the operation. Through the above operations, the image encoding device 1 can encode the object image with different encoding efficiency for each depth section specified in the distance image.
[0066] <Second embodiment: Configuration of an image encoding device> Next, the configuration of an image encoding device 1B according to a second embodiment of the present invention will be described with reference to FIG.
[0067] The image coding device 1B codes an object image by changing the compression efficiency according to the depth based on a distance image (distance information) of the object. Note that the difference from the image coding device 1 described in Fig. 1 is that the image coding device 1 performs filtering after generating depth-specific images from the object image, whereas the image coding device 1B filters the object image first.
[0068] As shown in Fig. 9, the image coding device 1B includes an image filtering means 10B and a coding means 20. The coding means 20 has the same configuration as the image coding device 1 described in Fig. 1, and therefore a description thereof will be omitted.
[0069] The image filtering means 10B converts the subject image into an image whose compression efficiency can be changed according to depth, based on the subject distance image (distance information). 9, the image filtering means 10B includes a threshold setting means 11, a depth-specific image generating means 12B, a filter parameter setting means 13, a depth-specific filtering means 14B, and a region integrating means 15. Since the components other than the depth-specific image generating means 12B and the depth-specific filtering means 14B have the same configuration as the image encoding device 1 described in FIG. 1, a description thereof will be omitted. The depth-based image generating means 12B and the depth-based filtering means 14B will be described below. First, the depth-based filtering means 14B, which operates first, will be described.
[0070] The depth-based filtering means 14B generates a plurality of filtered images from the subject image by filtering the gradation more roughly as the degree of compression efficiency preset for each depth section specified by the distance information increases. Here, the depth-based filtering means 14B includes a plurality of filtering means 140B.
[0071] The filtering means 140B filters the subject image based on the parameters (filter parameters) set by the filter parameter setting means 13. Here, the filtering means 140B is made up of a plurality of filtering means 140B1, 140B2, ..., 140B5 corresponding to the number of depth sections. Each of the filtering means 140B1, 140B2, . . . , 140B5 performs the same processing as the filtering means 1403, 1404, 1405 described with reference to FIG. 1, except that the filtering means 140B1, 140B2, . The filtering means 140B (140B1, . . . , 140B5) outputs images (filtered images) filtered with filter parameters corresponding to the respective depth intervals to the depth-specific image generating means 12B.
[0072] The depth-specific image generating means 12B extracts an image of a depth corresponding to a depth section from the plurality of filtered images generated for each depth section by the depth-specific filtering means 14B, and generates a plurality of depth-specific filtered images. Here, the depth-specific image generating means 12B includes a region dividing means 120 and a region image generating means 121B. The region dividing means 120 has the same configuration as the image encoding device 1 described in FIG.
[0073] The region image generating means 121B generates a depth-specific filtered image from the filtered image generated by the depth-specific filtering means 14B based on the region information generated by the region dividing means 120 for each depth section. Here, the region image generating means 121 is configured with a plurality of region image generating means 121B1, 121B2, ..., 121B5 for each depth section, the number of which corresponds to the number of depth sections. Each of the regional image generating means 121B1, 121B2, ..., 121B5 performs the same processing as the regional image generating means 1211, 1212, ..., 121 described in FIG. 1, except for generating an image corresponding to the depth (depth-based filtered image) from the filtered image. The depth-specific image generating means 12B outputs the generated depth-specific filtered images for each depth section to the region integrating means 15.
[0074] By configuring the image coding device 1B as described above, the image coding device 1B can code the object image by changing the coding efficiency for each depth section. This enables the image coding device 1 to suppress degradation of image quality at a desired depth compared to degradation of image quality at other depths, thereby generating coded data with improved compression efficiency. Furthermore, since the image encoding device 1B performs filtering on the entire subject image using the depth-specific filtering means 14B, when the depth-specific image generating means 12B extracts depth-specific images, ringing at the depth boundary is suppressed, thereby preventing a decrease in encoding efficiency. The image encoding device 1B can be operated by a program (image encoding program) for causing a computer to function as each of the means described above.
[0075] <Second embodiment: Operation of image encoding device> Next, the operation of the image encoding device 1B according to the second embodiment of the present invention will be described with reference to FIG. 10 (for the configuration, refer to FIG. 9 as appropriate). The operations in steps S1 and S2 are the same as those in the image encoding device 1 described with reference to FIG.
[0076] In step S3B, the depth-based filtering means 14B filters the subject image with the filter parameters for each depth section set in step S2. Here, the depth-based filtering means 14B performs predetermined filtering (resolution conversion or luminance gradation conversion) on the same subject image using a plurality of filtering means 140B1, . . . , 140B5 to generate a plurality of filtered images.
[0077] In step S4B, depth-specific image generating means 12B generates region information in which the region of the distance image is divided for each depth section specified by the threshold value set in step S1 by region dividing means 120. Here, region dividing means 120 generates, for each depth section, a set of pixels having corresponding depth values in the distance image as region information (mask data) indicating the region corresponding to the depth section.
[0078] In step S5B, the depth-specific image generating means 12B extracts a filtered image corresponding to the region information generated in step S4B for each depth section by the region image generating means 121B to generate a depth-specific image. Here, the depth-specific image generating means 12B generates a depth-specific filtered image for each depth section by multiplying the filtered image by mask data for each depth section by a plurality of region image generating means 121B1, ..., 121B5. The operations in steps S6 to S8 are the same as those of the image encoding device 1 described with reference to FIG. Through the above operations, image encoding device 1B can encode the object image with different encoding efficiency for each depth section specified in the distance image.
[0079] <Modification> Although the configurations and operations of the image encoding devices 1, 1B according to the embodiments of the present invention have been described above, the present invention is not limited to these embodiments.
[0080] (Variation 1) Here, the image encoding device 1B generates a plurality of filtered images from the subject image by the depth-based filtering means 14B. However, since the filtered image is generated with filter parameters corresponding to discrete depth intervals, it is preferable to correct the filtered image according to the actual depth.
[0081] Specifically, the image filtering means 10B of the image encoding device 1B may be replaced with image filtering means 10C shown in FIG. 11, the image filtering means 10C can be configured by adding a filtering correction means 16 to the image filtering means 10B. The other configurations are the same as those of the image filtering means 10B, and therefore the description thereof will be omitted. The filtering correction means 16 corrects the filtered image by proportionally adding (blending) the filtered images generated by the depth-specific filtering means 14B corresponding to adjacent depth sections based on the representative values of the depth sections.
[0082] Here, a specific example of the correction method in the filtering correction means 16 will be described with reference to FIG. Figure 12 shows the threshold (zp n-1 ,zp n ,zp n+1 ,zp n+2 ) indicates a part of the depth range specified by zp n-1 and zp n The length of the section between n , zp n and zp n+1 The length of the section between n+1 , zp n+1 and zp n+2 The length of the section between n+2 Let us assume that. Here, the representative value of the depth of each section is set as the section center point, and that value (section representative value) is c n ,c n+1 ,c n+2 Let us assume that.
[0083] The filtering correction means 16 performs correction on each of the filtered images generated by the multiple filtering means 140B. Here, the depth interval [zpn ,zp n+1 ] will be described as an example in which a filtered image generated using filter parameters corresponding to the above-mentioned [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, The filtering correction means 16 is n-1 ,zp n ] interval mean value c n , depth interval [zp n ,zp n+1 ] interval mean value c n+1 is calculated using the following formula (1).
[0084]
number
[0085] In addition, when the encoding target is a group of elemental images in the integral method, the infinity is included in the depth of the lens array, and the section length becomes extremely long. Therefore, in such a depth section including infinity, the middle position is not used as the section representative value, but a position at a predetermined small distance from the adjacent depth section is used as the section representative value.
[0086] Then, the filtering correction means 16 calculates the distance information d0 of the pixel of the filtered image to be corrected and the section representative value c n ,c n+1 The ratio R1:R2 of the distances between the two points is expressed as the section length L n ,L n+1 The calculation is performed by normalizing the above.
[0087]
number
[0088] Assuming that the pixel value (luminance value) of the distance information d0 is affected in inverse proportion to the section length, the filtering correction means 16 calculates a corrected filtered image I(d0) for each pixel using the following equation (3).
[0089]
number
[0090] Here, I(c n ),I C n+1 ) are the respective c n ,c n+1 A filtered image generated by the filtering means 140B corresponding to a depth section including the above-mentioned, and proportional addition is performed for each corresponding pixel. This enables the filtering correction means 16 to correct the filtered image to more accurate pixel values (luminance values).
[0091] (Variation 2) Here, the image encoding devices 1 and 1B are configured to include image filtering means 10, 10B, and 10C and encoding means 20. However, in the present invention, the image filtering means 10, 10B, 10C and the encoding means 20 may be configured separately from each other. That is, the present invention may be configured with an image filtering device 2 and an encoding device 3 as shown in FIG.
[0092] The image filtering device 2 has the same configuration as the image filtering means 10, 10B, and 10C. Also, the image filtering device 2 can be operated by a program (image filtering program) for making a computer function as each of the image filtering means 10, 10B, and 10C. The encoding device 3 is identical to the encoding means 20 . In this way, by configuring the image filtering device 2 and the encoding device 3 separately, the present invention makes it possible to easily change the encoding method to a different existing encoding method. Furthermore, the image filtering device 2 can generate images whose compression efficiency can be changed according to the depth specified in the distance image, regardless of the encoding method used by the encoding device 3.
[0093] (Variation 3) Here, the threshold setting means 11 of the image encoding devices 1 and 1B sets the threshold by being specified from outside. However, these thresholds may be preset in an internal memory or the like of the image coding device 1, 1B. In this case, the image coding device 1, 1B may omit the threshold setting means 11 from its configuration. In this case, the region dividing means 120 may divide the regions of the distance image by the preset thresholds. These thresholds may also be calculated by the threshold setting means 11 from the distance image. For example, the threshold setting means 11 inputs the distance image and generates the frequency of the same distance as a distance distribution. Then, the threshold setting means 11 detects a predetermined number of peaks from the peak with the highest frequency, and sets the distance of the valley corresponding to the peak as the threshold. In this case, the threshold setting means 11 may set filter parameters for the filter parameter setting means 13 so that the compression efficiency is increased for depth sections with lower frequencies. In this way, the threshold setting means 11 can prevent one subject from being divided into different depth sections by the threshold.
[0094] (Variation 4) Here, the depth-specific image generating means 12, 12B of the image encoding devices 1, 1B are configured to operate in parallel using a plurality of regional image generating means 121 (1211, 1212, ..., 1215), 121B (121B1, 121B2, ..., 121B5). Further, the depth-based filtering means 14, 14B are configured to be operated in parallel by a plurality of filtering means 140 (1401, 1402, ..., 1405), 140B (140B1, 140B2, ..., 140B5). However, the regional image generating means 121, 121B and the filtering means 140, 140B do not necessarily have to be configured to operate in parallel, and may be configured as a single means to operate in sequence. [Explanation of symbols]
[0095] 1 Image encoding device 10 Image Filtering Methods 11 Threshold setting means 12 Depth-specific image generation method 120 Area segmentation means 121 Area image generation means 13 Filter parameter setting means 14 Depth-specific filtering methods 140 Filtering Methods 15 Domain integration means 16 Filtering correction means 20 Encoding means 2. Image Filtering Device 3 Encoding device
Claims
1. An image filtering device that converts an object image into an image capable of changing compression efficiency according to depth based on distance information of the object, a depth-classified image generating means for extracting, for each preset depth section, an image corresponding to the depth section specified by the distance information from the subject image, and generating a plurality of depth-classified images; a depth-based filtering means for filtering gradations more roughly as a degree of compression efficiency preset for each depth section increases, thereby generating a plurality of depth-based filtered images from the plurality of depth-based images; a region merging means for merging the plurality of depth-specific filtered images into one image; An image filtering device comprising:
2. An image filtering device that converts an object image into an image capable of changing compression efficiency according to depth based on distance information of the object, a depth-based filtering means for generating a plurality of filtered images from the subject image by filtering the gradation more roughly as a degree of compression efficiency preset for each depth section specified by the distance information increases; a depth-specific image generating means for extracting an image of a depth corresponding to a depth section from the plurality of filtered images generated for each depth section, and generating a plurality of depth-specific filtered images; a region merging means for merging the plurality of depth-specific filtered images into one image; An image filtering device comprising:
3. The image filtering device according to claim 2, further comprising a filtering correction means for correcting the filtered images generated by the depth-specific filtering means corresponding to adjacent depth sections by proportional addition based on a representative value of the depth section.
4. The subject image is an individual viewpoint image constituting a multi-viewpoint image for generating an integral type elemental image group, 4. The image filtering device according to claim 1, wherein the compression efficiency is increased as the section is farther away from a depth section including the position of the integral type lens array.
5. 5. The image filtering device according to claim 1, wherein the filtering in said depth-based filtering means is a resolution conversion.
6. 5. The image filtering device according to claim 1, wherein the filtering in the depth-based filtering means is a luminance gradation conversion.
7. An image filtering program for causing a computer to function as the image filtering device according to any one of claims 1 to 6.
8. An image encoding device that encodes an object image by changing a compression efficiency according to depth based on distance information of the object, a depth-classified image generating means for extracting, for each preset depth section, an image corresponding to the depth section specified by the distance information from the subject image, and generating a plurality of depth-classified images; a depth-based filtering means for filtering gradations more roughly as a degree of compression efficiency preset for each depth section increases, thereby generating a plurality of depth-based filtered images from the plurality of depth-based images; a region merging means for merging the plurality of depth-specific filtered images into one image; encoding means for encoding the image integrated by the region integration means; An image encoding device comprising:
9. An image encoding device that encodes an object image by changing a compression efficiency according to depth based on distance information of the object, a depth-based filtering means for generating a plurality of filtered images from the subject image by filtering the gradation more roughly as a degree of compression efficiency preset for each depth section specified by the distance information increases; a depth-specific image generating means for extracting an image of a depth corresponding to a depth section from the plurality of filtered images generated for each depth section, and generating a plurality of depth-specific filtered images; a region merging means for merging the plurality of depth-specific filtered images into one image; encoding means for encoding the image integrated by the region integration means; An image encoding device comprising:
10. An image coding program for causing a computer to function as the image coding device according to claim 8 or 9.
Citation Information
Patent Citations
Image organizing device, image organizing method, image recorder, image reproducing device, image recording and reproducing device, image pickup device and recording medium readable by computer in which data is recorded
JP2000244810A
Image coder, image coding method, image coding program and recording medium with the program recorded thereon
JP2004350072A
Image processing apparatus, image processing program and image processing method
JP2005191865A
Encoding method
JP2007266748A
Distance image coding device and program of the same, and distance image decoding device and program of the same
JP2020120322A
