Video information processing device, method and program

Abstract

When generating a depth map from a video image, an embodiment of the present invention generates corrected second depth map information by performing each of a first filtering process for using separately generated segment information to perform edge-preserving smoothing on first depth map information generated for each of a plurality of frames constituting the video image, and a second filtering process for smoothing, between the plurality of frames and in the time direction, the pixel values of the pixels corresponding to in-frame position coordinates in the first depth map information.

Description

[Technical field]

[0001] One aspect of the present invention relates to a video information processing device, method, and program used for generating, for example, three-dimensional moving images. [Background technology]

[0002] One method for generating 3D images is the depth map method, which maps distance information from the viewpoint using a gradation representation with the far and near positions at both ends. A method has also been proposed to generate a more accurate depth map by combining the depth information of this depth map with segmentation information that divides the image of an object in an image frame into multiple regions in the two-dimensional direction.

[0003] However, when this depth map generation method is applied to video information such as moving images, the depth map is generated independently for each frame without considering the correlation between frames. As a result, the gradation of the depth direction of an object in an image changes for each frame, and when the generated depth map is viewed as a moving image, the object appears to fluctuate in the depth direction, resulting in an unnatural moving image.

[0004] In view of this, a method has been proposed in which smoothing processing in the time direction is performed while preserving the edges of an object for frames that are consecutive in the time direction, for example by using a motion-compensated temporal filter (see, for example, Patent Document 1). This method enables smoothing of moving images in the time direction, for example by dividing an image of an object into a plurality of pixel blocks for each frame and performing motion prediction processing of the image of the object for each pixel block. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2009-55146 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the method described in Patent Document 1 performs motion prediction processing of a moving image of an object for each of a plurality of divided pixel blocks, which imposes a very high processing load on the device and is therefore unsuitable for practical use.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables smoothing in the time direction while preserving edges of objects in moving images, with a small processing load. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of a video information processing device or a video information processing method according to the present invention is adapted to generate a depth map from a moving image by acquiring first depth map information generated for each of a plurality of frames constituting the moving image, and acquiring segmentation information generated by dividing an image region including an object into a plurality of pixel blocks for each of the plurality of frames constituting the moving image, and then performing a first filtering process for edge-preserving smoothing on the first depth map information using the segmentation information as a guide image for each of the plurality of frames, and performing a second filtering process for smoothing in the time direction pixel values ​​of pixels whose positions in the first depth map information correspond between the plurality of frames, thereby generating corrected second depth map information.

[0009] According to one aspect of the present invention, a first filter process for performing edge-preserving smoothing is performed on the first depth map information generated for each of a plurality of frames, and a second filter process for smoothing pixel values ​​of pixels corresponding to the first depth map information in the time direction between the frames is performed. Therefore, the second filter process reduces fluctuations in the time direction of the first depth map information between the frames, and even if the edge portion of the object image becomes unclear due to the second filter process, the edge portion is sharpened by the first filter process. Therefore, it is possible to generate depth map information in which the fluctuation of inter-frame correlation is reduced while suppressing blurring of the edge portion of the object image. Moreover, by combining the smoothing process in the time direction and the edge-preserving smoothing process, the above-mentioned sharpening of the edge portion and the reduction of the fluctuation in the time direction are realized, so that a motion prediction process for each pixel block is not required, and the above-mentioned effect can be obtained with a small processing load. Effect of the Invention

[0010] That is, according to one aspect of the present invention, it is possible to provide a technique that enables smoothing in the time direction while preserving edges of objects in a video image with a small processing load. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of a video information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a software configuration of the video information processing device according to the embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing a more detailed configuration of the smoothing processing unit shown in FIG. [Figure 4] FIG. 4 is a diagram showing a first example of a frame used in the smoothing process by the temporal filter. [Diagram 5]FIG. 5 is a flowchart showing the procedure and contents of the depth map generating process executed by the control unit of the video information processing device shown in FIG. [Figure 6] FIG. 6 is a flowchart showing in more detail the procedure and contents of the smoothing process of the processing procedure shown in FIG. [Figure 7] FIG. 7 is a diagram showing a second example of a frame used in the smoothing process by the temporal filter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One embodiment] (Configuration example) A video information processing device according to an embodiment of the present invention has a function of generating a depth map for generating parallax images in a display system that displays three-dimensional moving images.

[0014] 1 and 2 are block diagrams showing an example of a hardware configuration and a software configuration, respectively, of a video information processing device 1 according to an embodiment of the present invention.

[0015] The video information processing device 1 is, for example, a general-purpose personal computer, and includes a control unit 10 using a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 20 and a data storage unit 30, and an input / output I / F unit 40 are connected to the control unit 10 via a bus 50.

[0016] The control unit 10 may include a graphics processing unit (GPU) in addition to the CPU. A communication I / F unit for communicating with an external device via a network may be connected to the control unit 10. Furthermore, the video information processing device 1 may be an ASIC (Application Specific Integrated Circuit) for image processing, or may be a server device located on the Web or cloud in some cases.

[0017] The input / output I / F unit 40 is connected to the external devices, namely, the moving image generating device 2 and the moving image display device 3. When the moving image generating device 2 and the moving image display device 3 are installed at a distance, the moving image generating device 2 and the moving image display device 3 may be connected to a communication I / F unit of the video information processing device 1.

[0018] The moving image generating device 2 includes, for example, a camera, and generates and outputs moving images. The moving image display device 3 includes a display device or a projector using a liquid crystal or an organic EL, and generates a 3D moving image including parallax images using the depth map generated by the video information processing device 1, and displays it on the display device.

[0019] The program storage unit 20 is configured, for example, by combining a non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores various programs necessary for executing various control processes according to one embodiment of the present invention, in addition to middleware such as an OS (Operating System).

[0020] The data storage unit 30 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and is provided with an RGB image storage unit 31 and a depth map storage unit 32 as main storage areas required to implement one embodiment of the present invention.

[0021] The RGB image storage unit 31 is used to sequentially store RGB images of each frame of the moving image output from the moving image generating device 2. The depth map storage unit 32 is used as a video buffer, and temporarily stores depth maps for multiple frames that the control unit 10 uses for the time-direction smoothing process of the depth map by a temporal filter.

[0022] Here, the number of frames stored in the depth map storage unit 32 is set according to the number of taps of the temporal filter. In general, the number of taps of the temporal filter can be set arbitrarily depending on the buffer amount of the depth map storage unit 32 and the requirements of the processing delay of the entire system, but is set to about 5 taps, for example. Therefore, in this case, the number of frames stored in the depth map storage unit 32 is set to 5 frames as shown in FIG.

[0023] In addition, the frames stored in the depth map memory unit 32 are not limited to past frames Fp for the frame F0 to be processed as illustrated in FIG. 4, but may include, for example, multiple past frames Fp of the frame F0 to be processed and multiple future frames Ff added thereto as illustrated in FIG. 7.

[0024] In addition, the data storage unit 30 also has a memory area for temporarily storing depth maps and segmentation results generated during a series of processing steps by the control unit 10, as well as an area for storing various thresholds used in the smoothing process.

[0025] The control unit 10 includes, as processing functions according to an embodiment of the present invention, an RGB image acquisition processing unit 11, a depth estimation processing unit 12, a segmentation processing unit 13, a size change processing unit 14, and a smoothing processing unit 15. Each of these processing units 11 to 15 is realized by causing a processor such as a CPU and a GPU of the control unit 10 to execute an application program stored in a program storage unit 20.

[0026] The RGB image acquisition processing unit 11 receives RGB images of each frame constituting the moving image output from the moving image generating device 2 via the input / output I / F unit 40 , and performs processing to store the images in the RGB image storage unit 31 .

[0027] The depth estimation processing unit 12 reads an RGB image for each frame from the RGB image storage unit 31, estimates a depth map from the read RGB image, and outputs it. The depth map is image data that expresses the depth of each pixel with 256 gray levels, for example, from 0 to 255. For example, the gray level is set to 0 at the back and 255 at the front, but a gray level other than 256 may be used. For example, a method called Depth from Videos in the Wild is used to estimate the depth map.

[0028] The segmentation processing unit 13 reads the RGB image from the RGB image storage unit 31 for each frame, detects objects such as animals from within the read RGB image, and outputs segmentation information in which an image region, for example a rectangle, including the detected object is divided into a plurality of blocks in units of pixels. The segmentation information is made up of data in which a segment ID is assigned to each block for each divided pixel. For example, a method called Mask R-CNN can be used for the segmentation processing.

[0029] The resizing processing unit 14 inputs the depth map and the segmentation information from the depth estimation processing unit 12 and the segmentation processing unit 13, respectively. Then, the resizing processing unit 14 resizes the depth map and the segmentation information so that they are the same size, and outputs the resized depth map and segmentation information.

[0030] The smoothing processing unit 15 inputs the depth map and segmentation information whose size has been changed by the resizing processing unit 14 for each frame. The smoothing processing unit 15 then performs a two-dimensional smoothing process on the input depth map using an edge-preserving smoothing filter, and also performs a time-direction smoothing process using a temporal filter and depth maps of other frames stored in the depth map storage unit 32, and outputs a depth map corrected by these smoothing processes. Note that an example of the smoothing process on the depth map using the edge-preserving smoothing filter and the temporal filter will be described in detail in an operation example.

[0031] (Example of operation) Next, a description will be given of an example of the operation of the video information processing device 1 configured as above. Fig. 5 is a flowchart showing the overall processing procedure and processing contents by the control unit 10 of the video information processing device 1.

[0032] (1) Acquiring RGB images In step S10, the control unit 10 of the video information processing device 1 monitors whether an RGB image is input. In this state, when RGB images of a plurality of frames constituting a moving image are input from the moving image generating device 2, the control unit 10 of the video information processing device 1, under the control of the RGB image acquisition processing unit 11, imports the RGB images of each frame via the input / output I / F unit 40 in step S11, and stores them in the RGB image storage unit 31 sequentially. The RGB image acquisition processing section 11 may perform a process of separating and extracting an RGB image for each frame from the input moving image.

[0033] (2) Depth estimation When the RGB image is input, in step S12, the control unit 10 of the video information processing device 1 reads the RGB image from the RGB image storage unit 31 for each frame under the control of the depth estimation processing unit 12, performs depth estimation on the read RGB image, generates a depth map DMin, and outputs it to the size change processing unit 14. As described above, the depth map is image data that represents the depth of each pixel of the RGB image in 256 shades of gray from 0 to 255, for example.

[0034] (3) Generation of segmentation information In parallel with the depth map estimation process, the control unit 10 of the video information processing device 1 performs segmentation processing on the RGB image in step S13 under the control of the segmentation processing unit 13. For example, the segmentation processing unit 13 first detects all objects such as moving objects from within the RGB image. Then, for each detected object, the segmentation processing unit 13 divides, for example, a rectangular image area including the object into a plurality of pixel blocks in pixel units, and assigns a segment ID to each divided pixel block. For example, when the image is divided into nine pixel blocks, segment IDs 1 to 9 are assigned to these pixel blocks. Then, the segmentation processing unit 13 outputs segmentation information SG including the segment ID to the resizing processing unit 14 for each frame.

[0035] (4) Size change Next, in step S14, the control unit 10 of the video information processing device 1, under the control of the size change processing unit 14, performs a process of changing the size of the depth map DMin and segmentation information SG outputted respectively from the depth estimation processing unit 12 and the segmentation processing unit 13 so that the frame sizes are the same.

[0036] Generally, depth estimation and segmentation are often performed using images that are reduced from the original RGB image. Using a reduced RGB image reduces the processing cost of depth map estimation and segmentation, and shortens each processing time, which ultimately shortens the processing time of the entire system.

[0037] In order to cope with the case where the depth map DMin and the segmentation information SG have different sizes due to the effect of the reduction process, the size change processing unit 14 changes the sizes of the depth map DMin and the segmentation information SG to, for example, the same size as the original RGB image. Note that, if the depth map DMin and the segmentation information SG have the same size, the size change process is omitted.

[0038] The resizing unit 14 outputs the resizing processed depth map DMin and the segmentation information SG to the smoothing unit 15. In step S15, the resizing unit 14 stores the resizing processed depth map DMin in the depth map storage unit 32 for use in a time direction smoothing process by a temporal filter, which will be described later.

[0039] (5) Smoothing Next, under the control of the smoothing processor 15, the controller 10 of the video information processing device 1 performs smoothing processing on the depth map output from the resizing processor 14 in step S16 as follows.

[0040] FIG. 3 is a block diagram showing an example of the functional configuration of the smoothing processing unit 15, and FIG. 6 is a flowchart showing an example of the processing procedure and processing contents of the smoothing processing by the smoothing processing unit 15.

[0041] The smoothing processing unit 15 includes, as its processing functions, an edge-preserving smoothing filter 151, a temporal filter 152, and a filter accuracy determination unit 153. All of these processing functions 151 to 153 are realized by causing a processor such as a CPU or a GPU to execute a program.

[0042] (5-1) First smoothing process First, in step S20, the smoothing processing unit 15 performs a filtering process for edge-preserving smoothing on the input resized depth map DMin using the segmentation information SG of the same frame as a guide by the edge-preserving smoothing filter 151. For this edge-preserving smoothing process, for example, a Joint Bilateral Filter or a Guided Filter is used, but other filters can also be used.

[0043] If the edge-preserving smoothing filter 151 is performing the filtering process on the depth map DMin for the first time, the edge-preserving smoothing filter 151 transfers the depth map DM1 after the filtering process to the temporal filter 152 via the filter accuracy determination unit 153. At this time, the filter accuracy determination unit 153 temporarily stores the depth map DM1 after the edge-preserving smoothing process in a buffer area of ​​the data storage unit 30.

[0044] Next, in step S21, the smoothing processing unit 15 uses the temporal filter 152 to smooth the pixel values ​​in the time direction for each pixel whose coordinate position in the frame corresponds to the depth map DM1 after the edge-preserving smoothing processing, using depth maps of multiple past frames stored in the depth map memory unit 32.

[0045] For example, if a frame F0 at time t is to be processed as shown in FIG. 4, the depth map of this frame F0 is smoothed in the time direction for each pixel whose position coordinates in the frame correspond to each other using depth maps of four frames Fp at past times t-1, t-2, t-3, and t-4. For example, a low-pass filter is used for this smoothing process. The depth map DM2 that has been smoothed is returned from the temporal filter 152 to the filter accuracy determination unit 153.

[0046] Next, in step S22, the smoothing processing unit 15, under the control of the filter accuracy determination unit 153, calculates the sum of absolute differences DM3 between the depth map DM2 output from the temporal filter 152 and the depth map DM1 output from the edge-preserving smoothing filter 151 and before being supplied to the temporal filter 152.

[0047] In step S23, the filter accuracy determination unit 153 compares the calculated sum of absolute differences DM3 with a threshold value TH1 previously stored in a threshold value storage area of ​​the data storage unit 30, and determines whether the sum of absolute differences DM3 is equal to or smaller than the threshold value TH1. If the sum of absolute differences DM3 is equal to or smaller than the threshold value TH1, in step S26, the depth map DM2 output from the temporal filter 152 is output as a corrected depth map DMout.

[0048] At the same time, in step S27, the smoothing processing unit 15 outputs the corrected depth map DMout to the depth map storage unit 32, and updates the depth map Din of the corresponding frame F0 that has been stored up to that point to the corrected depth map DMout.

[0049] (5-2) Repeated smoothing process On the other hand, when smoothing processing is performed by the temporal filter 152, the image of the edge portion of the object may become blurred or unclear as if it were hazy. In this case, the sum of absolute difference values ​​DM3 does not become equal to or less than the threshold value TH1.

[0050] Therefore, if the result of the determination in step S23 above is that the sum of absolute differences DM3 is not equal to or less than the threshold value TH1, the filter accuracy determination unit 153 performs control to limit the repeated execution process described below in steps S24 and S25, and then passes the depth map DM2 output from the temporal filter 152 to the edge-preserving smoothing filter 151 to perform edge-preserving smoothing processing again.

[0051] In step S20, the edge-preserving smoothing filter 151 executes edge-preserving smoothing processing on the depth map DM2. That is, a second edge-preserving smoothing processing is performed here. Then, the filter accuracy determination unit 153 temporarily stores the depth map DM1 that has been subjected to the second edge-preserving smoothing processing by the edge-preserving smoothing filter 151 in a buffer area of ​​the data storage unit 30, and then transfers it to the temporal filter 152.

[0052] The temporal filter 152 performs a second temporal filtering process on the depth map DM1 in step S21, and returns the depth map DM2 after the filtering process to the filter accuracy determination unit 153.

[0053] In step S22, the filter accuracy determination unit 153 calculates the sum of absolute differences DM3 between the depth map DM2 on which the second temporal filtering process has been performed and the depth map DM1 before the temporal filtering process, and determines again in step S23 whether the calculated sum of absolute differences DM3 is equal to or smaller than the threshold value TH1. If the sum of absolute differences DM3 is equal to or smaller than the threshold value TH1, in step S26, the depth map DM2 on which the second temporal filtering process has been performed is output as a corrected depth map DMout.

[0054] On the other hand, if the sum of absolute difference values ​​DM3 is not equal to or less than the threshold value TH1, the filter precision determination unit 153 returns the depth map DM2 to the edge-preserving smoothing filter 151 to perform edge-preserving smoothing processing again. Thereafter, in the same manner, the edge-preserving smoothing processing by the edge-preserving smoothing filter 151 and the smoothing processing in the time direction by the temporal filter 152 are alternately and repeatedly performed on the depth map DM2 until the sum of absolute difference values ​​DM3 becomes equal to or less than the threshold value TH1.

[0055] (5-3) Restrictions on repeated execution of processes Meanwhile, the filter accuracy determination unit 153 of the smoothing processing unit 15 limits the repeated execution process by each of the filters 151, 152 to prevent the smoothing process by the edge-preserving smoothing filter 151 and the time-direction smoothing process by the temporal filter 152 from being performed indefinitely.

[0056] That is, if the sum of absolute difference values ​​DM3 is not equal to or less than the threshold value TH1 as a result of the determination in step S23, the filter accuracy determination unit 153 counts up the number of repetitions C in step S24. Then, in step S25, it is determined whether the number of repetitions C after the count-up has reached an upper limit value TH2. As this upper limit value TH2, a value stored in advance in a threshold storage area in the data storage unit 30 is used. If the number of repetitions C has not yet reached the upper limit value TH2, the filter accuracy determination unit 153 returns the depth map DM2 to the edge-preserving smoothing process by the edge-preserving smoothing filter 151.

[0057] In contrast, if it is determined in step S25 that the number of repeated executions C after counting up has reached the upper limit TH2, the filter accuracy determination unit 153 does not repeat the smoothing process any further, and proceeds to step S26 to output the depth map DM2 as the corrected depth map DMout.

[0058] (Action and effect) As described above, in the video information processing device 1 in one embodiment, the smoothing processing unit 15 includes an edge-preserving smoothing filter 151 and a temporal filter 152, and performs edge-preserving smoothing processing by the edge-preserving smoothing filter 151 and smoothing processing in the time direction by the temporal filter 152 on a depth map DMin estimated from an RGB image for each frame.

[0059] Therefore, the temporal smoothing process by the temporal filter 152 reduces fluctuations in the depth map between frames, and even if the smoothing process by the temporal filter 152 causes blurring or haze in the image of the edge portion of the object, making the image quality unclear, the edge-preserving smoothing process by the edge-preserving smoothing filter 151 reduces the blurring or haze in the image of the edge portion, making the image quality clear. Therefore, it is possible to generate a depth map DMout that suppresses blurring or the like in the image of the edge portion of the object and reduces fluctuations in inter-frame correlation. Moreover, motion prediction process for each pixel block of the RGB image is not required, and the above-mentioned image quality improvement effect can be obtained with a small processing load.

[0060] Furthermore, the smoothing processing unit 15 is provided with a filter accuracy determination unit 153, which calculates a sum of absolute differences DM3 between the depth map DM2 output from the temporal filter 152 and the depth map DM1 after edge-preserving smoothing processing by the edge-preserving smoothing filter 151 and before input to the temporal filter 152, and repeatedly performs edge-preserving smoothing processing by the edge-preserving smoothing filter 151 and time-direction smoothing processing by the temporal filter 152 on the depth map DM2 until the calculated sum of absolute differences DM3 becomes equal to or less than a threshold value TH1.

[0061] This makes it possible to generate a high-quality depth map DMout in which the image of the edge of an object is clear with little blurring, and in which fluctuations in inter-frame correlation are sufficiently suppressed.

[0062] Furthermore, in the smoothing processing unit 15, the filter precision determination unit 153 counts the number of times C that the edge-preserving smoothing process by the edge-preserving smoothing filter 151 and the time-direction smoothing process by the temporal filter 152 are repeatedly executed, and the repeated execution process is terminated when the number of times C that the repeated execution reaches an upper limit value TH2. This makes it possible to prevent a problem in which the repeated execution process is executed endlessly.

[0063] [Other embodiments] (1) In the above embodiment, the filter accuracy determination unit 153 of the smoothing processing unit 15 compares the sum of absolute differences DM3 between the depth map DM1 output from the edge-preserving smoothing filter 151 and the depth map DM2 output from the temporal filter 152 with a threshold value TH1, and when the sum of absolute differences DM3 becomes equal to or smaller than the threshold value TH1, outputs the depth map DM2 at that time as a smoothed depth map DMout. However, the filter accuracy determination process using the sum of absolute differences DM3 does not necessarily have to be performed. For example, the smoothing process by the edge-preserving smoothing filter 151 and the smoothing process by the temporal filter 152 may be unconditionally repeated alternately a preset number of times, and the resulting depth map DM2 may be output as a corrected depth map DMout.

[0064] (2) In general, when a moving image contains a visual effect that causes a large change in the inter-frame correlation value of an object image, such as a scene change or cross-fade, a sufficient smoothing effect cannot be obtained even if a time-direction smoothing process is performed using a temporal filter. Therefore, the video information processing device 1 may receive, for example, detection information of a visual effect in a moving image from the moving image generating device 2, and may perform control so as not to perform a time-direction smoothing process using a temporal filter on a frame in which a visual effect is detected based on this detection information.

[0065] (3) In the above embodiment, the depth estimation process for generating depth map information from an input RGB image and the process for generating segmentation information of an image region including an object from the input RGB image are performed within the video information processing device 1. However, for example, if the moving image generating device 2 or another external device has a function for generating the depth map information and segmentation information, the video information processing device 1 may acquire the depth map information and segmentation information from the moving image generating device 2 or another external device.

[0066] (4) In the above embodiment, a case where a depth map is generated from an RGB image extracted for each frame from a video has been described as an example, but a depth map may be generated from a two-dimensional video image obtained by a monocular camera, or from a stereo image. Furthermore, a depth map may be generated from a monochrome image other than an RGB image.

[0067] (5) In the above embodiment, a case where a program for executing a series of processes according to the present invention is stored in advance in the program storage unit 20 of the video information processing device has been described as an example. However, in addition to this, the video information processing device may read an application program from an external storage medium, such as a magnetic disk, an optical disk, or a semiconductor memory such as a USB memory, when necessary, and have the control unit 10 execute the program, or may download an application program from a server device or the like located on the Web or in the cloud, when necessary, and have the control unit 10 execute the program.

[0068] (6) In the above embodiment, all the processing functions according to the present invention are provided in one video information processing device. However, all the processing functions according to the present invention may be distributed among a plurality of information processing devices (for example, personal computers, mobile terminals such as smartphones, and server devices).

[0069] (7) In addition, the functional configuration of the video information processing device, its processing procedures and contents, the types of moving images, and the like can be modified in various ways without departing from the spirit and scope of the present invention.

[0070] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiment may be appropriately adopted.

[0071] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0072] 1...Video information processing device 2...Video generation device 3...Moving image display device 10...Control section 11…RGB image acquisition processing section 12...Depth estimation processing unit 13...Segmentation processing section 14...Size change processing section 15...Smoothing processing section 20...Program memory section 30...Data storage unit 31…RGB image storage section 32…Depth map memory unit 40...Input / output interface section 50…Bus 151...Edge-preserving smoothing filter 152...Temporal filter 153...Filter accuracy determination unit

Claims

1. A video information processing device that generates a depth map from a moving image, comprising: a depth map information acquisition processing unit that acquires first depth map information generated for each of a plurality of frames constituting the moving image; a segmentation information acquisition processing unit that acquires segmentation information generated by dividing an image area including an object into a plurality of pixel blocks for each of the plurality of frames; a smoothing processing unit that generates corrected second depth map information by performing a first filtering process for performing edge-preserving smoothing on the first depth map information using the segmentation information as a guide image for each of the plurality of frames, and a second filtering process for smoothing pixel values ​​of pixels whose positions correspond to those of the first depth map information between the plurality of frames in a time direction; A video information processing device comprising:

2. The image information processing device according to claim 1 , wherein the smoothing processing section alternately and repeatedly executes the first filtering process and the second filtering process.

3. 3. The video information processing device according to claim 2, wherein the smoothing processing unit calculates a difference value between the first depth map information after the second filtering process has been performed and the first depth map information after the first filtering process has been performed and before the second filtering process has been performed, and alternately repeats the first filtering process and the second filtering process until the calculated difference value becomes equal to or smaller than a preset threshold value.

4. 3. The video information processing device according to claim 2, wherein the smoothing processing unit counts the number of times the first filter processing and the second filter processing are repeatedly executed, and terminates the repeated execution process of the first filter processing and the second filter processing when the counted value of the number of times the first filter processing and the second filter processing reaches a preset upper limit value.

5. 5. The video information processing device according to claim 1, wherein, when a visual effect in which a correlation value between frames exceeds a predetermined amount is detected in the moving image, the smoothing processing unit does not perform the second filtering process on the first depth map information generated for the frame in which the visual effect is detected.

6. 1. A video information processing method executed by an apparatus for generating a depth map from a video, comprising: obtaining first depth map information generated for each of a plurality of frames constituting the moving image; obtaining, for each of the plurality of frames, segmentation information generated by dividing an image region including an object into a plurality of pixel blocks; generating corrected second depth map information by performing a first filtering process for performing edge-preserving smoothing on the first depth map information using the segmentation information as a guide image for each of the plurality of frames, and a second filtering process for smoothing pixel values ​​of pixels whose positions correspond to those of the first depth map information between the plurality of frames in a time direction; A video information processing method comprising:

7. 6. A program for causing a processor included in the video information processing apparatus to execute processing by each of the processing units included in the video information processing apparatus according to claim 1.

Images