Image processing device and image processing method

Abstract

A video processing device of an embodiment of the present invention generates, from a raw image, a stereoscopic image to be presented to a plurality of users. The video processing device is a computer equipped with a processor. The processor: discretely divides an assumed viewpoint position of a tracking user, who is the primary viewer of a stereoscopic image; acquires an actual viewpoint position of the tracking user; generates, from viewpoint images capturing an object included in a raw image from a plurality of viewpoint positions, left and right parallax induction patterns based on a viewpoint image form the actual viewpoint position; and generates stereo pair images which include an image obtained by adding the parallax induction patterns to a reference image to be presented, and an image obtained by subtracting the parallax induction patterns from the reference image.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to techniques for generating stereoscopic images. [Background technology]

[0002] In recent years, there has been much research into the generation of stereoscopic images, also known as stereoscopic videos. For example, a gaze-tracking naked-eye three-dimensional (3D) display is known (see Non-Patent Document 1). This technology tracks the positions of both eyes of a recognized user's face, including the depth direction, and presents stereoscopic images using lenticular or parallax barriers, optimized to match the positions of both eyes, thereby presenting high-resolution stereoscopic (3D) images.

[0003] Normally, naked-eye 3D displays using lenticular parallax barrier technology split the space into multiple viewpoints, resulting in a lower resolution due to the number of viewpoints. In contrast, viewpoint-tracking 3D displays replace pixels in real time using only the viewpoints of one user's left and right eyes, allowing for the presentation of high-resolution images.

[0004] However, the images presented by a viewpoint-tracking naked-eye 3D display are optimized only for the tracked user (hereafter referred to as the tracking user), who is the primary viewer of the stereoscopic image. Therefore, the viewpoint images are not completely separated at the viewpoints of other users (hereafter referred to as non-tracking users), resulting in the observation of ghosts such as double images. Hidden stereo can be an effective solution to this problem. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "Press release for the 'ELF-SR1' spatial reproduction display, offering an overwhelming sense of reality and allowing you to enjoy 3D images with the naked eye," SONY, October 2020, [online], [searched August 6, 2021], Internet,<URL:https: / / www.sony.jp / CorporateCruise / Press / 202010 / 20-1016 / > Summary of the Invention [Problem to be solved by the invention]

[0006] HiddenStereo is a technology for generating stereoscopic images that allows viewers without 3D glasses to see clear 2D images, while viewers with glasses can see 3D images. By displaying stereoscopic images created using the basic viewpoint image Hidden Stereo, non-tracking users can see ghost-free two-dimensional (2D) images. However, in this case, it is not possible to reproduce the motion parallax caused by the viewpoint movement of a tracking user.

[0007] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that can simultaneously present a stereoscopic image including motion parallax to a tracking user and a ghost-free image to a non-tracking user. [Means for solving the problem]

[0008] A video processing device according to one aspect of the present invention generates a stereoscopic image to be presented to multiple users from an original image. The video processing device is a computer including a processor. The processor discretely divides an assumed viewpoint position of a tracking user who is a primary viewer of the stereoscopic image, acquires the actual viewpoint position of the tracking user, generates left and right parallax induction patterns based on viewpoint images of the actual viewpoint positions from viewpoint images obtained by capturing objects included in the original image from the multiple viewpoint positions, and generates a stereo pair image including an image obtained by adding the parallax induction pattern to a reference image to be presented and an image obtained by subtracting the parallax induction pattern from the reference image. [Effects of the Invention]

[0009] According to one aspect of the present invention, there is provided an image processing device that can simultaneously present a stereoscopic image including motion parallax to a tracking user and a ghost-free image to a non-tracking user. and image processing method It will be possible to provide. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of a video processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example in which the assumed viewpoint position of a tracking user is discretely divided. [Figure 3] FIG. 3 is a diagram for explaining the generation of a stereo pair of images corresponding to the viewpoint position Center. [Figure 4] FIG. 4 is a diagram for explaining the generation of a stereo pair of images corresponding to the viewpoint position L1. [Figure 5] FIG. 5 is a diagram for explaining the generation of a stereo pair of images corresponding to the viewpoint position R1. [Figure 6] FIG. 6 is a diagram illustrating an example of parallax guidance according to the embodiment. [Figure 7] FIG. 7 is a diagram for explaining an example of parallax induction using existing technology for comparison. [Figure 8] FIG. 8 is a diagram for explaining a method for reproducing motion parallax in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of a video processing device according to an embodiment. The video processing device 20 of the embodiment may be configured as a computer. The video processing device 20 does not need to be a single computer, and may be configured by multiple computers. As shown in Fig. 2, the video processing device 20 has a processor 201, a read-only memory (ROM) 202, a random access memory (RAM) 203, a storage 204, an input device 205, and a communication module 206. Here, the video processing device 20 may further have a display or the like.

[0012] The processor 201 is a processing circuit capable of executing various programs and controls the overall operation of the video processing device 20. The processor 201 may be a processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit). The processor 201 may also be an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. Furthermore, the processor 201 may be configured with a single CPU or the like, or may be configured with multiple CPUs or the like.

[0013] The ROM 202 is a non-volatile semiconductor memory that stores programs and control data for controlling the video processing device 20 .

[0014] The RAM 203 is, for example, a volatile semiconductor memory, and is used as a working area for the processor 201 .

[0015] The storage 204 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores a program 2041 and original image data 2042.

[0016] The program 2041 is a program for processing the original image data 2042 and generating a 3D (three-dimensional) image. The program 2041 is a program for causing the processor 201 to execute the following processes: a process of discretely dividing an assumed viewpoint position of a tracking user who is a main viewer of a stereoscopic image; a process of acquiring an actual viewpoint position of the tracking user; a process of generating left and right parallax induction patterns based on viewpoint images of the actual viewpoint positions from viewpoint images obtained by photographing an object included in the original image from a plurality of viewpoint positions; and a process of generating a stereo pair image including an image obtained by adding the parallax induction pattern to a reference image to be presented and an image obtained by subtracting the parallax induction pattern from the reference image.

[0017] The input device 205 is an interface device that allows an administrator of the video processing device 20 to operate the video processing device 20. The input device 205 may include, for example, a touch panel, a keyboard, a mouse, various operation buttons, various operation switches, etc. The input device 205 may be used to input, for example, original image data 2042.

[0018] The communication module 206 is a module including a circuit used for communication between the video processing device 20 and the 3D display 100. The communication module 206 may be, for example, a communication module conforming to a wired LAN standard. Alternatively, the communication module 206 may be, for example, a communication module conforming to a wireless LAN standard.

[0019] FIG. 2 is a diagram showing an example in which the expected viewpoint position of a tracking user is discretely divided. FIG. 2 shows the 3D display 100 as viewed from above. For example, the viewpoint position with respect to the 3D display 100 can be divided into a Center in the center of the field of view and two regions, L1 and R1, on the left and right of the Center. Of course, the expected viewpoint position can also be divided into more regions. For example, one Center can be set, and three regions, L1, L2, and L3, can be set on the left side, and similarly three regions, R1, R2, and R3, can be set on the right side.

[0020] FIG. 3 is a diagram for explaining the generation of a stereo pair image corresponding to a center viewpoint position. The processing shown in FIG. 3 is similar to the known HiddenStereo processing. Three viewpoint images of a target 3D object captured from multiple viewpoint positions are input to the left and right of a center reference image. The phase difference from the reference image increases by 45 degrees as you move to the right and decreases by 45 degrees as you move to the left.

[0021] Here, a parallax induction pattern can be generated by inputting L2 and R2 viewpoint images with a phase difference of 180 degrees and a Center viewpoint image. Then, a stereo pair image is generated including an image (+1) obtained by adding the parallax induction pattern to the reference image (Center) to be presented, and an image (-1) obtained by subtracting the parallax induction pattern from the reference image.

[0022] The stereo pair images generated in this manner are output when the tracking user's viewpoint position is at the center. This allows the tracking user to perceive the stereo pair images as 3D images. However, it is difficult to reproduce motion parallax with this processing alone. Below, we will explain an embodiment that can reproduce motion parallax for the tracking user.

[0023] [First embodiment] 4 is a diagram for explaining the generation of a stereo pair image corresponding to viewpoint position L1. First, processor 201 discretely divides the user's expected viewpoint position, and generates HiddenStereo pair images having motion parallax corresponding to each viewpoint position based on viewpoint images obtained by capturing a 3D object to be displayed from multiple viewpoint positions, and stores the generated images in storage 204, for example.

[0024] Next, the processor 201 detects the viewpoint position of the tracked user and determines which assumed viewpoint position it corresponds to. In Fig. 4, it is assumed that the viewpoint position is detected at position L1. The processor 201 then reads out the HiddenStereo pair image corresponding to the assumed viewpoint position from the storage 204 and outputs it.

[0025] In Fig. 4, a parallax induction pattern is generated using as input a viewpoint image L1 at viewpoint position L1 and viewpoint images L3 and R1 having a phase difference of 180 degrees from L1. Then, a stereo pair image is generated including an image (+1) obtained by adding the parallax induction pattern to a reference image (Center) to be presented, and an image (-1) obtained by subtracting the parallax induction pattern from the reference image.

[0026] The stereo pair images generated in this way are output when the tracking user's viewpoint position is L1. This allows the tracking user to perceive the stereo pair images as 3D images even at viewpoint position L1. In other words, it is possible to generate stereo pair images corresponding to viewpoint position L1 (asymmetric parallax induction).

[0027] Fig. 5 is a diagram for explaining generation of a stereo pair image corresponding to a viewpoint position R1. In Fig. 5, a parallax induction pattern is generated using as input a viewpoint image R1 at the viewpoint position R1 and viewpoint images L1 and R3 having a phase difference of 180 degrees from the viewpoint image R1. Then, a stereo pair image is generated including an image (+1) obtained by adding the parallax induction pattern to a reference image (Center) to be presented, and an image (-1) obtained by subtracting the parallax induction pattern from the reference image.

[0028] The stereo pair images generated in this manner are output when the tracking user's viewpoint position is R1. This allows the tracking user to perceive the stereo pair images as 3D images even at viewpoint position R1. In other words, it is possible to generate stereo pair images corresponding to viewpoint position L1. Furthermore, by generating images for other viewpoints in the same way and switching the stereo pair images to be output depending on the tracking user's viewpoint position, it is possible to reproduce motion parallax using a parallax induction pattern corresponding to the viewpoint position.

[0029] 6 is a diagram for explaining an example of parallax induction in the embodiment, in which an asymmetric parallax induction pattern is generated. In Fig. 6(a), from left to right, the L1-based disparity induction pattern (-), the edge of the reference image (Center), and the L1-based disparity induction pattern (+) are shown. The edge of the reference image (Center) is assumed to be 45 degrees to the right of the L1 edge.

[0030] As shown in Figure 6(b), the left-eye image is generated by combining the L1-based disparity induction pattern (-) with the edge of the reference image (Center). The right-eye image is generated by combining the edge of the reference image (Center) with the L1-based disparity induction pattern (+). The edges are induced in the left-eye image, and a viewpoint image in the L3 direction (Center -135 [deg]) is perceived. The edges are induced in the right-eye image, and a viewpoint image in the R1 direction (Center +45 [deg]) is perceived.

[0031] As shown in Fig. 6(c), when the left and right viewpoint images are combined, the parallax induction pattern is canceled and only the center edge is perceived. Here, the processor 201 may be provided with an adjustment function to shift the viewpoint image pair that creates the parallax induction pattern or to add processing to widen the parallax interval so that the edge perception of the reference image is at a desired position.

[0032] 7 is a diagram for explaining an example of parallax induction by an existing technique for comparison. In the existing HiddenStereo, a bilaterally symmetric parallax induction pattern is generated. In FIG. 7(a), from the left, an L1-based parallax induction pattern (-), an edge of the viewpoint image L1, and an L1-based parallax induction pattern (+) are shown. As shown in Figure 7(b), a left-eye image is generated by combining an L1-based disparity induction pattern (-) with the edge of viewpoint image L1. A right-eye image is generated by combining the edge of viewpoint image L1 with an L1-based disparity induction pattern (+). Edges are induced in the left-eye image, and a viewpoint image equivalent to L3 (L1-90[deg]) is perceived. Edges are induced in the right-eye image, and a viewpoint image equivalent to R1 (L1+90[deg]) is perceived. As shown in Figure 7(c), when the left and right viewpoint images are synthesized, the disparity-induced pattern is canceled and only the edge of L1 is perceived.

[0033] As described above, in the embodiment, by generating a left-right asymmetric parallax induction pattern and switching the stereo pair images to be output according to the viewpoint position of the tracking user, it is possible to realize the reproduction of motion parallax by the parallax induction pattern corresponding to the viewpoint position. That is, according to the embodiment, it is possible to present a 3D image including motion parallax due to viewpoint movement to the tracking user, and a ghost-free 2D image (reference image) to the non-tracking user. That is, according to the embodiment, a video processing device that can simultaneously present a stereoscopic image including motion parallax to the tracking user and a ghost-free image to the non-tracking user, and image processing method It will be possible to provide.

[0034] [Second embodiment] In the second embodiment, a stereo pair image generation method different from that of the first embodiment will be disclosed. In particular, optimization of the phase shift amount will be described. For example, instead of using viewpoint image L1 as input, three viewpoint images L3, Center, and R1 may be used as input, and stereo pair images with optimized phase shift amounts may be generated by the following procedure.

[0035] The phase of the viewpoint image Center is defined as x, the phase of the viewpoint image L3 as l_3, the phase of the viewpoint image R1 as r_1, the phase shift amount (and direction) of the desired parallax induction pattern as y, and the amplitude as A.

[0036] The phase shift amount (and direction) z after adding the parallax-inducing pattern is expressed by equation (1).

[0037]

number

[0038] The phase shift amount (and direction) z' after subtraction of the parallax-induced pattern is expressed by equation (2).

[0039]

number

[0040] A set of (A, y) that minimizes equation (3) is found by exhaustive search.

[0041]

number

[0042] Furthermore, for each frequency component in the image, the optimal set of (A, y) is found using the above procedure. This procedure not only allows us to present a 3D image with motion parallax to a tracking user and a ghost-free image to a non-tracking user, but also optimizes the amount of phase shift.

[0043] [Third embodiment] In the third embodiment, reproduction of motion parallax by HiddenStereo presentation corresponding to the viewpoint position will be described.

[0044] Fig. 8 is a diagram for explaining a method of reproducing motion parallax in the third embodiment. In Fig. 8, HiddenStereo images corresponding to the assumed viewpoint positions are created and switched and presented according to the viewpoint position of the tracking user, thereby reproducing motion parallax while allowing the non-tracking user to perceive a 2D image (reference image) without ghosts. At this time, the processor 201 switches the reference image according to the movement of the viewpoint of the tracking user.

[0045] In Figure 8, based on the reference image at viewpoint L1, the reference image at viewpoint Center, and the reference image at viewpoint R1, a parallax induction pattern is generated from the reference image at each viewpoint and the two viewpoint images on either side of it. A stereo pair image is also generated by adding or subtracting the parallax induction pattern at each viewpoint position to the reference image. The stereo pair image to be output is then switched depending on the viewpoint position of the tracking user. In this way, the 3D image seen by the tracking user can be shared with the non-tracking user as a ghost-free 2D image.

[0046] As described above, according to the above-described embodiments, a video processing device that can present a stereoscopic video including motion parallax to a tracking user and a ghost-free video to a non-tracking user is provided. and image processing method It will be possible to provide.

[0047] The program for realizing the above-described processing may be provided by being stored on a computer-readable recording medium (or storage medium). The program is stored on the recording medium as an installable file or an executable file. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. The program for realizing the above-described processing may also be stored on a computer (server) connected to a network such as the Internet and downloaded to a computer (client) via the network.

[0048] The video processing device according to the embodiment can be configured by configuring the operations of each component as a program, which can be installed and executed on a computer used as the video processing device, or distributed via a network. The present invention is not limited to the above-described embodiments, and various modifications and applications are possible.

[0049] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0050] 20...Video processing device, 100...display, 201...processor, 202...ROM, 203...RAM, 204...Storage, 205...input device, 206...communication module, 2041… Program, 2042...Original image data.

Claims

1. a storage unit that stores a program; a memory into which the program is loaded from the storage unit; a processor for processing information according to instructions written in a program loaded into said memory; The processor: The assumed viewpoint position of a tracking user viewing an image presented by the viewpoint-tracking naked-eye 3D display is discretely divided; Acquire a real viewpoint position of the tracking user; generating left and right parallax induction patterns based on viewpoint images of the actual viewpoint positions from viewpoint images obtained by photographing an object included in the original image from a plurality of viewpoint positions; creating a stereo pair of images for each of the assumed viewpoint positions, the stereo pair including an image obtained by adding the parallax induction pattern to a reference image to be presented and an image obtained by subtracting the parallax induction pattern from the reference image; An image processing device that switches the stereo pair images according to the viewpoint position of the tracking user and presents them on the viewpoint tracking naked-eye 3D display.

2. The image processing device according to claim 1 , wherein the processor presents the reference image to a user other than the tracking user.

3. The image processing device according to claim 1 , wherein the processor adjusts the positions of the pair of viewpoint images used to generate the parallax-inducing pattern to set the edge perception of the reference image to a desired position.

4. The video processing device according to claim 1 , wherein the processor adjusts a parallax interval between a pair of viewpoint images used to generate the parallax induction pattern, and sets the edge perception of the reference image to a desired position.

5. The image processing device according to claim 1 , wherein the processor optimizes the amount of phase shift of the stereo pair images.

6. 1. A video processing method for a computer including a storage unit that stores a program, a memory into which the program is loaded from the storage unit, and a processor that processes information in accordance with instructions written in the program loaded into the memory, The processor discretely divides an assumed viewpoint position of a tracking user who watches an image presented by the viewpoint-tracking naked-eye 3D display; The processor acquires a real gaze position of the tracking user; The processor generates left and right parallax induction patterns based on a viewpoint image of the actual viewpoint position from viewpoint images obtained by photographing an object included in an original image from a plurality of viewpoint positions; The processor creates a stereo pair image for each of the assumed viewpoint positions, the stereo pair image including an image obtained by adding the parallax induction pattern to a reference image to be presented and an image obtained by subtracting the parallax induction pattern from the reference image; The processor switches the stereo pair images according to the viewpoint position of the tracking user and presents them on the viewpoint tracking naked-eye 3D display.

Images