Display device, image generating method and program

The display device integrates external scenery with the displayed image by using multiple sensors to calculate the photographer's viewpoint, addressing the miniature garden effect and enhancing immersion in augmented reality.

JP7593334B2Active Publication Date: 2024-12-03SONY GROUP CORP
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2021571151
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-01-16
Filing Date
2021-01-06
Publication Date
2024-12-03
Estimated Expiration
2041-01-06

AI Technical Summary

Technical Problem

Existing display devices fail to seamlessly integrate the image displayed on the screen with the scenery outside, creating a miniature garden effect that diminishes the immersive and realistic experience, especially in augmented reality applications.

Method used

A display device equipped with a first image sensor and a first distance sensor on one surface, a second sensor on the opposite surface, and a display section, generates a display image by calculating the photographer's viewpoint based on three-dimensional position information, ensuring the image and outside scenery appear connected.

Benefits of technology

The solution allows for a realistic and immersive experience by making the displayed image appear as if the device is transparent, enhancing the sense of presence and realism in augmented reality environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593334000001
    Figure 0007593334000001
  • Figure 0007593334000002
    Figure 0007593334000002
  • Figure 0007593334000003
    Figure 0007593334000003
Patent Text Reader

Abstract

[Problem] To provide a display device, an image generation method and a program with which it is possible to produce such image display that the image displayed on a display unit and the scenery on the outside of a display device look connected. [Solution] The display device is provided with a first image sensor, a first distance sensor, a second sensor, a display unit, and an image generation unit. The first image sensor is arranged on a first face side of the device body. The first distance sensor is arranged on the first face side. The second sensor is arranged on a second face side that is opposite the first face. The display unit is arranged on the second face side. On the basis of the three-dimensional position information of a photographer's viewpoint calculated on the basis of the sensing result acquired by the second sensor, the image generation unit generates, using the two-dimensional image of a subject acquired by the first image sensor and the distance image of the subject acquired by the first distance sensor, a display image that is displayed on the display unit.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to a display device, an image generating method, and a program. [Background technology]

[0002] Mobile devices with displays, such as mobile phones such as smartphones and tablet terminals, are becoming widespread. Mobile devices equipped with cameras can display images captured by the camera on the display, and can present Augmented Reality (AR) information by overlaying the images. For example, a highly immersive and realistic AR experience can be provided to the photographer by displaying the scenery behind the mobile device captured by the camera mounted on the mobile device on the display.

[0003] Patent Document 1 discloses a technology in which the scenery on the other side of the device changes and is displayed according to the user's viewpoint position. Patent Document 1 describes a technology for improving the sense of realism of remote communication, and describes how a special display panel with embedded display and imaging elements can be used to display different images in various directions and simultaneously capture an object from various directions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2002-300602 A Summary of the Invention [Problem to be solved by the invention]

[0005] When an image of the view beyond the display device, captured by a camera mounted on the display device, is displayed on the display unit, the displayed image is visually recognized as a miniature garden within the display device.

[0006] In consideration of the above circumstances, an object of the present technology is to provide a display device, an image generating method, and a program that can display an image so that the image displayed on the display unit appears connected to the scenery outside the display device. [Means for solving the problem]

[0007] In order to achieve the above object, a display device according to the present technology includes a first image sensor, a first distance sensor, a second sensor, a display unit, and an image generation unit. The first image sensor is disposed on a first surface side of the device body. The first distance sensor is disposed on the first surface side. The second sensor is disposed on a second surface side opposite to the first surface. The display unit is disposed on the second surface side. The image generation unit generates a display image to be displayed on the display unit using the two-dimensional image of the subject acquired by the first image sensor and the distance image of the subject acquired by the first distance sensor based on three-dimensional position information of the photographer's viewpoint calculated based on the sensing results acquired by the second sensor.

[0008] According to such a configuration of the present invention, a display image can be generated as seen from the photographer's viewpoint, and the photographer can visually recognize that the scenery in the image displayed on the display unit is connected to the scenery outside the display device.

[0009] In order to achieve the above object, the image generating method according to the present technology includes: a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface side, and a display unit arranged on the second surface side, acquiring a two-dimensional image of a subject from the first image sensor; acquiring a distance image of the subject from the first distance sensor; Obtaining a sensing result of the second sensor; Calculating three-dimensional position information of the photographer's viewpoint based on the sensing result; Based on the three-dimensional position information, a display image to be displayed on the display unit is generated using the two-dimensional image and the distance image.

[0010] In order to achieve the above objective, the program related to this technology is: A step of acquiring a two-dimensional image of a subject from the first image sensor of a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface, and a display unit arranged on the second surface side; acquiring a distance image of the object from the first distance sensor; acquiring a sensing result of the second sensor; calculating three-dimensional position information of the viewpoint of the photographer based on the sensing result; generating a display image to be displayed on the display unit using the two-dimensional image and the distance image based on the three-dimensional position information; is executed on the display device. [Brief description of the drawings]

[0011] [Figure 1] 1 is a perspective view of a display device according to each embodiment of the present technology as viewed from the front side and the rear side; [Diagram 2] FIG. 2 is a configuration block diagram of the display device. [Diagram 3] FIG. 1 is a diagram showing a state in which a photographer holds a display device according to a first embodiment in his / her hand. [Figure 4] 4A to 4C are diagrams illustrating an outline of a method for generating a display image in the display device. [Diagram 5] FIG. 4 is a flowchart of a display image generating method in the display device. [Figure 6] 11A to 11C are diagrams for explaining a calculation process of a three-dimensional position of a viewpoint in the display image generating method. [Figure 7]11A to 11C are diagrams illustrating a coordinate conversion process in the display image generating method. [Figure 8] 11A to 11C are diagrams illustrating a process of complementing an occlusion region when an image is generated in the display device. [Figure 9] FIG. 13 is a diagram showing an example of a display image on a display device according to a second embodiment, showing a state in which a photographer is holding the display device in his / her hand. [Figure 10] FIG. 13 is a diagram showing an example of a display image on a display device according to a second embodiment, illustrating a state in which a photographer is holding the display device in his / her hand. [Figure 11] 13A and 13B are diagrams illustrating an example of a display image displayed by a display device according to a third embodiment. [Figure 12] 13A to 13C are diagrams illustrating a method for detecting the line of sight of a photographer in a method for generating a display image by a display device in a fourth embodiment. [Figure 13] 13A and 13B are diagrams illustrating an example of a display image displayed by a display device according to a fourth embodiment. [Figure 14] 13A to 13C are diagrams illustrating a method for generating a display image by a display device in a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, a display device according to the present technology will be described with reference to the drawings. <First embodiment> [Display device configuration] The present technology can be suitably used as a display device in a mobile device such as a smartphone or other mobile phone having a display unit, or a tablet, etc. In the following embodiment, a display device in the form of a smartphone will be described as an example.

[0013] Fig. 1 is a perspective view for explaining the outer shape of a display device according to a first embodiment of the present technology, Fig. 1(A) is a perspective view of the display device as seen from the front side where a display unit is located, and Fig. 1(B) is a perspective view as seen from the rear side. 1, the display device 1 includes a housing 10, a rear camera 3, a front camera 6 as a second sensor, and a display unit 4. The display device 1 is configured such that the rear camera 3, the front camera 6, and a display panel constituting the display unit 4, a drive circuit, various sensors, etc. are held in the housing 10.

[0014] The main body of the display device 1 has a first surface 2 on the rear side and a second surface 5 on the front side located opposite the first surface 2. The first surface 2 and the second surface 5 are in a parallel positional relationship. The mutually orthogonal xyz coordinate directions in the figure correspond to the width, length, and height of the approximately rectangular parallelepiped display device 1. The plane parallel to the first surface 2 is the xy plane, and the thickness direction of the display device 1, which corresponds to the height direction, is the z axis.

[0015] An imaging lens 3a of a rear camera 3 is disposed on the side of the first surface 2, and the rear camera 3 captures an image of a subject facing the first surface 2. An imaging lens 6a of a front camera 6 is disposed on the second surface 5 side, and the front camera 6 captures an image of a subject facing the second surface 5. The subject is usually the photographer. A display panel constituting the display unit 4 is provided on the second surface 5. The display unit 4 is composed of an image display means such as a liquid crystal display or an organic electro-luminescence display (EL display). The display unit 4 is configured to be capable of displaying images transmitted and received from an external device via a communication unit (not shown), input operation buttons, images captured by the front camera 6 and the rear camera 3, and the like. The images include still images and videos. Generally, a photographer, who is a user who takes a photograph using the display device 1, looks at an image displayed on the display unit 4 and performs input operations and the like on the operation screen displayed on the display unit 4. Therefore, the photographer is positioned on the second surface 5 side of the display device 1 in order to view the display unit 4. In the specification, the expression "the other side of the display device 1" may be used, but this refers to the direction seen from the photographer and corresponds to the first surface 2 side of the display device 1. On the other side of the display device 1, a subject to be photographed by the rear camera 3 is located.

[0016] FIG. 2 is a block diagram showing the functional configuration of the display device 1. As shown in FIG. As shown in FIG. 2, the display device 1 includes a rear camera 3, a front camera 6, an image generating unit 7, a storage unit 8, and a display unit 4.

[0017] In this embodiment, the rear camera 3 and the front camera 6 each have a function as an image sensor and a function as a distance sensor. The image sensor captures a two-dimensional color image (hereinafter sometimes referred to as an RGB two-dimensional image or an RGB image) of a subject. The distance sensor captures a distance image of the subject. A ToF (Time of flight) type can be suitably used for the distance sensor, and in this embodiment, a ToF type distance sensor is used as an example. In the ToF type distance sensor, a distance image having distance information between the distance sensor and the subject is obtained using near infrared light (NIR light). The distance image, which is the sensing result obtained by the ToF type distance sensor of the front camera 6 as the second sensor, contains not only the distance information but also two-dimensional position information of the photographer's eyes. In this embodiment, an example will be given in which the rear camera 3 and the front camera 6 are each a single imaging device that can capture both an RGB image and a distance image. Hereinafter, the RGB image and the distance image may be collectively referred to simply as an image.

[0018] As shown in FIG. 2, the rear camera 3 has an RGB image imaging element 31 that forms part of the image sensor, a distance image imaging element 32 that forms part of the distance sensor, an image processing circuit 33, an on-chip lens (not shown), a color filter (not shown), and a light emitting unit (not shown). Similarly, the front camera 6 has an RGB image imaging element 61 that forms part of the image sensor, a distance image imaging element 62 that forms part of the distance sensor, an image processing circuit 63, an on-chip lens (not shown), and a color filter (not shown). In the rear camera 3 (front camera 6), by providing color filters between the RGB image imaging element 31 (61) and the distance image imaging element 32 (62) and the on-chip lens, R pixels, G pixels, B pixels, and NIR pixels can be arranged.

[0019] Here, the R pixel is a pixel that obtains an electric charge corresponding to the red component of light from light that has passed through a color filter that transmits the red (R: Red) wavelength component. The G pixel is a pixel that obtains an electric charge corresponding to the green (G: Green) component of light from light that has passed through a color filter that transmits the green (G: Green) wavelength component. The B pixel is a pixel that obtains an electric charge corresponding to the blue (B: Blue) component of light from light that has passed through a color filter that transmits the blue (B: Blue) wavelength component. The NIR pixel is a pixel that obtains an electric charge corresponding to the wavelength band of near-infrared light (NIR light) from light that has passed through a filter that transmits the NIR wavelength component.

[0020] The imaging processing circuit 33 (63) processes the imaging signals obtained by the RGB image imaging element 31 (61) and the distance image imaging element 32 (62) to generate an RGB image and a distance image corresponding to the subject. An on-chip lens is provided for each pixel, and collects external light and makes it incident on the color filter of each pixel. The light-emitting unit, which constitutes part of the distance sensor, is used to measure the distance between the camera and the subject. The light-emitting unit emits NIR light. The distance image capture element 32 (62) receives the return light when the NIR light emitted from the light-emitting unit is reflected by the subject. The light-emitting unit includes a light-emitting member such as a light-emitting diode (LED) and a driver circuit for emitting the light.

[0021] The display unit 4 displays the display image generated by the image generation unit 7. The display unit 4 is configured to be able to display images captured by the rear camera 3 and the front camera 6, but in the following description, an example in which an image captured by the rear camera 3 is displayed on the display unit 4 will be given.

[0022] The image generating unit 7 includes an image information acquiring unit 70 , a viewpoint position calculating unit 71 , a coordinate conversion unit 72 , and a complementing unit 73 . The image information acquisition unit 70 acquires an RGB image and a distance image of the subject from the rear camera 3, and an RGB image and a distance image of the photographer from the front camera 6. As will be described later, when generating a display image in which the scenery displayed on the display unit 4 and the scenery outside the display device 1 are continuously connected and the other side of the display device 1 is seen through, it is sufficient to acquire at least a distance image from the front camera 6. The viewpoint position calculation unit 71 calculates the three-dimensional position of the photographer's viewpoint based on the distance image captured by the front camera 6. The coordinate conversion unit 72 converts the coordinates of the image captured by the rear camera 3 based on the three-dimensional position information of the photographer's viewpoint, and generates a display image that is an RGB image seen from the photographer's viewpoint. If an occlusion region exists in the display image generated by the coordinate changing section 72, the complementing section 73 complements this occlusion region to generate a display image to be displayed on the display section 4. A display image generating method in the image generating unit 7 will be described later.

[0023] Fig. 3 is a diagram showing a state in which a photographer P holds a display device 1 according to this embodiment in a left hand 21. In Fig. 3, an image captured on the other side of the display device 1 is displayed on a display unit 4 of the display device 1. In this embodiment, the image acquired based on the rear camera 3 is converted into an image seen from the viewpoint of the photographer P to generate the display image 51. As a result, as shown in FIG. 3, the scenery 13 outside the display device 1 and the scenery of the display image 53 displayed on the display unit 4 of the display device 1 appear to be continuously connected. As a result, the photographer P recognizes the area blocked by the presence of the display device 1 as if the scenery on the other side is visible through the display device 1, and the scenery displayed on the display unit 4 is no longer like a miniature garden. Therefore, for example, when an image is displayed as AR by superimposing an image for superimposition such as a virtual object, which is Augmented Reality (AR) information, on the image, the user can experience a high sense of immersion and realism in the AR world. The superimposition of virtual images such as virtual objects and virtual characters will be described later as another embodiment.

[0024] The storage unit 8 includes a memory device such as a RAM and a non-volatile recording medium such as a hard disk drive, and stores a program for causing the display device to execute a process for generating a display image to be displayed on the display unit 4 of the display device 1. The program stored in the recording unit 8 causes the display device to execute the following steps: acquiring an RGB image, which is a two-dimensional image of the subject, and a distance image of the subject from the rear camera 3; acquiring a distance image that is the sensing result of the front camera 6; calculating three-dimensional position information of the viewpoint of the photographer P based on the distance image of the front camera 6; and generating a display image to be displayed on the display unit 4 using the RGB image and distance image acquired by the rear camera 3 based on the three-dimensional position information.

[0025] [Display image generation method] FIG. 4 is a diagram for explaining an outline of a display image generating method. As shown in FIG. 4(A), an image of an object 11 serving as a subject captured by the rear camera 3 is an image based on the position of the rear camera 3. As shown in Fig. 4(B), in the display image generating method of this embodiment, a display image is generated by performing coordinate conversion on an image of an object 11 captured by a rear camera 3 so that the image is an image seen from a viewpoint E. Furthermore, if an occlusion area exists in the display image generated by the coordinate conversion, a process of complementing the occlusion area is performed to generate a display image to be displayed on the display unit 4. The following will explain this.

[0026] FIG. 5 is a flow diagram of a display image generating method. 5, the image information acquisition unit 70 acquires an RGB image and a distance image of the subject from the rear camera 3, and an RGB image and a distance image of the photographer from the front camera 6 (ST1). The rear camera 3 is calibrated to the rear side, and the front camera 6 is calibrated to the front side. From the RGB image and distance image acquired by the rear camera 3, point cloud information from the rear camera 3 to the object 11 that is the subject can be acquired. Next, the viewpoint position calculation unit 71 calculates the three-dimensional position of the photographer's viewpoint based on the distance image captured by the front camera 6 (ST2). The calculation process will be described later. Next, the coordinate conversion unit 72 converts the coordinates of the image captured by the rear camera 3 into an image seen from the photographer's viewpoint, using the three-dimensional position information of the photographer's viewpoint (ST3). The coordinate conversion process will be described later. Next, if there is an occlusion region in the display image that has been coordinate-converted by the coordinate conversion unit 72, the complement unit 73 complements this occlusion region to generate a display image to be displayed on the display unit 4 (ST4). The complement process of the occlusion region will be described later.

[0027] (3D viewpoint position calculation process) The viewpoint will be described with reference to FIG. As shown in Figure 6(A), when both eyes of photographer P are open, the viewpoint E is the midpoint that bisects the line segment connecting the centers of the pupils (irises) of photographer P's right eye 9R and left eye 9L. As shown in Fig. 6(B), when one eye of the photographer P is closed and the other eye is open, the center point of the pupil of the open eye is taken as viewpoint E. In the example shown in Fig. 6(B), the right eye 9R is closed, and the center point of the pupil of the open left eye 9L is taken as viewpoint E. In Fig. 6(B), viewpoint E is represented by a white circle. Viewpoint E is used as a reference when generating a display image seen by photographer P using an image captured by rear camera 3.

[0028] In this embodiment, the front camera 6 is equipped with a ToF sensor, which is a distance sensor. The viewpoint position calculation unit 71 performs face detection and detection of the two-dimensional positions of the pupils of the left and right eyes by a conventional method for the NIR image, which is a distance image acquired by the ToF sensor. Then, it acquires three-dimensional position information of the center points of the pupils of the left and right eyes from the ToF distance values ​​of the detected pixels. Furthermore, the viewpoint position calculation unit 71 calculates three-dimensional position information of the viewpoint E from the acquired three-dimensional position information of the center points of the pupils of the left and right eyes. As described above, when both eyes of the photographer P are open, the center point of the left and right pupils is set as viewpoint E, and when one eye is open, the center of the pupil of the open eye is set as viewpoint E. In this manner, in this embodiment, an optimal display image can be generated depending on whether the eyes are open or closed.

[0029] (Coordinate conversion processing) The coordinate conversion process will be described with reference to FIG. The coordinate conversion unit 72 calculates to which coordinate values ​​on the display unit 4 of the display device 1 the point cloud of the object 11, which is the subject captured by the rear camera 3, is mapped. Hereinafter, symbols E, D, A, F, R, and O shown in FIG. 7 and in each equation described later indicate the three-dimensional position of each point. The meaning of each symbol is as follows. That is, E: Viewpoint A: The display origin at the top left of the display F: Front camera R: Rear camera O: Object Viewpoint E is the position of the viewpoint calculated by the above-mentioned method for calculating the three-dimensional position of the viewpoint. The display unit origin A is a point located at the upper left corner of the surface of the rectangular display unit 4 when the photographer P holds the display device 1 vertically, facing the second surface 5 on which the display unit 4 is located, and so that the rear camera 3 and the front camera 6 are located above the display unit 4. A point F of the front camera 6 is the origin of the camera coordinates of the front camera 6. A point R of the rear camera 3 is the origin of the camera coordinates of the rear camera 3 . A point O of an object 11 which is a subject is an arbitrary point on the object 11. The subscripts of vectors between three-dimensional points represent the reference coordinate positions and are expressed in lower case letters corresponding to the upper case letters of the symbols above. xyz represent the coordinate axis directions.

[0030] The calculations for coordinate transformation are performed in the following order: 1, 2. 1. In the viewpoint coordinate system e, the straight line from the viewpoint E to an arbitrary point O on the object 11 and the plane D of the display unit 4 are each expressed by a formula. The viewpoint coordinate system e is a coordinate system with the viewpoint E as its origin. The plane D of the display unit 4 corresponds to the surface of the display unit 4. 2. Find the coordinates of the intersection between the line in 1 above and the plane. It is assumed that the positions of the front camera 6, rear camera 3, and display unit 4 are calibrated and can be treated as being parallel to each other. In the xyz coordinate directions, the z coordinate has a negative value in the direction from the second surface 5 to the first surface 2. For example, the z coordinate of the viewpoint coordinate detected by the front camera 6 always has a negative value. When the photographer P faces the second surface 5 on which the display unit 4 is disposed and holds the display device 1 in the vertical direction so that the rear camera 3 and the front camera 6 are located above the display unit 4, the y coordinate has a positive value in the direction from top to bottom as seen by the photographer P, and the x coordinate has a positive value in the direction from left to right as seen by the photographer P.

[0031] In the drawings and the following description, Or indicates the xyz coordinate values ​​of point O of the object 11 when point R of the rear camera 3 is the origin. Orx, Ory, and Orz respectively indicate the x-coordinate value, y-coordinate value, and z-coordinate value of point O of the object 11 when point R of the rear camera 3 is the origin. Oe indicates the xyz coordinate values ​​of point O of object 11 when viewpoint E is the origin. Oex, Oey, and Oez respectively indicate the x-coordinate value, y-coordinate value, and z-coordinate value of point O of object 11 when viewpoint E is the origin. Df indicates the xyz coordinate values ​​of a point on plane D when point F of front camera 6 is the origin. Dfx, Dfy, and Dfz respectively indicate the x-, y-, and z-coordinate values ​​of a point on plane D when point F is the origin. Ef indicates the xyz coordinate values ​​of the viewpoint E when the point F of the front camera 6 is the origin. Efx, Efy, and Efz respectively indicate the x-, y-, and z-coordinate values ​​of the viewpoint E when the point F is the origin. Af indicates the xyz coordinate values ​​of the display unit origin A when the point F of the front camera 6 is the origin. Afx, Afy, and Afz respectively indicate the x-coordinate value, the y-coordinate value, and the z-coordinate value of the display unit origin A when the point F is the origin. Fr indicates the xyz coordinate values ​​of point F of the front camera 6 when point R of the rear camera 3 is the origin. Frx, Fry, and Frz respectively indicate the x-coordinate value, y-coordinate value, and z-coordinate value of point F of the front camera 6 when point R of the rear camera 3 is the origin.

[0032] First, the above item 1 will be explained. The equation for the coordinates Oe of the object in the viewpoint coordinate system e is Oe=Or-Fr-Ef The line from point E to point O in the viewpoint coordinate system e can be expressed by the following three equations. These three equations are called equation (1). x=a(Orx-Frx-Efx) y=a(Ory-Fry-Efy) …Equation (1) z=a(Orz-Frz-Efz) Here, a represents a parameter.

[0033] The equation of plane D in the viewpoint coordinate system e is De=-Ef+Af+Df This can be expressed by the following three equations. These three equations are called (2). x=-Efx+Afx+Dfx y=-Efy+Afy+Dfy …Equation (2) z=-Efz+Afz

[0034] Next, we will explain the above item 2. (Dfx, Dfy) indicate coordinates on the display unit 4 (called display unit coordinates). The display unit coordinates onto which an arbitrary point O of the object 11 is mapped are obtained, that is, (Dfx, Dfy) is expressed as coordinates with the arbitrary point O of the object 11 as the origin. This allows calculation of which display unit coordinate value on the display unit 4 the point cloud of the object 11 obtained by the rear camera 3 is mapped. The specific calculation method is as follows. From the above formula (1) and formula (2), a(Orx-Frx-Efx)=-Efx+Afx+Dfx...Equation (3) a(Ory-Fry-Efy)=-Efy+Afy+Dfy...Equation (4) a(Orz-Frz-Efz)=-Efz+Afz...Equation (5) From the above formula (5), a=(-Efz+Afz) / (Orz-Frz-Efz)...Equation (6) From the above formulas (3), (4) and (6), Dfx=(Orx-Frx-Efx)(-Efz+Afz) / (Orz-Frz-Efz)+Efx-Afx Dfy=(Ory-Fry-Efy)(-Efz+Afz) / (Orz-Frz-Efz)+Efy-Afy In this way, (Dfx, Dfy) can be expressed in coordinates with an arbitrary point O of the object 11 as the origin. This coordinate conversion process calculates which display coordinate value of the display unit 4 each point of the point cloud of the object 11 obtained by the rear camera 3 is mapped to. By performing this for all points of the point cloud of the object 11, it is possible to know where on the display unit 4 the object 11 should be displayed. As a result, the image of the scenery beyond the display device 1 displayed on the display unit 4 is an image seen from the viewpoint E of the photographer P. Therefore, the photographer P can visually recognize that the scenery of the image displayed on the display unit 4 and the scenery outside the display device 1 are continuously connected, giving the photographer a feeling that the other side of the display device 1 is as if it is see-through.

[0035] In this way, in the coordinate conversion process, the intersection coordinates between the straight line from the photographer P's viewpoint E to each point of the point cloud of the object (subject) and the plane of the display unit 4 are calculated for each point, and a display image is generated so that each point of the subject's point cloud is mapped to the corresponding intersection coordinates.

[0036] (Occlusion Compensation Processing) When the above-mentioned coordinate conversion process is performed using point cloud data of the image before the coordinate conversion process to generate an image from a different viewpoint from the image before the coordinate conversion process, an occlusion area occurs in the generated image. In the occlusion complement process, a process for complementing this occlusion area is performed, and a display image that is finally displayed on the display unit 4 is generated.

[0037] FIG. 8 is a diagram for explaining an occlusion region. For example, suppose that a person 12, who is a subject, is photographed from the front and diagonally left using the rear camera 3 to obtain an image. When the above-mentioned coordinate conversion process is performed using point cloud data of the image obtained by the rear camera 3 to generate an image of the person viewed from the front, as shown in FIG. 8(A), an area hidden and not visible due to the position of the person 12 in the image before the conversion process occurs as an occlusion region 81 in the image 52 after the coordinate conversion process. This occlusion region 81 is an area where no point cloud data exists and an image cannot be generated. In FIG. 8(A), the occlusion region 81 is indicated by diagonal lines. By complementing this occlusion region 81, a display image 53 is generated in which the image of the missing occlusion region 81 is complemented, as shown in Fig. 8(B). This makes it possible to obtain a display image that does not feel unnatural, and enhances the sense of realism and immersion.

[0038] The occlusion region can be complemented by using image correction processing such as an image generation model GAN ​​(Generative Adversarial Networks). In addition, when time-series image data is input, the DVD-GAN (Dual Video Discriminator GAN) method, which uses time-series information, can be applied. This method uses information from previous and subsequent frames, rather than the conventional GAN ​​that works on a frame-by-frame basis, so occlusion areas can be complemented with higher accuracy, resulting in a more natural display image. For example, when an area previously captured by a camera becomes an occlusion area, image information captured in the past can be used.

[0039] As described above, in this embodiment, the distance image from the front camera 6 and the distance image and RGB image from the rear camera 3 are used to convert the field of view image captured by the image sensor of the rear camera 3, with the rear camera 3 as the viewpoint, into a field of view image based on the photographer's viewpoint, and used as the display image.

[0040] Although the display image generating method has been described using an example in which the display device 1 is held vertically so that the cameras 3 and 6 are located above the display unit 4, the present technology can also be applied when the display device 1 is held horizontally. Regardless of the camera orientation, the display image can be obtained by the above-mentioned calculation method without changing the relative orientation of the coordinate systems of the cameras 3 and 6 or the position of the display unit origin A.

[0041] <Second embodiment> In this embodiment, an example will be described in which an image for superimposition is superimposed on an image captured on the far side by the display device 1 of the present technology. Here, an example will be described in which the image for superimposition is a virtual object. 9 and 10 show how a photographer P holds a display device 1 with left hand 21. Display unit 4 of display device 1 in Fig. 9 displays display image 54 in which virtual object 85 in the form of a bear is superimposed on an image captured of the other side of display device 1. Display unit 4 of display device 1 in Fig. 10 displays display image 55 in which virtual object 85 is superimposed on an image captured of the other side of display device 1. In Fig. 10, photographer P looks at display image 55 with virtual object 85 superimposed, and points with right hand 22 at virtual object 85 in display image 55 in space.

[0042] In this way, a display image may be generated by superimposing the virtual object 85 on the display image generated by the above-described display image generating method. The virtual object 85 is a virtual object in a virtual space. In this embodiment, a display image in which the virtual object 85 is superimposed on an image of an actual subject photographed by the rear camera 3 of the display device 1 is displayed on the display unit 4. As shown in Figs. 9 and 10, the display image is generated by superimposing the virtual object 85 on an image that appears to be continuously connected to the scenery 13 on the other side of the display device 1, so that the photographer P can feel as if the virtual object 85 exists in the actual space, and can experience a high sense of immersion and realism in the AR world. Furthermore, as shown in FIG. 10, when photographer P points to virtual object 85 with right hand 22, right hand portion 22a in display image 55 and right hand portion 22b located outside display device 1 are visually perceived as being continuously connected, so that photographer P feels as if he or she is touching virtual object 85 more realistically.

[0043] 10, when it is assumed that an operation will be performed with the photographer's fingers while looking at a display image in which virtual object 85 is superimposed, a display image may be generated as follows. That is, a hand area reflected in the display image generated by the above-described display image generation may be extracted, and a display image may be generated in which virtual object 85 is deformed or moved accordingly if part of the hand area comes into contact with virtual object 85 according to distance information of the hand area obtained by rear camera 3 and the position of virtual object 85. This allows the photographer to feel more immersed and present in the AR world.

[0044] <Third embodiment> The image generating unit 7 may change the display content of the virtual image to be superimposed on the display image according to the distance between the viewpoint of the photographer P and the display unit. Hereinafter, a description will be given with reference to Fig. 11. Here, an example will be described in which the superimposed image is a character image.

[0045] Figures 11(A) and (B) show examples of the appearance of a photographer P when he uses the rear camera 3 of the display device 1 to photograph a plate with multiple dishes placed on a tray, and examples of display images 56 and 57 displayed on the display unit 4 of the display device 1. A display image 56 in FIG. 11(A) is an image displayed on the display unit 4 when the photographer P takes a picture by moving the display device 1 away from the eye and bringing it close to the plate. A display image 57 in FIG. 11(B) is an image displayed on the display unit 4 when the photographer P takes a picture by holding the display device 1 close to his / her eye and away from the plate. In an image captured by bringing the display device 1 shown in FIG. 11(A) closer to the plates, the subject matter, that is, the multiple plates, appears to be enlarged compared to the image captured by moving the display device 1 away from the plates, as shown in FIG. 11(B). In this embodiment, as in the above-described embodiment, an image seen from the viewpoint of photographer P is generated, and a display image is generated so that the view on the other side displayed on display unit 4 and the view outside display device 1 that is not obstructed by the presence of display device 1 appear to be continuously connected.

[0046] The distance between the viewpoint of the photographer P and the display unit 4 can be found by the three-dimensional position calculation process of the viewpoint described in the first embodiment. In this embodiment, if the calculated distance is equal to or greater than a preset threshold, a display image 56 is generated by superimposing a character image 86 of the name of the food served on each plate, as shown in Fig. 11(A). If the calculated distance is less than the threshold, a display image 57 is generated by superimposing a character image 87 indicating food information such as calorie information and allergy information in addition to the character image 86 of the food name, as shown in Fig. 11(B). In the example shown in FIGS. 11(A) and (B), in addition to changing the display content, the size of the text image is also changed according to the distance between the viewpoint of the photographer P and the display unit. Compared to Fig. 11(A), in the example shown in Fig. 11(B), the photographer P looks at the display unit 4 by bringing his / her eyes closer to the display device 1, so that the photographer P can read even relatively small characters. Also, since the characters are displayed in small characters, more information can be displayed. On the other hand, in Fig. 11(A), by reducing the information about the contents of the food on the plate and making the characters larger, the photographer P can grasp the general information about the food even if he / she looks away from the display device 1.

[0047] In this way, the image generating unit 7 may generate a display image by changing the display content information of the superimposition image according to the distance between the viewpoint of the photographer P and the display unit. Note that changes in the display content of the superimposition image include cases where the displayed information content is different, as well as cases where the size of the displayed superimposition image (here, the size of the characters) is different even if the information content is the same. Furthermore, in this embodiment, a display image is displayed that allows one to see through the display device 1 and is further superimposed with a text image, so that the food viewed with direct vision appears to have text indicating the name or other food information placed on it.

[0048] <Fourth embodiment> In the third embodiment, an example was given in which the display content of the image to be superimposed is changed according to the distance between the viewpoint and the display unit 4. Alternatively, the user's line of sight may be estimated, and the display content of the image to be superimposed may be changed according to the estimated line of sight. This will be described below with reference to Figs. 11(B), 12 and 13.

[0049] An example of processing for detecting the gaze from a distance image or an RGB image captured by the front camera 6 will be described with reference to Fig. 12. In Fig. 12, the area indicated by dots indicates the area of ​​the pupil when the pupil is located at the center of the eye. The gaze detection processing is performed by a gaze detection processing unit (not shown). The gaze detection processing unit detects the photographer's face from the image, and the positions of the left and right eyes of the face are detected by image recognition processing. Furthermore, gaze detection is performed based on the positions of the pupils in the eyes detected by the image recognition processing. Generally, when moving the eyes unconsciously, the pupils of the left and right eyes behave in the same way. For example, when the face is not moved and the gaze is directed upward, the pupils of the left and right eyes move upward. Therefore, gaze detection can be performed based on the position of the pupil of one open eye.

[0050] As shown in Fig. 12, when state 90 where the pupil is in the center of the eye is detected by image recognition, the gaze is deemed to be toward the center. When state 91 where the pupil is on the left side of the eye is detected by image recognition, the gaze is deemed to be leftward. When state 92 where the pupil is on the right side of the eye is detected by image recognition, the gaze is deemed to be rightward. When state 93 where the pupil is on the upper side of the eye is detected by image recognition, the gaze is deemed to be upward. When state 94 where the pupil is on the lower side of the eye is detected by image recognition, the gaze is deemed to be downward.

[0051] FIG. 13 shows an example of a display image generated by changing the display content of the superimposition image according to the line of sight of the photographer. For example, when the line of sight is directed toward the center, as shown in FIG. 11(B), a text image 86 indicating the name of the food and a text image 87 indicating food information are superimposed for each of the dishes. On the other hand, when the line of sight is in the upward direction, it is assumed that photographer P is paying attention to the upper part of the display image, and as shown in Fig. 13, a text image 86 of the food name and a text image 87 showing food information are superimposed on the plates of grilled salmon, tofu, and sauteed pork located at the top of the display image to display more detailed information. On the other hand, only text image 86 showing the food name is superimposed on the plates of rice, pickles, and miso soup located at the bottom of display image 58 and the plates of salad and custard pudding located in the center of display image 58 in the vertical direction to display simplified information. In this way, the display content of the superimposed image may be changed depending on the line of sight of photographer P.

[0052] <Fifth embodiment> In the above-described embodiment, an example was given in which a display image is generated by using the center of the left and right eyes as a viewpoint when the left and right eyes are open, and using the center of the other open eye as a viewpoint when one eye is closed. As described in this embodiment, the present technology can also be applied to binocular stereoscopic vision in which different images are presented to the left and right eyes by the display unit 4. Hereinafter, a description will be given with reference to FIG. 14.

[0053] FIG. 14 is a schematic diagram illustrating binocular stereoscopic vision. In this embodiment, the image generator 7 calculates the three-dimensional positions of the left eye viewpoint EL of the left eye 9L and the right eye viewpoint ER of the right eye 9R in the same manner as in the above embodiment, using the distance image obtained by the front camera 6. The center of the pupil of each eye is set as viewpoint E. Furthermore, the image generation unit 7 performs coordinate conversion processing and occlusion compensation processing using the left eye viewpoint EL and the right eye viewpoint ER in the same manner as in the above embodiment, and generates a left eye display image and a right eye display image.

[0054] To realize a stereoscopic display, there are a lenticular method in which a lenticular lens is combined with the display unit 4, a parallax barrier method using a parallax barrier, etc. Here, the parallax barrier method will be explained as an example. The parallax barrier is a slit array in which vertical slits are arranged in the horizontal direction, and can be fabricated on the display unit 4 by lithography or the like. The horizontal direction of travel of the light rays emitted from the display unit 4 can be controlled by restricting the horizontal direction of travel of the light rays with the slits of the parallax barrier. The multiple display pixels that make up the display unit 4 are arranged in a horizontal pattern in which display pixels that display images for the left eye and display pixels that display images for the right eye are alternately arranged.

[0055] In this way, by configuring the display device to be capable of stereoscopic display, the photographer P can further experience a high sense of immersion and realism in the AR world.

[0056] <Other configuration examples> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology. For example, in coordinate conversion processing, it is necessary to recognize the relative positional relationship between the position of an object or a viewpoint and the display device. In the above embodiment, an example was given in which the result of image recognition is used when recognizing the relative positional relationship, but in addition to this, position and orientation information of the display device 1 acquired from an IMU (inertial measurement unit) as a third sensor mounted on the display device 1 may also be used. This makes it possible to realize image display with higher response. Furthermore, self-position and orientation information of the display device estimated using SLAM (Simultaneous Localization and Mapping) may also be used.

[0057] In the above embodiment, the camera equipped with both the image sensor and the distance sensor is described as an example, but the image sensor and the distance sensor may be provided separately. In this case, the image sensor and the distance sensor are arranged close to each other. When the image sensor and the distance sensor are provided separately, the coordinate conversion process in the display image generation method is performed taking into account the positional relationship between the image sensor and the distance sensor. For example, in order to simplify the calculation, the image sensor of the rear camera is calibrated separately from the distance sensor of the rear camera, and the RGB values ​​are mapped to distance values ​​to obtain a point cloud, so that only the distance sensor coordinate system can be the object of calculation.

[0058] Also, for example, in the above embodiment, when calculating the three-dimensional position of the viewpoint, a distance image obtained from the ToF distance sensor of the front camera 6 is used, but this is not limited to this. The distance image, which is the sensing result obtained from the ToF distance sensor, contains both distance information and two-dimensional position information of the photographer's eyes. To calculate the three-dimensional position of the viewpoint, it is sufficient to have the distance information and the two-dimensional position information of the photographer's eyes. These pieces of information may be obtained by the same sensor, or each piece of information may be obtained by a different sensor. For example, a distance sensor such as a structured light sensor or a patterned stereo sensor may be used as the second sensor to acquire distance information and two-dimensional position information of the photographer's eyes. Alternatively, a stereo camera using two cameras (image sensors) may be used as the second sensor. A sensor for acquiring two-dimensional position information of the eyes may be provided separately from the sensor for acquiring distance information, and the second sensor for acquiring distance information and two-dimensional position information of the photographer's eyes may be composed of one or more sensors. For example, when the texture of an object in an image is small, the ToF method produces fewer occlusion areas than the structured light sensor, the patterned stereo sensor, and the stereo camera. For this reason, it is more preferable to use a ToF distance sensor in order to stably generate a display image that does not look unnatural.

[0059] In addition, for example, in the above embodiment, when calculating the three-dimensional position of the viewpoint, an example has been given in which face detection and two-dimensional positions of the pupils of the left and right eyes are detected using a distance image acquired by the front camera 6, but the present invention is not limited to this. Face detection and two-dimensional positions of the pupils of the left and right eyes may also be detected using an RGB image acquired by the image sensor of the front camera 6.

[0060] In the above embodiment, the virtual object and the character image are exemplified as the superimposition image, but the superimposition image is not limited thereto. For example, an object region or the like cut out from an actually captured image may be used as the superimposition image, or the cut out object region or the like may be processed to change the color or the like as the superimposition image.

[0061] This technology can also be applied when displaying virtual objects fixed in a room using AR, allowing the photographer to experience a more realistic sense of the size of the virtual objects.

[0062] The present technology can also be configured as follows. (1) A first image sensor disposed on a first surface side of the device body; A first distance sensor disposed on the first surface side; a second sensor disposed on a second surface side opposite to the first surface; A display unit disposed on the second surface side; an image generating unit that generates a display image to be displayed on the display unit using a two-dimensional image of the subject acquired by the first image sensor and a distance image of the subject acquired by the first distance sensor, based on three-dimensional position information of the viewpoint of the photographer calculated based on the sensing result acquired by the second sensor. A display device comprising: (2) The display device according to (1) above, The image generating unit performs coordinate conversion on the point cloud of the subject obtained from the two-dimensional image and the distance image to generate the display image seen from the viewpoint of the photographer. Display device. (3) The display device according to (2) above, The image generating unit calculates the coordinates of intersections between a straight line from the viewpoint of the photographer to each point of the point cloud of the subject and the plane of the display unit, with the three-dimensional position of the viewpoint of the photographer as the origin, and generates the display image by performing coordinate transformation of the point cloud so that each point of the point cloud of the subject is mapped to the corresponding coordinates of the intersections. Display device. (4) The display device according to (2) or (3), The image generating unit generates the display image by complementing an occlusion area in the image seen from the viewpoint of the photographer, which is generated by performing coordinate transformation. Display device. (5) The display device according to any one of (2) to (4) above, A third sensor is further provided for acquiring position and orientation information of the display device, The image generating unit generates the display image using the position and orientation information. Display device. (6) The display device according to any one of (2) to (5) above, When calculating three-dimensional position information of the viewpoint of the photographer from the sensing result of the second sensor, the image generating unit calculates the three-dimensional position information of the viewpoint according to an open / closed state of the photographer's eyes. Display device. (7) The display device according to (6) above, The image generating unit calculates the three-dimensional position information using the open eye as a viewpoint when one eye of the photographer is closed, and using the center of a line segment connecting both eyes as a viewpoint when both eyes of the photographer are open. Display device. (8) The display device according to any one of (2) to (7) above, The image generating unit generates the display image using a right-eye image and a left-eye image generated with the positions of the photographer's left and right eyes as viewpoints. Display device. (9) The display device according to any one of (2) to (8), The image generating unit generates the display image on which the image to be superimposed is superimposed. Display device. (10) The display device according to (9) above, The image generating section generates the display image by superimposing an image for superimposition, the display content of which is changed according to the distance between the viewpoint of the photographer and the display section. Display device. (11) The display device according to (9) above, The image generating section generates the display image on which an image for superimposition, the display content of which is changed in accordance with the line of sight of the photographer, is superimposed. Display device. (12) The display device according to any one of (1) to (11) above, The sensing result obtained by the second sensor includes distance information and two-dimensional position information of the photographer's eyes. Display device. (13) The display device according to (12) above, The second sensor is a ToF (Time of Flight) sensor. Display device. (14) a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface side, and a display unit arranged on the second surface side, acquiring a two-dimensional image of a subject from the first image sensor; acquiring a distance image of the subject from the first distance sensor; Obtaining a sensing result of the second sensor; Calculating three-dimensional position information of the photographer's viewpoint based on the sensing result; A display image to be displayed on the display unit is generated using the two-dimensional image and the distance image based on the three-dimensional position information. Image generation method. (15) A step of acquiring a two-dimensional image of a subject from the first image sensor of a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface, and a display unit arranged on the second surface side; acquiring a distance image of the object from the first distance sensor; acquiring a sensing result of the second sensor; calculating three-dimensional position information of the viewpoint of the photographer based on the sensing result; generating a display image to be displayed on the display unit using the two-dimensional image and the distance image based on the three-dimensional position information; A program that causes a display device to execute the above. [Explanation of symbols]

[0063] 1...Display device 2. First side 3...Rear camera (first image sensor, first distance sensor) 4...Display section 5. The second side 6…Front camera (second sensor) 7...Image generation section E…Perspective P…Photographer 9L…Left eye 9R…Right eye 11...Subject 51, 53, 54-58…Display image 81…Occlusion area 85...Virtual object (image for overlay) 86, 87...Text image (overlay image)

Claims

1. A first image sensor disposed on a first surface side of the device body; A first distance sensor disposed on the first surface side; a second sensor disposed on a second surface side opposite to the first surface; A display unit disposed on the second surface side; A display device comprising: acquiring a sensing result acquired by the second sensor, a two-dimensional image of the object acquired by the first image sensor, and a distance image of the object acquired by the first distance sensor; calculating a three-dimensional position of a viewpoint of a photographer holding the display device based on the sensing result; calculating coordinates of intersections between a straight line from the three-dimensional position of the viewpoint to each point of the point cloud of the subject obtained by using the two-dimensional image and the distance image and a plane of the display unit; converting the coordinates of each point of the point cloud of the subject so that each point is mapped to the corresponding coordinates of the intersection point, thereby generating a display image seen from the viewpoint to be displayed on the display unit; generating a superimposition image including text information describing an object in a display image seen from the viewpoint, the amount of the text information being varied in accordance with a line of sight of the photographer; A display image is generated by superimposing the generated image for superimposition on a display image seen from the viewpoint. Image Generation Section A display device comprising:

2. The display device according to claim 1 , The image generating unit complements an occlusion area in an image viewed from the viewpoint of the photographer, which is generated by performing coordinate transformation, to generate a display image viewed from the viewpoint. Display device.

3. The display device according to claim 1 , a third sensor for acquiring position and orientation information of the display device; The image generation unit generates a display image seen from the viewpoint using the position and orientation information. Display device.

4. The display device according to claim 1 , When calculating three-dimensional position information of the viewpoint of the photographer from the sensing result of the second sensor, the image generating unit calculates the three-dimensional position information of the viewpoint according to an open / closed state of the photographer's eyes. Display device.

5. The display device according to claim 4, The image generating unit calculates the three-dimensional position information using the open eye as a viewpoint when one eye of the photographer is closed, and using the center of a line segment connecting both eyes as a viewpoint when both eyes of the photographer are open. Display device.

6. The display device according to claim 1 , The image generating unit generates the display image using a right-eye image and a left-eye image generated with the positions of the photographer's left and right eyes as viewpoints. Display device.

7. The display device according to claim 1 , The sensing result acquired by the second sensor includes distance information and two-dimensional position information of the photographer's eyes. Display device.

8. The display device according to claim 7, The second sensor is a Time of Flight (ToF) sensor. Display device.

9. a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface, and a display unit arranged on the second surface side, acquiring a two-dimensional image of a subject from the first image sensor; acquiring a distance image of the subject from the first distance sensor; Obtaining a sensing result of the second sensor; calculating a three-dimensional position of a viewpoint of a photographer holding the display device based on the sensing result; calculating coordinates of intersections between a straight line from the three-dimensional position of the viewpoint to each point of the point cloud of the subject obtained by using the two-dimensional image and the distance image and a plane of the display unit; converting the coordinates of each point of the point cloud of the subject so that each point is mapped to the corresponding coordinates of the intersection point, thereby generating a display image seen from the viewpoint to be displayed on the display unit; generating a superimposition image including text information describing an object in a display image seen from the viewpoint, the amount of the text information being varied in accordance with a line of sight of the photographer; A display image is generated by superimposing the generated image for superimposition on a display image seen from the viewpoint. Image generation method.

10. A step of acquiring a two-dimensional image of a subject from a first image sensor of a display device including a first image sensor arranged on a first surface side of a device body, a first distance sensor arranged on the first surface side, a second sensor arranged on a second surface side opposite to the first surface, and a display unit arranged on the second surface side; acquiring a distance image of the object from the first distance sensor; acquiring a sensing result of the second sensor; calculating a three-dimensional position of a viewpoint of a photographer holding the display device based on the sensing result; calculating coordinates of intersections between a straight line from the three-dimensional position of the viewpoint to each point of the point cloud of the subject obtained by using the two-dimensional image and the distance image and a plane of the display unit; a step of converting coordinates of each point of the point cloud of the subject so that each point of the point cloud is mapped to the corresponding coordinates of the intersection point, and generating a display image seen from the viewpoint to be displayed on the display unit; generating a superimposition image including text information describing an object in a display image seen from the viewpoint, the superimposition image being generated by varying an amount of the text information in accordance with a line of sight of the photographer; generating a display image by superimposing the generated image for superimposition on a display image seen from the viewpoint; A program that causes a display device to execute the above.

Citation Information

Patent Citations

  • Window-type image pickup / Display device and two-way communication method using the same

    JP2002300602A

  • camera

    JP2005121838A

  • Image processing method, and image processor

    JP2006267026A

  • Device and method for presenting information, imaging apparatus, and computer program

    JP2009271732A

  • Portion detector and program

    JP2013015896A