Projection device, display system, and display method

Abstract

To present information necessary for a user not through a display screen in a real space.SOLUTION: A projection device 1 has a camera 11 that photographs a real space 3 in which an object is arranged, and a projector 12 that projects an image indicated by video data generated based on an image photographed by the camera 11 in a photographing area R1 of the camera 11 in the real space 3. The video data is an indication image 2 such as an arrow or pointing indicating the object 5A arranged in the real space 3. The projection device 1 detects the object 5A from the photographed image through image recognition, and generates the video data for displaying the indication image 2 indicating the object 5A at a position superimposed on or adjacent to the object 5A. When an image indicated by the generated video data is projected from the projector 12, the indication image 2 is displayed at the position superimposed on or adjacent to the object 5A in the real space 3.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a projection device, a display system, and a display method. [Background technology]

[0002] Patent Document 1 describes a terminal device including an imaging unit that captures images of a real space and a display unit. The control unit of the terminal device executes an AR application to display on the display unit an output image in which a virtual object is superimposed on the input image, using the image captured by the imaging unit as an input image.

[0003] The terminal device in Patent Document 1 captures images of real space, detects objects in the captured images using image recognition, and displays virtual objects associated with the detected objects superimposed on the captured image. Examples of objects include floor plans of station signboards, exit guides, and timetables. The terminal device also includes an input unit such as a touch sensor, a mouse, and a module that recognizes gestures and gaze, and the movement of the virtual objects displayed on the display unit is controlled based on user input from the input unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-212345 Summary of the Invention [Problem to be solved by the invention]

[0005] The terminal device of Patent Document 1 uses a display unit of a terminal display to view a display in which a virtual object is superimposed on an image captured of a real space. The terminal device may be a smartphone, a PC, a game console, or the like. A head-mounted display (HMD) may also be used as the display unit, but in either case, the user must carry or wear some kind of device. Therefore, in order for multiple users to comfortably view the display, multiple terminal devices and HMDs are required, and each user must operate the device. Therefore, multiple users cannot easily share a display of a virtual object. [Means for solving the problem]

[0006] In order to solve the above problem, the projection device of the present invention is characterized by having a camera that captures an image of a real space in which an object is placed, and a projector that projects an image generated based on the image captured by the camera and containing information about the object into the camera's capture area in the real space.

[0007] The display system of the present invention comprises a first device equipped with a camera and a projector, and a second device capable of communicating with the first device, wherein the second device receives from the first device an image of a real space in which an object is placed, captured by the camera, and generates video data for projecting an image including information about the object based on the captured image, and the first device receives the video data from the second device and projects the image into the camera's capture area in the real space using the projector.

[0008] The display method of the present invention is characterized by comprising a first step of photographing a real space in which an object is placed using a camera, a second step of generating video data for projecting an image containing information about the object based on the image photographed in the first step, and a third step of projecting the image using a projector into the camera's photographing area in the real space. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a flowchart of a display method to which the present invention is applied. [Figure 2] FIG. 1 is a plan view schematically illustrating a state in which the projection device of the first embodiment is installed in a real space. [Figure 3] 1 is a block diagram showing the functional configuration of a projection apparatus according to a first embodiment. [Figure 4] FIG. 10 is a block diagram showing the functional configuration of a projection device in which the optical system of the camera and the projection optical system of the projector are made common. [Figure 5] FIG. 10 is a block diagram showing the functional configuration of a projection device 1 in which an imaging element of a camera is arranged in an image forming section of a projector. [Figure 6] FIG. 2 is a front view showing a photographing area of ​​a camera. [Figure 7] FIG. 10 is a front view showing a projection area onto which an instruction image is projected. [Figure 8] 5 is a flowchart showing a specific example of a display method by the projection device of the first embodiment. [Figure 9] FIG. 10 is a block diagram showing the functional configuration of a projection device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of a display system according to a third embodiment. [Figure 11] FIG. 10 is a block diagram showing the functional configuration of a display system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A projection device, a display system, and a display method according to embodiments of the present invention will be described below with reference to the accompanying drawings.

[0011] 1.Display method FIG. 1 is a flowchart of a display method to which the present invention is applied. First, an overview of the display method to which the present invention is applied will be described with reference to FIG. 1. As will be described later, a projection device and a display system to which the present invention is applied include a camera that captures a real space and a projector that projects an image containing a video into the real space. The display method to which the present invention is applied displays various images in the real space by projecting an image containing a video into the real space. In this case, the projected image is generated each time based on an image captured of the real space. The image to be displayed can be any image. For example, a figure, a picture, or text can be displayed. The display form may be a still image or a moving image.

[0012] As shown in FIG. 1, a display method to which the present invention is applied includes steps ST11 (first step), ST12 (second step), and ST13 (third step). In step ST11, a real space in which an object is placed is captured by a camera of a projection device or a display system. In step ST12, video data for displaying an image containing information about the object in real space is generated based on the image captured by the camera in step ST11. In step ST13, an image represented by the video data generated in step ST12 is projected from a projector of the projection device or the display system onto the capture area of ​​the camera in real space. This displays an image containing information about the object in real space. The projection device synchronizes the timing of projection by the projector with the timing of capture by the camera. In the case of a stationary projection device, synchronizing the timing of capture and projection so that they are different makes it easier to detect an object and also suppresses degradation in the quality of the projected image.

[0013] In step ST12, an object is detected from the captured image in real space by image recognition, and video data is generated for projecting an image that presents information about the object to the user. In the embodiment 1 described below, video data is generated for projecting an instruction image that indicates the position of the detected object into real space. In this case, in step ST12, the position at which the instruction image is displayed is determined based on the position of the object in the captured area. For example, the projection position of the instruction image is set to a position adjacent to the detected object or a position superimposed on the detected object. Furthermore, step ST11 can be triggered by input information about the object. In this case, in step ST12, the object is detected from the captured image based on the input information.

[0014] In step ST13, the projector projects an image represented by the generated video data onto a region overlapping the camera's image capture region in real space, thereby displaying an image for presenting information to the user directly on the surface of the object in real space, allowing the user to view the image together with the object itself.

[0015] Conventional augmented reality (AR) display methods present information to users by superimposing various images on a real-space image displayed on a device screen. However, the display method of the present invention presents information to users by directly superimposing an image on the surface of an object in real space. Therefore, users can directly view the image displayed on the surface of an object in real space without using a display screen. This method does not require a display device or the wear of a display device, as is the case with conventional AR display methods, and therefore multiple users can easily share information about the real space. Furthermore, because the display image is generated on a case-by-case basis based on a captured image of the real space, it is possible to display information corresponding to the actual location of objects in real space, even without map information that previously registers the location of objects in real space. This provides high convenience. Furthermore, accurate and effective display is possible.

[0016] 2.Projection equipment (Embodiment 1) FIG. 2 is a plan view schematically showing a state in which the projection device 1 of embodiment 1 is installed in a real space 3. In the drawings of this specification, the three directions X, Y, and Z are mutually orthogonal. The X direction is the left-right direction (width direction) of the real space 3, the Y direction is the front-back direction (depth direction) of the real space, and the Z direction is the up-down direction (height direction) of the real space 3. The projection device 1 may be installed in the real space 3 for use, or may be a portable device that is carried and used by a user 4. FIG. 2 shows an example in which the projection device 1 is installed in the real space 3.

[0017] FIG. 3 is a block diagram showing the functional configuration of the projection device 1 of the first embodiment. The projection device 1 includes a camera 11, a projector 12, an input unit 13, and a video data generation unit 14. The camera 11 includes an image sensor 111 and an optical system 112, and captures an image of the real space 3 and outputs the captured image. The projector 12 includes a projection optical system 122 and an image forming unit 121, and projects the video data generated by the video data generation unit 14. The input unit 13 accepts input from the user 4. For example, the input unit 13 may be an input device for voice input, a touch panel, a pen tablet, or the like. Alternatively, the input unit 13 may be any of various input means provided in a general-purpose terminal device.

[0018] The image captured by camera 11 and the input information input from input unit 13 are input to video data generation unit 14. The function of video data generation unit 14 is realized by a processor such as a CPU executing a display program read from a storage unit such as a non-volatile memory or HDD. Video data generation unit 14 generates video data based on the input information from input unit 13 and the image captured by camera 11. Video data generation unit 14 determines the position in real space 3 where an image should be displayed for user 4, and generates video data in which the image to be displayed is positioned at the determined position. Image formation unit 121 forms an image represented by the generated video data. The image formed by image formation unit 121 is enlarged and projected by projection optical system 122, and is focused on the projection target in real space 3, thereby displaying the image. Note that in this embodiment, the display of this image may also be simply described as "projector 12 projects video data."

[0019] For example, the video data generation unit 14 determines the display position where the image should be displayed within the shooting area R1 of the camera 11 based on the image captured by the camera 11, converts the display position within the shooting area R1 to a display position within the projection area R2 of the projector 12, and generates video data in which the image is placed at the display position within the projection area R2.

[0020] As shown in FIG. 3, in the projection device 1, the imaging angle of view θ1 of the camera 11 and the projection angle of view θ2 of the projector 12 are substantially the same. The camera 11 and the projector 12 are disposed facing the same side. In the first embodiment, the imaging area R1 of the camera 11 (see FIG. 6) and the projection area R2 of the projector 12 (see FIG. 7) are configured to substantially coincide with each other. Therefore, the projector 12 projects an image onto the imaging area R1 of the camera 11. Here, in the example shown in FIG. 2, the camera 11 and the projector 12 are arranged side by side, but the camera 11 and the projector 12 may be stacked one above the other. The projection device 1 is used at a position where the distance to an object placed in the real space 3 is within the projection distance of the projector 12. With this arrangement, the image projected from the projector 12 is formed on the surface of the object in the real space 3.

[0021] Furthermore, by sharing the optical system 112 of the camera 11 and the projection optical system 122 of the projector 12, the shooting angle of view θ1 and the projection angle of view θ2 can be made the same, and the shooting area R1 and the projection area R2 can be made to coincide. FIG. 4 is a block diagram showing the functional configuration of the projection device 1 when the optical system 112 of the camera 11 and the projection optical system 122 of the projector 12 are shared. In the configuration of FIG. 4, a separation optical element 110 causes light from the shared optical systems 112 and 122 to enter the image sensor 111, and light from the image forming unit 121 to exit the optical systems 112 and 122. FIG. 5 is a block diagram showing the functional configuration of the projection device 1 when the image sensor 111 is disposed in the image forming unit 121 of the projector 12. In the configuration of FIG. 5, the image forming unit 121 is formed of a self-luminous imager equipped with light-emitting elements, and the image sensor 111 and the light-emitting elements are stacked or formed on the same surface in the image forming unit 121.

[0022] (Image display in real space) FIG. 6 is a front view showing the imaging area R1 captured by the camera 11. FIG. 7 is a front view showing the projection area R2 onto which the instruction image 2 is projected. FIGS. 6 and 7 are front views of the real space 3 shown in FIG. 2 as seen from the user 4 side. The projection device 1 displays, in the real space 3, an instruction image 2 indicating the position of an object placed in the real space 3. As shown in FIG. 6, objects 5, 6, and 7 are placed in the imaging area R1 of the camera 11 in the real space 3. In the examples shown in FIGS. 2, 6, and 7, the object 5 is a book, and the object 6 is a bookshelf. An object 7 other than a book and a bookshelf is also placed. A plurality of books (objects 5) are lined up on the bookshelf (object 6).

[0023] 7, the pointing image 2 is an image of an arrow, and is displayed in a manner that points to one of a plurality of books (hereinafter referred to as object 5A). Specifically, the pointing image 2 is displayed in a position adjacent to the object 5A with its tip pointing toward the object 5A.

[0024] The type of image to be used as the instruction image 2 indicating the position of the object 5A is set in advance in a display program executed by the video data generation unit 14. The video data generation unit 14 reads data of the image to be used as the instruction image 2 from a storage unit and generates video data. The video data generation unit 14 determines the orientation and display position of the instruction image 2 (arrow) based on input information output from the input unit 13 and the image captured by the camera 11. Then, it generates video data in which the instruction image 2 is placed at the determined display position.

[0025] For example, when a book title "ABC" is input as input information, the video data generation unit 14 uses image recognition to detect a book with a title matching the input information from among images captured by the camera 11. In the example shown in FIG. 6, object 5A is detected as a book with a title matching the input information. Then, video data is generated in which an instruction image 2 is placed in a position pointing to the detected object 5A. For example, video data is generated in which an instruction image 2 with its tip pointing toward object 5A is placed in a position adjacent to object 5A. At this time, the video data is generated on the assumption that the shooting area R1 and the projection area R2 approximately match.

[0026] Fig. 8 is a flowchart showing a specific example of a display method by the projection device 1 of embodiment 1. The flowchart in Fig. 8 includes a first step (step ST22) of capturing an image of the real space 3 with the camera 11, a second step (steps ST23 and ST24) of generating video data for displaying an image (instruction image 2) containing information about the object 5A in the real space 3 based on the captured image, and a third step (step ST25) of projecting an image indicated by the video data generated in the second step from the projector 12 into the capturing region R1 of the camera 11 in the real space 3 to display the image (instruction image 2) containing information about the object 5A.

[0027] In the display method of FIG. 8, first, in step ST21, input of input information related to an object is accepted from the input unit 13. In step ST22, the input of the input information from the input unit 13 is used as a trigger to capture an image of the real space 3 by the camera 11. Next, in step ST23, the title "ABC" is searched for in the captured image by image recognition, and the object 5A is detected. In step ST24, video data is generated in which an instruction image 2 is placed at a position pointing to the detected object 5A. Subsequently, in step ST25, the image indicated by the video data generated in step ST24 is projected by the projector 12 onto the capture area R1 of the camera 11 in the real space 3. As a result, the instruction image 2 pointing to the object 5A is displayed in the position pointing to the object 5A in the real space 3.

[0028] (Major Functions and Effects of the Projection Device of Embodiment 1) As described above, the projection device 1 of embodiment 1 includes a camera 11 that captures an image of a real space 3 in which an object is placed, and a projector 12 that projects an image generated based on the image captured by the camera 11 and including information about the object into the capture area R1 of the camera 11 in the real space 3.

[0029] The projection device 1 of the first embodiment executes the steps shown in FIG. 8 to display an image in the real space 3. As a result, the user 4 can directly view the image presented on the surface of an object in the real space 3 without using a display device. Therefore, unlike conventional AR displays, a display device or the wearing of such a device is not required, and multiple users 4 can easily share information about objects in the real space 3. Furthermore, since the image to be displayed is generated each time based on an image captured of the real space 3, it is possible to display an image corresponding to the actual arrangement of objects in the real space 3 even without map information that registers the arrangement of objects in the real space 3 in advance. This provides high convenience. Furthermore, accurate and effective display can be achieved.

[0030] In the projection device of embodiment 1, the projection angle of view θ2 of the projector 12 matches the shooting angle of view θ1 of the camera 11. When generating video data for the projector 12 based on the image captured by the camera 11, the video data can be generated on the premise that the shooting area R1 and the projection area R2 match, making it easy to generate the video data. In addition, by matching the shooting area R1 of the camera 11 and the projection area R2 of the projector 12, it is possible to present the user with information about all objects that have been detected in the image captured by the camera 11.

[0031] 4 and 5, the projection device 1 of the first embodiment can employ a configuration in which the projection optical system 122 of the projector 12 and the optical system 112 of the camera 11 are common to each other. This allows the imaging angle of view of the camera 11 and the projection angle of view of the projector 12 to be the same, and allows the imaging area R1 and the projection area R2 to coincide with each other.

[0032] In the projection device 1 of the first embodiment, the image projected by the projector 12 includes an instruction image 2 that indicates an object 5A placed in the shooting area R1. By displaying the instruction image 2, it is possible to provide the user 4 with information that is useful for finding the object 5A. In the first embodiment, as shown in FIG. 7, the instruction image 2 is projected at a position adjacent to the object 5A. Therefore, the object 5A can be easily found.

[0033] The projection device 1 of the first embodiment includes a video data generation unit 14 that generates video data for projecting an image. The video data generation unit 14 performs steps ST23 and ST24 in FIG. 8. That is, the video data generation unit 14 detects the position of the object 5A in the shooting area R1 of the camera 11, and based on the detected position, determines the position at which to project the instruction image 2, thereby generating video data. For example, the video data generation unit 14 generates video data for projecting the instruction image 2 at a position superimposed on or adjacent to the object 5A. By including the video data generation unit 14 in the projection device 1, the projection device 1 can generate the necessary video data and project an image by itself. Therefore, the projection device 1 can be used even in places without a communication network. Furthermore, since the position of the object 5A is confirmed from the shot image and the display position of the instruction image 2 is determined, the display position of the instruction image 2 is accurate.

[0034] The projection device 1 of the first embodiment includes an input unit 13 that accepts input of information used to generate video data. In steps ST23 and ST24 of Fig. 8, the video data generation unit 14 uses the input information input from the input unit 13 to detect an object 5A from the image captured by the camera 11 through image recognition. This allows the user 4 to input information about a necessary object on the spot (for example, the title of a book) and display an instruction image 2 indicating the position of the necessary object, which is highly convenient.

[0035] The projection device 1 of the first embodiment includes a projector 12 equipped with a self-luminous imager. A self-luminous imager is a display panel with a light-emitting element for each pixel. This allows the projection device 1 to be made smaller, thereby reducing the installation space. The projection device 1 is also suitable for portable use.

[0036] (Embodiment 2) FIG. 9 is a block diagram showing the functional configuration of a projection device 1A according to a second embodiment. The projection device 1A includes a camera 11, a projector 12, a video data generation unit 14, and a communication unit 15. The projection device 1A communicates with a terminal device (not shown) via the communication unit 15 and a network 100. The projection device 1A is installed in the real space 3, as in the first embodiment. Alternatively, the projection device 1A can be carried by a user 4. The projection device 1A captures an image of the real space 3 in which an object is placed using the camera 11, triggered by receiving input information from the terminal device. The projection device 1A then generates video data based on the input information and the image captured by the camera 11, and projects an image represented by the video data into the real space 3. The image represented by the video data includes information about the object. The terminal device that transmits the input information can be a terminal device such as a smartphone or tablet owned by the user 4. Alternatively, the terminal device can be a device owned by an administrator who manages the real space 3.

[0037] The configurations of camera 11 and projector 12 are the same as those in embodiment 1. Also, video data generation unit 14 generates video data including instruction image 2 based on input information and an image captured by camera 11, as in embodiment 1. Therefore, instruction image 2 can be displayed in real space 3 in the same manner as in embodiment 1, and therefore the same effects as in embodiment 1 can be obtained.

[0038] In the second embodiment, when input information is input from the terminal device of the user 4, the user 4 can input information about the object to be found and display the instruction image 2, as in the first embodiment. On the other hand, when input information is input from a terminal device located in a different location from the real space 3, information about the location of the object that the user 4 is to recognize can be remotely displayed from the projection device 1. For example, if the real space 3 is a warehouse and the user 4 is a worker performing picking work in the warehouse, the manager can input information about the object to be picked from an external terminal device and display the location of the object to be picked in the warehouse. In this case, the external terminal device may be controlled by a computer rather than a human manager.

[0039] (Aspects of the instruction image) 7, an arrow is used as the pointing image 2, but a pointing finger image can be used instead of the arrow. When using a pointing finger image, the pointing finger image is placed next to the object 5A with the tip of the finger pointing toward the object 5A, just like an arrow. Alternatively, an image of a character pointing at the object 5A can be used as the pointing image.

[0040] Furthermore, the instruction image 2 is not limited to being displayed in a position adjacent to the object 5A. For example, an image superimposed on the object 5A may be displayed to indicate the position of the object 5A. For example, a figure having the same shape as the object 5A may be superimposed on the object 5A as the instruction image. Alternatively, a frame-shaped figure surrounding the outline of the object 5A may be superimposed on the object 5A as the instruction image.

[0041] The instruction image 2 may be a still image or a moving image. For example, a figure superimposed on the object 5A is made to blink in a predetermined pattern. This embodiment makes the instruction image stand out. Alternatively, the instruction image 2 may be a moving arrow that moves toward the object 5A and stops at a position pointing at the object 5A. This embodiment makes the instruction image 2 stand out, making it easier to find the location of the object 5A.

[0042] 6, the image projected by the projector 12 is only one instruction image 2, but two or more instruction images can also be displayed. For example, if the input information contains multiple book titles, the positions of the multiple books are detected and an instruction image 2 is displayed at each position. Alternatively, if multiple bookshelves are arranged in the real space, an image that highlights the bookshelf on which the books to be searched are lined may be displayed in combination with an instruction image 2 that points to the book.

[0043] If the object corresponding to the input information is not detected in the captured image, the video data generator 14 may display a message indicating that the object is not found. For example, the message may be displayed by projecting letters or marks. Therefore, the instruction image 2 includes not only an image such as an arrow, but also an image of additional information related to the object 5A represented by letters or marks.

[0044] (Display format for other uses) 7 displays the instruction image 2 to help the user 4 find the object 5A, but the projection devices 1, 1A can be used for other purposes. For example, the intrusion of a suspicious person can be detected by image recognition from an image captured by the camera 11, and a warning display can be displayed. For example, video data in which a strong light is shone on the suspicious person can be used as video data for the warning display, and a bright image can be projected onto the suspicious person.

[0045] The projection devices 1 and 1A can also be used to alert users. For example, a change in the location of an object is detected from the image captured by the camera 11, and if the object disappears, an alert image is displayed at that position. In this case, an instruction image may be displayed, or an alert image different from the instruction image may be used.

[0046] Alternatively, a dramatic display can be performed using the projection devices 1 and 1A. For example, the objects 5, 6, and 7 are detected from the image captured by the camera 11, and the display position of the dramatic image is determined according to the arrangement of the objects 5, 6, and 7 in the real space 3. Alternatively, a dramatic image can be displayed that moves according to the arrangement of the objects 5, 6, and 7. For example, a character image can be used as the dramatic image. cormorant.

[0047] 2 shows an embodiment of a stationary projection device 1, but by using a self-luminous imager for the image forming unit 121, the projection device 1 can be made smaller and more portable. In this case, it may not be possible to increase the projection angle of view of the projection device 1, but it is possible to easily match the projection angle of view with the capture angle of view, making it possible to display only the information about objects that are necessary for the user 4 in real space. In a portable projection device 1, the timing of image projection by the projector 12 and the timing of image capture by the camera 11 are synchronized, and if an object is included in the captured image generated each time the camera 11 captures an image, the projector 12 projects an image including information about the object.

[0048] For example, if the object is a product on display, the projection device 1, 1A can project additional information about the product. Furthermore, in the case of a portable projection device 1, additional information about only the product at which the projection device 1 is pointed by the user 4 can be displayed in real space.

[0049] Furthermore, if the object is an unassembled part, the projection devices 1 and 1A can project a video showing the assembly procedure for that part. If there are multiple parts, the portable projection device 1 can be used to select the part for which the video showing the assembly procedure is to be projected.

[0050] Furthermore, if the object is in the form of a question, the projection device 1, 1A can project the answer to the question. If there are multiple forms, the answer to the question can be selected and projected by the portable projection device 1. For example, "an object in the form of a question" means a piece of paper or the like on which letters or figures expressing a question are written.

[0051] (Display orientation adjustment function) In the projection devices 1 and 1A, the camera 11 and the projector 12 may be configured to be able to shift the shooting direction of the camera and the projection direction of the projector 12 while maintaining a state in which the shooting area R1 of the camera 11 and the projection area R2 of the projector 12 substantially coincide with each other. For example, the projection devices 1 and 1A may be provided with an adjustment mechanism that shifts the shooting direction of the camera and the projection direction of the projector 12 in conjunction with each other so as to maintain a state in which the shooting area R1 and the projection area R2 substantially coincide with each other. This allows display in an optimal range depending on the size of the real space 3 and the arrangement of objects in the real space 3.

[0052] 3. Display System (Embodiment 3) 10 is a block diagram showing the functional configuration of a display system 20 according to a third embodiment. The display system 20 includes a first device 21 and a second device 22. The first device 21 is a projection device installed in the real space 3, or a projection device carried by a user and used in the real space 3. The second device 22 is capable of communicating with the first device 21 via a network 100.

[0053] The first device 21 includes a camera 11, a projector 12, and a communication unit 15. The camera 11 and the projector 12 are configured in the same manner as in the first embodiment. The second device 22 includes an input unit 13, a video data generation unit 14, and a communication unit 16. For example, a general-purpose personal computer can be used as the second device 22. When a general-purpose personal computer is used, a keyboard or a mouse can be used as the input unit 13. Furthermore, as in the first embodiment, a voice input device, a touch panel, a pen tablet, or the like can be used as the input unit 13.

[0054] As in the above embodiments, the display system 20 can perform the following steps: a first step of capturing an image of the real space 3 in which an object is placed using the camera 11; a second step of generating video data for projecting an image including information about the object based on the captured image; and a third step of projecting an image indicated by the video data from the projector 12 onto the capturing area R1 of the camera 11 in the real space 3, thereby displaying the image in the real space 3. Specifically, the steps of the flowchart in Fig. 8 can be performed as follows.

[0055] In the display system 20, when input information is input from the input unit 13 to the second device 22, the second device 22 requests an image captured by the camera from the first device 21 via the communication unit 16. Based on this, the first device 21 captures an image of the real space 3 using the camera 11 and transmits the image captured by the camera 11 to the second device 22 via the communication unit 15. In the second device 22, the video data generation unit 14 generates video data for displaying an image including the instruction image 2 based on the input information input from the input unit 13 and the image captured by the camera 11. That is, based on the input information, the second device 22 detects an object 5A from the captured image by image recognition, and determines a position for projecting the instruction image 2 based on the position of the detected object 5A to generate video data. Next, the second device 22 transmits the video data to the first device 21 via the communication unit 16. In the first device 21, the projector 12 projects an image represented by the video data received from the second device 22 onto the capture area R1 of the camera 11 in the real space 3.

[0056] Therefore, the display system 20 of the third embodiment can display the instruction image 2 in the real space 3 in the same manner as in the first embodiment, and therefore can obtain the same operational effects as in the first embodiment. Also, like the first embodiment, various other display modes can be displayed, and the display system can be used for various purposes. Also, like the second embodiment, input information is input by the second device 22 that is separate from the first device 21 placed in the real space 3, and therefore, like the second embodiment, information that the user 4 wants to recognize can be displayed in the real space 3 by remote control.

[0057] (Embodiment 4) 11 is a block diagram showing the functional configuration of a display system 20A according to a fourth embodiment. The display system 20A includes a first device 21A and a second device 22A. The first device 21A is a projection device installed in a real space 3, or a projection device carried by a user. The second device 22A is capable of communicating with the first device 21A via a network 100.

[0058] First device 21A includes camera 11, projector 12, input unit 13, and communication unit 15. Camera 11, projector 12, and input unit 13 are configured in the same manner as in embodiment 1. Second device 22A includes video data generation unit 14 and communication unit 16. As in embodiment 3, a general-purpose personal computer can be used as second device 22A.

[0059] As in the above-described embodiments, the display system 20A can perform the following steps: a first step of capturing an image of the real space 3 in which an object is placed using the camera 11; a second step of generating video data for projecting an image including information about the object based on the captured image; and a third step of projecting an image indicated by the video data from the projector 12 onto the capturing area R1 of the camera 11 in the real space 3, thereby displaying the image in the real space 3. Specifically, the steps of the flowchart in FIG. 8 can be performed as follows.

[0060] In display system 20A, input of input information from input unit 13 in first device 21 is used as a trigger to capture an image of real space 3 with camera 11. First device 21 transmits the input information and the captured image to second device 22 via communication unit 15. In second device 22, video data generation unit 14 generates video data including instruction image 2 based on the input information and the captured image. That is, based on the input information, object 5A is detected from the captured image by image recognition, and based on the position of detected object 5A, a position for projecting instruction image 2 is determined and video data is generated. In first device 21, projector 12 projects an image indicated by the video data received from second device 22 onto capture area R1 of camera 11 in real space 3.

[0061] Therefore, the display system 20A of the fourth embodiment can display the instruction image 2 in the real space 3 in the same manner as in the first embodiment, and can therefore achieve the same operational effects as in the first embodiment. Also, like the first embodiment, various other display modes can be displayed, and the display system can be used for various purposes. Also, like the first embodiment, input information is input in the first device 21 placed in the real space 3, and thus, like the first embodiment, the user 4 can input information about a required object (for example, the title of a book) on the spot and display the instruction image 2 indicating the position of the required object, which is highly convenient. [Explanation of symbols]

[0062] 1, 1A...projection device, 2...instruction image, 3...real space, 4...user, 5, 5A, 6, 7...object, 11...camera, 12...projector, 13...input unit, 14...video data generation unit, 15, 16...communication unit, 20, 20A...display system, 21, 21A...first device, 22, 22A...second device, 100...network, 110...separation optical element, 111...imaging element, 112...optical system, 121...image formation unit, 122...projection optical system, R1...shooting area, R2...projection area, θ1...shooting angle of view, θ2...projection angle of view.

Claims

1. a camera that captures a real space in which an object is placed; a projector that projects an image, which is generated based on an image captured by the camera and includes information about the object, into a capture area of ​​the camera in the real space; a video data generating unit that generates video data for projecting the image; an input unit that accepts input of input information related to the object; and the projection angle of the projector and the photographing angle of the camera are the same, the image includes an instruction image that indicates the object placed in the shooting area, the camera captures an image of the real space when input information is input to the input unit, the video data generation unit detects the object from the captured image based on the captured image by the camera, and determines a position to project the instruction image based on a position of the object in the captured area; the projector projects an image including the instruction image; The projection timing of the projector is different from the photographing timing of the camera, the projection optical system of the projector and the optical system of the camera are common; the projector includes a self-luminous imager having a light-emitting element for each pixel; the camera has an image sensor; The projection device is characterized in that the imaging element is formed on the same surface as or stacked with the light emitting element.

2. A first device includes a camera and a projector, and a second device is capable of communicating with the first device, the second device receives from the first device a captured image of a real space in which an object is placed, captured by the camera, and generates video data for projecting an image including information about the object based on the captured image; the first device receives the video data from the second device, and projects the image onto a capture area of ​​the camera in the real space using the projector; the projection angle of the projector and the photographing angle of the camera are the same, the image includes an instruction image indicating an object placed in the shooting area, the second device includes a video data generation unit that generates the video data; an input unit that receives input of input information related to the object is provided in the first device or the second device; the camera captures an image of the real space when input information is input to the input unit, the video data generation unit detects the object from the captured image based on the captured image by the camera, and determines a position to project the instruction image based on a position of the object in the captured area; the projector projects an image including the instruction image; The projection timing of the projector is different from the photographing timing of the camera, the projection optical system of the projector and the optical system of the camera are common; the projector includes a self-luminous imager having a light-emitting element for each pixel; the camera has an image sensor; The display system is characterized in that the imaging element is stacked on or formed on the same surface as the light emitting element.

Images