Virtual image display device, imaging device, virtual image display system and method
By introducing virtual image display devices and AR technology into the camera device, the overlapping display of virtual images and the real world is realized, and the problem that the prior art cannot display virtual images is solved, and the photography experience and operational nature are improved.
Patent Information
- Application Number
- CN202380030955.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-28
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art cannot display virtual images in existing imaging devices, and in particular, it cannot realize effective overlapping display of virtual images and the real world.
By designing a virtual image display device, the device includes a processor and a display, it can receive instant preview images and perform geometric transformations, so that the virtual image coincides with the display area of the camera device, and displays with glasses-type or goggles-type wearable devices. The virtual image is combined with AR technology, and the photographic image of the imaging device is analyzed to determine the display area, and the three-dimensional position and posture information of the virtual camera are obtained for transformation.
It realizes high-quality overlapping display of virtual images and the real world, enhances the photography experience, especially when moving photography, which can maintain a clear preview image display, improving the naturalness and convenience of operation.
Smart Images

Figure CN119096554A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a virtual image display device, a camera device, a virtual image display system and a method, and in particular to a technology for making a virtual image overlap and be displayed on the real world. Background Art
[0002] Conventionally, there has been proposed a method for performing photography in a virtual space and making it possible to easily check a photograph (image) generated by the photography (Patent Document 1).
[0003] In the method described in Patent Document 1, a virtual camera is arranged at the center of a virtual space provided using a head-mounted device (HMD), an image corresponding to the photographic range of the virtual camera is generated according to the position and posture of the virtual camera, and the generated image is displayed on a preview screen of the virtual camera. Furthermore, if photography is performed in the virtual space according to a photography instruction from the user, a photo image is generated and stored in a memory, and the generated photo image is arranged at a predetermined position in the virtual space, and the position where the photo image is arranged is notified to the user wearing the HMD.
[0004] The virtual camera described in Patent Document 1 is a camera object generated and configured in a virtual space based on object information stored in a memory module. If a button set on the camera object is pressed by a hand object corresponding to the hand part of an avatar object, photography in the virtual space is performed. In addition, the hand object corresponding to the hand part of the avatar object moves in a manner linked to the position of the user's hand. In Patent Document 1, an HMD device is used to provide a photographic image of a virtual camera, but methods such as transmissive smart glasses and retinal projection can also be considered.
[0005] Furthermore, Patent Document 2 proposes an information processing device that effectively applies the technology of augmented reality (AR) that merges the real world and the virtual world to games and the like.
[0006] The information processing device described in Patent Document 2 acquires a photographic image from a rear camera of a photographic real space, and if a mark included in the photographic image is detected, information processing (game) corresponding to the mark is started, and a virtual object of a character or icon corresponding to the mark is arranged in a three-dimensional coordinate system corresponding to the photographed space and depicted on the photographic image, and displayed on a display device on the surface. The icon is displayed in a manner of being placed on the plane where the mark is placed, and can be operated based on either contact with the touch panel on the display screen or indication of a corresponding position in the photographed space by a finger.
[0007] Previous technical literature
[0008] Patent Literature
[0009] Patent Document 1: Japanese Patent Application Publication No. 2019-21236
[0010] Patent Document 2: Japanese Patent Application Publication No. 2015-41126 Summary of the invention
[0011] Technical issues to be solved by the invention
[0012] An embodiment of the technology involved in the present invention provides a virtual image display device, a camera device, a virtual image display system and a method. The virtual image display device makes the virtual image overlap with the real world and displays an image that cannot be displayed in an existing camera device alone.
[0013] Means for solving technical problems
[0014] The invention involved in the first embodiment is a virtual image display device, which has a processor and a first display. The virtual image display device receives a real-time preview image from a first camera device, and the processor performs the following processing: obtaining information used when making the real-time preview image overlap with the display area of the first camera device; and transforming the real-time preview image according to the information, and the first display displays the transformed real-time preview image.
[0015] In the virtual image display device according to the second aspect of the present invention, preferably, the first display is a glasses-type or goggle-type wearable device having a virtual image display function.
[0016] In the virtual image display device according to the third aspect of the present invention, the second camera device is provided to capture the user's field of view, and the processor analyzes the image captured by the second camera device to determine the display area and acquire information.
[0017] In the virtual image display device according to the fourth aspect of the present invention, the processor acquires information based on the position of the mark of the first imaging device photographed by the second imaging device.
[0018] In the virtual image display device according to the fifth aspect of the present invention, the processor acquires information indicating the three-dimensional position and posture of the first imaging device with respect to the first display, and acquires information based on the information indicating the three-dimensional position and posture.
[0019] In the virtual image display device according to the sixth aspect of the present invention, preferably, the first imaging device includes a second display for displaying a live preview image, and the display area is a display area of the second display.
[0020] In the virtual image display device according to the seventh aspect of the present invention, preferably, the second display has a rotation mechanism capable of changing a relative posture relationship with a device body of the first camera device by rotation, and the display area is the display area of the second display after rotation.
[0021] In the virtual image display device according to the eighth aspect of the present invention, preferably, the first imaging device includes a viewfinder, and the display area is a display area of the viewfinder.
[0022] In the virtual image display device according to the ninth aspect of the present invention, preferably, the first display is a display that transmits a visual field and displays a live preview image converted into a visual field of the real world.
[0023] In the virtual image display device involved in the 10th mode of the present invention, it is preferred that the second camera device has a field of view of the photographing user, and the first display is an immersive display covering the field of view, which displays a composite image in which the transformed real-time preview image overlaps with the photographic image photographed by the second camera device.
[0024] In the virtual image display device according to the eleventh aspect of the present invention, preferably, the processor combines the virtual object image with the live preview image.
[0025] The invention involved in the 12th embodiment of the present invention is a virtual image display device, which has a processor, a first display and a second camera device for photographing the field of view of the user, and the processor performs the following processing: cutting out an image corresponding to the field of view of a virtual camera having at least a handle portion from the image photographed by the second camera device and obtaining it as an instant preview image; obtaining information used when making the instant preview image overlap with the display area of the virtual camera; transforming the instant preview image according to the information; and displaying the transformed instant preview image on the first display.
[0026] In the virtual image display device according to the thirteenth aspect of the present invention, the transformation is preferably a geometric transformation.
[0027] The invention involved in the 14th mode is a camera device, which comprises: a camera unit, which shoots a real-time preview image; a mode switching unit, which switches between a first mode and a second mode; a second display or viewfinder, which displays the real-time preview image when switched to the first mode; and a communication unit, which sends the real-time preview image to a virtual image display device of any one of the first to 11th modes when switched to the second mode.
[0028] In the imaging device according to the fifteenth aspect of the present invention, preferably, when switching to the second mode, a mark is displayed on the second display or the viewfinder.
[0029] The invention according to the sixteenth aspect is a virtual image display system, which is composed of the virtual image display device according to any one of the first to thirteenth aspects and the imaging device according to the fourteenth or fifteenth aspect.
[0030] In the virtual image display system involved in the 17th embodiment of the present invention, the processor of the virtual image display device performs the following processing: synthesizing the virtual object image with the real-time preview image; and if photography for recording is performed in the state switched to the second mode, the virtual object image is sent to the camera device, and if the camera device performs photography for recording in the state switched to the second mode, the photographed image is recorded in association with the virtual object image received from the virtual image display device, or the virtual object image is synthesized with the photographed image for recording.
[0031] The invention involved in the 18th mode is a virtual image display system, which is composed of a virtual image display device of any one of the 1st mode to the 11th mode and a server communicating with the virtual image display device, the server manages a three-dimensional image with three-dimensional information of a real space, and the processor performs the following processing: based on the first information representing the three-dimensional position and posture of the first camera device, cutting out a three-dimensional image corresponding to the photographic area of the first camera device from the three-dimensional image managed by the server; and converting the cut-out three-dimensional image into a two-dimensional image, and using the two-dimensional image as an instant preview image.
[0032] The invention involved in the 19th mode is a virtual image display system, which is composed of the virtual image display device of the 12th mode and a server communicating with the virtual image display device, the server manages a three-dimensional image with three-dimensional information of a real space, and the processor performs the following processing: based on the first information representing the three-dimensional position and posture of the virtual camera, a three-dimensional image corresponding to the photographic area of the virtual camera is cut out from the three-dimensional image managed by the server; and the cut-out three-dimensional image is converted into a two-dimensional image, and the two-dimensional image is used as an instant preview image.
[0033] In the virtual image display system involved in the 20th mode of the present invention, it is preferred that the server generates and manages the latest three-dimensional image based on images photographed by multiple cameras including a multi-viewpoint camera, a surveillance camera, a driving recorder, an unmanned camera or a mobile terminal camera arranged in a real space.
[0034] In the virtual image display system according to the twenty-first aspect of the present invention, preferably, the server synthesizes the virtual object image with the three-dimensional image.
[0035] The invention involved in the 22nd embodiment is a virtual image display method, which is executed by a virtual image display device having a processor and a first display, and the virtual image display method includes the following steps: the processor receives a real-time preview image from the first camera device; the processor obtains information used when making the real-time preview image overlap with the display area of the first camera device; the processor transforms the real-time preview image according to the information; and the first display displays the transformed real-time preview image. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a diagram showing the system configuration of the virtual image display system involved in the present invention.
[0037] Figure 2 Observed from the front Figure 1 A perspective view of the camera device shown.
[0038] Figure 3 The diagram shows three open and closed states of the monitor panel of the imaging device.
[0039] Figure 4 This is a diagram showing a situation where an image including the field of view of the imaging device is captured by the second imaging device of the virtual image display device.
[0040] Figure 5 This is a diagram showing a state where a live preview image is superimposed on a specific display area of an imaging device.
[0041] Figure 6 It is a block diagram showing an embodiment of a virtual image display device constituting the virtual image display system according to the present invention.
[0042] Figure 7 It is a block diagram showing an embodiment of the imaging device according to the present invention.
[0043] Figure 8 This is a block diagram showing a main part of a first embodiment of a virtual image display system according to the present invention.
[0044] Fig. 9 A diagram showing the geometric relationship between a live preview image, a field of view image, and a display area of an imaging device.
[0045] Fig.10 This is a diagram showing the appearance of an imaging device provided with an AR marker.
[0046] Fig.11 It is a diagram showing a first embodiment of an XR display of a virtual image display device.
[0047] Fig.12 This is a diagram showing a first display example of a live preview image and the like when the imaging device is set to the XR mode.
[0048] Fig.13 This is a diagram showing a second display example of a live preview image when the imaging device is set to the XR mode.
[0049] Fig.14 This is a diagram showing a third display example of a live preview image when the imaging device is set to the XR mode.
[0050] Fig.15 This is a diagram showing a fourth display example of a live preview image when the imaging device is set to the XR mode.
[0051] Fig.16 This is a diagram showing a fifth display example of a live preview image when the imaging device is set to the XR mode.
[0052] Fig.17 This is a diagram showing a fifth display example of a live preview image when the imaging device is set to the XR mode.
[0053] Fig.18 This is a diagram showing a sixth display example of a live preview image when the imaging device is set to the XR mode.
[0054] Fig.19 This is a diagram showing a first external appearance example of the imaging device when the imaging device is set to the XR mode.
[0055] Fig. 20 This is a diagram showing a second external appearance example of the imaging device when the imaging device is set to the XR mode.
[0056] Fig.21 This is a diagram showing a third external appearance example of the imaging device when the imaging device is set to the XR mode.
[0057] Fig. 22 This is a conceptual diagram when the appearance of the imaging device is divided into a plurality of areas and changed.
[0058] Fig.23 This is a stereoscopic diagram of a camera device suitable for use in an XR environment.
[0059] Fig.24 It is a perspective view showing the back side of an embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0060] Fig.25 It is a perspective view showing the front side of an embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0061] Fig.26 It is a perspective view showing another embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0062] Fig. 27 It is a diagram showing an implementation method of a method for acquiring a live preview image photographed by a virtual camera.
[0063] Fig.28 It is a diagram showing a second embodiment of the XR display of the virtual image display device.
[0064] Fig.29 It is a block diagram showing a second embodiment of the virtual image display system according to the present invention.
[0065] Fig.30 This is a flowchart showing a first embodiment of a virtual image display method according to the present invention.
[0066] Fig.31 This is a flowchart showing a display method based on the imaging device involved in the present invention.
[0067] Fig.32 This is a flowchart showing a second embodiment of the virtual image display method according to the present invention.
[0068] Fig.33 Yes means to proceed Fig.32 The illustrated embodiment is a subroutine of the process of step S300 for confirming the imaging device.
[0069] Fig.34 Yes means to proceed Fig.32 The subroutine of the embodiment of the process of step S3200 of overlapping display of the live preview image LV1 is shown.
[0070] Fig.35 Yes means to proceed Fig.32 The above-mentioned subroutine is a subroutine of an embodiment of the process of step S330 of superimposed display of the photographic information INFO.
[0071] Fig.36 Yes means to proceed Fig.32 The illustrated embodiment is a subroutine of the process of step S340 of overlapping display of the playback images.
[0072] Fig.37 This is a flowchart showing an embodiment of a method for changing the appearance of an imaging device. DETAILED DESCRIPTION
[0073] Hereinafter, preferred embodiments of the virtual image display device, the imaging device, the virtual image display system and the method according to the present invention will be described with reference to the accompanying drawings.
[0074] [Overview of Virtual Image Display System]
[0075] Figure 1 It is a diagram showing the system configuration of the virtual image display system involved in the present invention.
[0076] Figure 1 The virtual image display system 10 shown is composed of a virtual image display device 100 and a first camera device (hereinafter referred to as "camera device") 200, and the virtual image display device 100 and the camera device 200 communicate with each other. The communication is preferably a standard short-range wireless communication such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark), but it may also be a wired communication.
[0077] The virtual image display device 100 of this embodiment is a glasses-type wearable device having a virtual image display function, and includes a processor, a first display, and a second camera device (hereinafter referred to as a “built-in camera”) not shown.
[0078] The first display is a display for realizing augmented reality (AR: Augmented Reality), virtual reality (VR: Virtual Reality), mixed reality (MR: Mixed Reality), or cross reality (XR: X Reality) which is a general term for AR, VR, and MR.
[0079] AR is a technology that "virtually expands" the real world in front of you by overlapping the virtual world (virtual images, etc.) with the real world seen through glasses. VR is a technology that displays images taken by a camera on VR goggles or VR headsets, and blocks the real world through VR goggles, etc., and has the feature of being able to penetrate into the unreal world. MR is a technology that overlaps virtual objects or information on the real world, which is the same as AR, but it is a technology that further develops AR, and can display digital information according to the user's position or action, or the user can directly touch the digital information to operate.
[0080] Figure 1 The first display (hereinafter referred to as “XR display”) of the virtual image display device 100 shown has an AR or MR display function of superimposing a virtual world on the real world viewed through glasses (glasses).
[0081] exist Figure 1 In the figure, 146 is an optical system including a half-mirror of a virtual image display device 100 (hereinafter also referred to as "smart glasses"), and the image (virtual image) displayed on the display element of the smart glasses is superimposed on the real world via the optical system 146. In addition, the details of the smart glasses will be described later.
[0082] Figure 2Observed from the front Figure 1 A perspective view of the camera device shown.
[0083] Figure 1 and Figure 2 The imaging device 200 shown is a mirrorless digital single-lens camera composed of an interchangeable lens 202 and a camera body (device body) 203 to which the interchangeable lens 202 can be detachably mounted. The imaging device 200 is not limited to a mirrorless digital single-lens camera, but may be a digital single-lens reflex camera, a lens-integrated digital camera in which the lens cannot be interchanged, or a smartphone or PDA (Personal Digital Assistants) having a camera function.
[0084] exist Figure 1 In the embodiment, a shutter button 232, a shutter speed / sensitivity dial 234, an exposure correction dial 236, etc. are provided on the upper surface of the camera body 203.
[0085] Furthermore, on the back surface which is one wall surface of the camera body 203 , a monitor panel 240 is rotatably provided via a rotation mechanism (not shown).
[0086] Figure 3 The diagram shows three open and closed states of the monitor panel of the imaging device.
[0087] like Figure 3 As shown, the monitor panel 240 is a tilt-type panel that can change its relative posture relationship with the camera body 203 by rotating.
[0088] Figure 3 (A) shows the imaging device 200 in a state where the monitor panel 240 is closed, Figure 3 (B) and (C) show the imaging device 200 in the states where the monitor panel 240 is rotated 90 degrees and 180 degrees.
[0089] A main monitor (second display) 241 is arranged on the inner surface (back surface) of the monitor panel 240 which is hidden when the monitor panel 240 is closed, and a sub-monitor 242 is arranged on the surface (front surface) which is substantially at the same level as the camera appearance when the monitor panel 240 is closed.
[0090] The main monitor 241 is Figure 3 (B) and (C) The monitor panel 240 can be operated when it is opened to display a real-time preview image, a recorded image (playback image), etc. The sub-monitor 242 can be operated when it is opened. Figure 1 and Figure 3 The monitor panel 240 shown in (A) can be operated when closed, and displays shutter speed, aperture value, ISO sensitivity, and other information related to photography (photography information).
[0091] And, if Figure 1 As shown, an eyepiece portion of a viewfinder 243 is arranged on the back of the camera body 203. The viewfinder 243 is a hybrid viewfinder capable of switching between an electronic viewfinder (EVF: Electronic View Finder) and an optical viewfinder (OVF: Optical View Finder).
[0092] Furthermore, in the OVF display, the Electronic Range Finder (ERF) (a small window that displays the EVF) can be displayed on a portion of the OVF, or can be hidden. The ERF can display three types of field of view: 100%, 2.5x magnification, and 6x magnification. When shooting in the OVF, focus confirmation, field of view, exposure, and white balance can be monitored.
[0093] like Figure 2 As shown in FIG. 1 , a viewfinder switching lever 249 for switching between the EVF and the OVF of the viewfinder 243 is provided on the front surface of the camera.
[0094] exist Figure 2 If the viewfinder switching lever 249 is turned clockwise, the EVF and OVF can be switched. Figure 2 When the viewfinder switching lever 249 is rotated counterclockwise, the display and non-display of the ERF can be switched.
[0095] Furthermore, a trigger switch 244 for indicating a plurality of directions such as up, down, left, and right, a menu / execution button 245 , a playback button 246 , a display / return button 247 , and the like are provided on the back side of the camera body 203 .
[0096] Furthermore, a plurality of function buttons to which various functions can be assigned are provided.
[0097] [Overview of Virtual Image Display System]
[0098] Figure 4 This is a diagram showing a situation where an image including the field of view of the imaging device is captured by the second imaging device of the virtual image display device.
[0099] exist Figure 4 In the embodiment, the built-in camera 101 of the virtual image display device 100 captures a visual field image corresponding to the visual field of the user including the camera 200. The built-in camera 101 captures a visual field image corresponding to the visual field of the user when the user wears the smart glasses and the pupil faces forward.
[0100] The user holds the camera 200 and performs various operations such as framing.
[0101] When in use in the XR environment, the camera 200 transmits a live preview image generated by the operation of the camera 200 to the virtual image display device 100 as continuous data.
[0102] The processor of the virtual image display device 100 determines the display area of the camera 200 based on the image (camera image) of the camera 200 present in the field of view image. The display area is not limited to Figure 3 The display area that can actually display an image, such as the main monitor 241 or the sub-monitor 242 of the imaging device 200 shown, may be, for example, an area set in advance on the back surface, which is one wall surface of the camera body 203 .
[0103] The user can visually recognize the camera device 200 in the real world through the smart glasses, but the processor transforms (geometric transformation) the instant preview image received from the camera device 200 and displays it on the smart glasses, thereby overlapping and displaying the instant preview image (virtual image) on a specific display area of the actual camera device 200 via the optical system 146 of the smart glasses.
[0104] Figure 5 This is a diagram showing a state where a live preview image is superimposed on a specific display area of an imaging device.
[0105] At a display position in front of the eyes of the user wearing the smart glasses, a live preview image LV1 geometrically transformed by the optical system 146 of the smart glasses is displayed. Figure 5 As shown, the user can visually recognize the actual camera device 200 through the smart glasses, and can observe the real-time preview image LV1 in a manner such that an image (real-time preview image LV1) on an XR environment that does not actually exist and is synthesized (attached) with a specific display area of the camera device 200 is displayed on the camera device 200.
[0106] Figure 1 and Figure 3 The camera device 200 shown does not have a display Figure 5 The real monitor of the real-time preview image LV1 of the size shown in the figure is not shown. However, by displaying the real-time preview image LV1, the virtual image display device 100 enables the user to visually recognize that the camera device 200 has a real-time preview image LV1. Figure 5 A display showing a live preview image LV1 is shown.
[0107] According to the virtual image display system 10, the user does not look at the specific display area (real world) on the back of the camera device 200, but observes the real-time preview image LV1 synthesized with the specific display area through the smart glasses. Therefore, even if the specific display area in reality is difficult to see due to direct sunlight, the real-time preview image LV1 can be well observed.
[0108] Furthermore, since the camera device 200 is a real-world camera product, it is easy to operate when performing photography operations for works, and the photography experience can be maintained at a high quality. Moreover, even when the position or angle of the camera device 200 is moved for framing, etc., the live preview image LV1 can be displayed on the smart glasses following the camera device 200, and the shutter opportunity can be avoided in photography with intense movement (sports photography or panning, photography of children, pets, etc.) that requires fast operation of the camera device 200.
[0109] In addition, the field of view image photographed by the built-in camera 101 of the smart glasses is used to obtain a specific display area (coordinates) of the camera image in the field of view image, and to obtain information (transformation information) used when making the live preview image overlap with the specific display area of the camera device 200, and when an immersive display (VR goggles) covering the field of view (field of view) is used instead of the smart glasses, it is used as a field of view image displayed on the VR goggles.
[0110] <Implementation Method of Virtual Image Display Device>
[0111] Figure 6 It is a block diagram showing an embodiment of a virtual image display device constituting the virtual image display system according to the present invention.
[0112] Figure 6 The virtual image display device 100 shown is a Figure 1 A block diagram of the internal structure of the virtual image display device 100 having the appearance shape shown.
[0113] The virtual image display device 100 has a built-in camera 101, a processor (XR processor) 110, a first memory 120, an operating unit 130, a first display with a virtual image display function, namely an XR display 140, a display control unit 142, a virtual object generation unit 150, a first communication unit 160, a GPS (Global Positioning System) receiver 170, a gyro sensor 172, an acceleration sensor 174 and a geomagnetic sensor 176, etc.
[0114] The built-in camera 101 includes a photographing lens 102 , an image sensor 104 , and an AFE (Analog Front End) 106 .
[0115] When photographing with the camera 200 handheld, the built-in camera 101 photographs a field of view image including the camera 200 (camera image), and the photographic lens 102 preferably has a field of view angle equal to or greater than that of a human being.
[0116] The image sensor 104 can be formed of a CMOS (Complementary Metal-Oxide Semiconductor) type color image sensor. The image sensor 104 is not limited to a CMOS type, but can be a CCD (Charge Coupled Device) type image sensor.
[0117] The optical image of the subject formed on the light receiving surface of the image sensor 104 through the photographic lens 102 is converted into an electrical signal by the image sensor 104. In each pixel of the image sensor 104, a charge corresponding to the amount of incident light is accumulated, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 104 as an image signal.
[0118] AFE106 performs various analog signal processing on the analog image signal output from the image sensor 104. AFE106 includes a correlated double sampling circuit, an AGC (Automatic Gain Control) circuit, and an analog-to-digital conversion circuit (A / D conversion circuit) (all omitted in the figure). The correlated double sampling circuit performs correlated double sampling processing on the analog signal from the image sensor 104 to remove noise caused by the reset of the signal charge. The AGC circuit amplifies the analog signal from which the noise has been removed by the correlated double sampling circuit so that the signal level of the analog signal falls within an appropriate range. The A / D conversion circuit converts the image signal whose gain has been adjusted in the AGC circuit into a digital signal.
[0119] In addition, when the image sensor 104 is a CMOS type image sensor, the AFE 106 is usually built in the image sensor 104 .
[0120] The XR processor 110 is composed of a CPU (Central Processing Unit), etc., and while centrally controlling each unit, performs various processes such as obtaining information (transformation information) used when the live preview image received from the camera 200 via the first communication unit 160 is superimposed on a specific display area of the camera 200, and transforming (geometric transformation) the live preview image based on the information and outputting it to the XR display 140. In addition, the details of the processing based on the XR processor 110 will be described later.
[0121] The first memory 120 includes a flash memory, a ROM (Read-only Memory), a RAM (Random Access Memory), etc. The flash memory and the ROM are non-volatile memories that store various programs including firmware, parameters, and images representing various virtual objects including avatars (virtual object images).
[0122] The RAM functions as a work area for processing by the XR processor, and temporarily stores firmware stored in the nonvolatile memory, etc. In addition, the XR processor 110 may incorporate a part of the first memory 120 (RAM).
[0123] The operation unit 130 is a part that performs various operation instructions, parameter settings, selection of virtual objects including avatars, control of avatars, etc. It has a microphone. In addition to a voice input device for operation input by voice, it is also possible to consider a gesture input unit that reads body movements such as hands or fingers from images taken by the built-in camera 101 and sets them as operation inputs, a touch panel provided on the frame of the smart glasses, and a dedicated controller for moving the GUI (Graphical UserIhinterface: graphical user interface) screen displayed on the XR display 140 and the cursor on the GUI screen.
[0124] Furthermore, the shaking, vibration, or light angle of the camera device 200 can be detected by the gyro sensor 272, the acceleration sensor 274, the geomagnetic sensor 276, and the like of the camera device 200, and the detected information can be used as various operation instructions and the like.
[0125] In addition, since both hands are used in the photographing operation of the camera device 200, a voice input device is preferably used.
[0126] When the display control unit 142 inputs a live preview image, a playback image, photographic information, and the like from the XR processor 110 , it converts the image into a display signal format and outputs the signal to the XR display 140 .
[0127] The XR display 140 has a display element (not shown) and displays a live preview image and the like on the display element driven by a display signal from the display control unit 142. The live preview image displayed on the display element is transmitted to the optical system 146 (see FIG. 14A ) of the smart glasses. Figure 1 ) overlaps with the real world beyond the glasses.
[0128] The XR display 140 displays a live preview image or the like that is geometrically transformed so as to overlap with a specific display area of the imaging device 200 at a display position in front of the user's eyes, and the user can observe the live preview image or the like superimposed on the real world.
[0129] The virtual object generation unit 150 reads a desired virtual object image indicated by the user from various virtual object images including the avatar stored in the first memory 120, and generates a virtual object image processed for synthesis into the live preview image. For example, if an avatar representing the user himself is selected, and a hairstyle or clothing is selected, an avatar wearing the selected hairstyle or clothing is generated, and further, when the position, direction, etc. of the avatar are indicated by the operation unit 130, processing such as moving the avatar to the position is performed.
[0130] Furthermore, the virtual object generation unit 150 receives a selection instruction of a virtual object representing a component (external component, switch, etc.) of the imaging device 200, and generates a virtual object image processed by processing the image of the selected component (component image) for synthesis into the imaging device 200. By synthesizing the component image into the imaging device 200, the appearance of the imaging device 200 observed through the XR display 140 can be changed.
[0131] The first communication unit 160 is a part that sends and receives necessary information with the camera 200. When the camera 200 shoots a live preview image in the XR environment, the first communication unit 160 receives a live preview image from the camera 200. Figure 1 ) and performs recording photography, the first communication unit 160 receives a shutter release signal. When the avatar image is synthesized on the live preview image upon receiving the shutter release signal, the first communication unit 160 transmits the avatar image to the camera 200.
[0132] The GPS receiver 170, the gyro sensor 172, the acceleration sensor 174, and the geomagnetic sensor 176 are sensors for detecting information indicating the three-dimensional position and posture of the XR display 140 (built-in camera 101). The information indicating the three-dimensional position and posture of the XR display 140 is used to obtain information indicating the three-dimensional position and posture of the camera device 200 based on the XR display 140. The information indicating the three-dimensional position and posture of the camera device 200 based on the XR display 140 can be used when calculating transformation information for geometrically transforming the live preview image.
[0133] <Implementation Method of Imaging Device>
[0134] Figure 7 It is a block diagram showing an embodiment of the imaging device according to the present invention.
[0135] Figure 7 The camera device 200 shown in the figure is a Figure 1 FIG. 1 is a block diagram of the internal structure of the imaging device 200 of the virtual image display system 10 shown in FIG.
[0136] The camera device 200 includes a camera unit 201, a camera processor 210, a second memory 220, an operation unit 230, a display control unit 248, a main monitor 241, a sub-monitor 242, a viewfinder 243, an image processing unit 250, a second communication unit (communication unit) 260, a GPS receiver 270, a gyro sensor 272, an acceleration sensor 274, a geomagnetic sensor 276, etc.
[0137] The imaging unit 201 includes an interchangeable lens 202 , an image sensor 204 , and an APE 206 .
[0138] The interchangeable lens 202 is mounted on the camera body 203 and is capable of bidirectional communication with the camera body 203 via a plurality of signal terminals provided on the lens mount of the camera body 203. For example, based on a request signal from the camera body 203, lens information (focal length, focus position, aperture value, etc.) of the interchangeable lens 202 is sent to the camera body 203, and the focusing lens, aperture, etc. are driven according to instructions from the camera body 203.
[0139] The image sensor 204 is composed of a CMOS type color image sensor, but is not limited to the CMOS type, and may be a CCD type image sensor.
[0140] The AFE 206 performs various analog signal processing on the analog image signal output from the image sensor 204 , converts the analog image signal into a digital image signal, and outputs the digital image signal as RAW data.
[0141] The camera processor 210 is composed of a CPU and the like, and centrally controls each unit and centrally controls each unit according to a user operation using the operation unit 230 , and performs various processes including AF (Auto Focus) control and AE (Automatic Exposure) control.
[0142] Note that the AF control and the AE control are automatically performed when the automatic mode is set by the operation unit 230 , and of course, the AF control and the AE control are not performed when the manual mode is set.
[0143] like Figure 1 As shown, the operating unit 230 includes, in addition to the shutter button 232, the shutter speed / sensitivity dial 234, and the exposure correction dial 236, a trigger switch 244, a menu / execution button 245, a playback button 246, a display / return button 247, etc., and various instructions and settings can be performed by operating these buttons, dials, etc.
[0144] The operating unit 230 of the camera device 200 of this example is different from the operating unit of the existing camera device in that it has a mode switching unit for switching between a first mode (non-XR mode) for using the camera device 200 in a non-XR environment (normal environment) and a second mode (XR mode) for using the camera device 200 in an XR environment.
[0145] The mode switching unit for the XR mode / non-XR mode can use a function button assigned with setting / releasing the XR mode, but a dedicated mode switching lever may also be provided. Furthermore, the trigger switch 244 and the menu / execution button 245 may be used to set / releasing the XR mode on the menu screen.
[0146] As a dedicated mode switching lever, for example, Figure 2 A mode switching lever having the same structure as the viewfinder switching lever 249 shown is provided on the front surface, the back surface or the upper surface of the camera body 203 .
[0147] Note that the details of the operation of the imaging device 200 when the XR mode is set and used in the XR environment will be described later.
[0148] The second memory 220 includes a flash memory, a ROM, a RAM, etc. The flash memory and the ROM are nonvolatile memories that store various programs including firmware, parameters, and photographed images (still images, moving images), etc.
[0149] The RAM functions as a work area for processing by the camera processor 210 and temporarily stores firmware stored in the nonvolatile memory, etc. The camera processor 210 may also incorporate a part of the second memory 220 (RAM).
[0150] When the display control unit 248 inputs live preview images, playback images, photographic information, etc. from the camera processor 210 or the image processing unit 250, it converts them into a display signal format and outputs them to the main monitor 241, the sub-monitor 242 and the viewfinder 243 (EVF).
[0151] The main monitor 241 is Figure 3 The monitor panel 240 shown in (B) and (C) can be operated when opened, and in the non-XR mode, displays a live preview image, a recorded image (playback image), etc. according to instructions from the camera processor 210.
[0152] The secondary monitor 242 is Figure 3 The monitor panel 240 shown in (A) can be operated when closed, and in the non-XR mode, it displays shooting information such as shutter speed, aperture value, ISO sensitivity, etc. according to the instruction from the camera processor 210.
[0153] The viewfinder 243 can switch between EVF and OVF by operating the viewfinder switching lever, and when switched to EVF and in non-XR mode, displays a live preview image according to an instruction from the camera processor 210 .
[0154] Therefore, when the monitor panel 240 is closed and the mode is not XR, the user can frame the image while checking the imaging range through the viewfinder 243 .
[0155] In addition, in non-XR mode, even if the monitor panel 240 is open, if the eye is close to the viewfinder 243, the display can be automatically switched to the viewfinder 243 through the action of the eye sensor (not shown), and if the eye is away, the display can be switched to the main monitor 241.
[0156] On the other hand, the main monitor 241 and the sub-monitor 242 do not display the live preview image and photographic information respectively in the XR mode. For example, they can display a mark (for example, an AR mark) that determines the display area of the main monitor 241 or the sub-monitor 242 according to instructions from the camera processor 210.
[0157] The image processing unit 250 performs RAW development processing such as offset processing, gain control processing including white balance correction and sensitivity correction, gamma correction processing, and demosaicing processing on the RAW data output from the camera unit 201. In addition, the image processing unit 250 performs compression processing on the RAW data acquired by pressing the shutter button 232 after the RAW development processing, and records it as an image file in the second memory 220 or a memory card (not shown).
[0158] Then, the image processing unit 250 performs RAW development processing on the continuous RAW data output from the imaging unit 201 , and outputs the continuous data as a live preview image LV to the main monitor 241 or the second communication unit 260 via the display control unit 248 .
[0159] The second communication unit 260 is a part that transmits and receives necessary information to and from the virtual image display device 100 . When the imaging device 200 is set to the XR mode and a live preview image is captured in the XR environment, the second communication unit 260 transmits the live preview image LV to the virtual image display device 100 .
[0160] Furthermore, when the shutter button 232 is pressed on the camera device 200 in the XR environment (see Figure 1 ) and performs photography for recording, the second communication unit 260 sends a shutter release signal to the virtual image display device 100. The virtual image display device 100 receives the shutter release signal via the first communication unit 160, and when the avatar image is synthesized with the live preview image upon receiving the shutter release signal, the avatar image is sent from the first communication unit 160 of the virtual image display device 100 to the camera 200.
[0161] The GPS receiver 270, the gyro sensor 272, the acceleration sensor 274, and the geomagnetic sensor 276 are sensors for detecting information indicating the three-dimensional position and posture of the camera 200. The information indicating the three-dimensional position and posture of the camera 200 is used to obtain information indicating the three-dimensional position and posture of the camera 200 based on the XR display 140. The information indicating the three-dimensional position and posture of the camera 200 based on the XR display 140 can be used to calculate transformation information for geometrically transforming the live preview image LV.
[0162] [First Embodiment of Virtual Image Display System]
[0163] Figure 8 is a block diagram showing a main part of a first embodiment of a virtual image display system according to the present invention, Figure 6 1 is a functional block diagram of the functions of the XR processor 110 of the virtual image display device 100 shown.
[0164] exist Figure 8 In the embodiment, the XR processor 110 functions as a transformation information acquisition unit 112 , a synthesis unit 114 , and a geometric transformation unit 116 .
[0165] The built-in camera 101 of the smart glasses captures a visual field image IM corresponding to the visual field of the user including the imaging device 200 , and outputs the captured visual field image IM to the conversion information acquisition unit 112 .
[0166] When the imaging device 200 is used in the XR environment, the imaging device 200 transmits the live preview image LV generated by the operation of the imaging device 200 as continuous data, and the first communication unit 160 receives the live preview image LV from the imaging device 200 .
[0167] The conversion information acquisition unit 112 is a part that acquires conversion information used when superimposing the live preview image LV on the display area of the imaging device 200 , and receives a captured image (field of view image IM) from the built-in camera 101 .
[0168] The conversion information acquisition unit 112 analyzes the inputted visual field image IM to acquire region information indicating the display region of the imaging device 200 , and acquires conversion information based on the region information.
[0169] Fig. 9 A diagram showing the geometric relationship between a live preview image, a field of view image, and a display area of an imaging device.
[0170] exist Fig. 9 In the above, the image coordinate system of the live preview image LV is set to the XY coordinate system, and the image coordinate system of the visual field image IM is set to the X'Y' coordinate system. Furthermore, the vertical and horizontal image sizes of the live preview image LV are set to h1 and w1, and the vertical and horizontal image sizes of the visual field image IM are set to h2 and w2. In addition, the coordinates of the four corner points (feature points) P1 to P4 in the XY coordinate system of the live preview image LV are known from the image size of the live preview image LV.
[0171] like Fig. 9 As shown, the field of view image IM includes a camera image representing the imaging device 200 .
[0172] The conversion information acquisition unit 112 acquires, for example, coordinates of four corner points of the display area (corresponding points P1 ′ to P4 ′ corresponding to the characteristic points P1 to P4 ) as area information indicating a specific display area of the imaging device 200 in the field of view image IM.
[0173] The conversion information acquisition unit 112 of the first embodiment can analyze the visual field image IM to detect the monitor panel 240 (see Figure 1 ) as corresponding points P1' to P4' corresponding to the feature points P1 to P4 at the four corners of the live preview image LV, and the points at the four corners of the outline are used as corresponding points P1' to P4'. In addition, in this example, the area surrounded by the quadrilateral outline of the closed monitor panel 240 is used as the display area, but the display area can be defined appropriately.
[0174] Next, the conversion information acquisition unit 112 acquires conversion information used when superimposing the live preview image LV on the display area (the area of the monitor panel 240 specified by the corresponding points P1 ′ to P4 ′) of the camera image of the field of view image IM.
[0175] Now, when the live preview image LV is geometrically deformed by affine transformation and superimposed on the display area on the camera image, it is necessary to obtain the parameters (transformation information) of the affine transformation.
[0176] Affine transformation can be performed by the following formula.
[0177] [Mathematical formula 1]
[0178]
[0179] In [Equation 1], (x, y) represents the coordinates of the live preview image LV (coordinates in the XY coordinate system), and (x', y') represents the coordinates after affine transformation (coordinates in the X'Y' coordinate system of the field of view image IM).
[0180] In [Equation 1], parameters a to d are parameters representing image enlargement, reduction, rotation, and skew (shearing), and parameter t x ,t y is the parameter representing parallel movement.
[0181] In the affine transformation, in order to estimate the parameters, it is necessary to provide three or more sets of matching points (feature points and corresponding points) of two images (in this case, the live preview image LV and the visual field image IM). However, since four sets of matching points, namely, the feature points P1 to P4 of the live preview image LV and the corresponding points P1' to P4' of the visual field image IM, are available, the parameters a to d, t of the affine transformation shown in [Formula 1] can be obtained. x ,t y .
[0182] The transformation information acquisition unit 112 can thus acquire the parameters a to d, t of the affine transformation. x ,t y as transformation information.
[0183] Also, as another method of geometrically transforming between two images, for example, a projective transformation may be considered.
[0184] Projection transformation can be performed by the following formula.
[0185] [Mathematical formula 2]
[0186]
[0187] In the projection transformation, it is necessary to find nine parameters (h11 ~h 33 ) requires finding more matching points between two images compared to affine transformation, but can perform geometric transformation with higher accuracy than affine transformation.
[0188] Return to Figure 8 The real-time preview image LV is added to the synthesis unit 114 via the first communication unit 160. The synthesis unit 114 is a part that synthesizes the virtual object on the real-time preview image LV. The image of the virtual object generated by the virtual object generation unit 150 is added to other inputs of the synthesis unit 114.
[0189] The virtual object generation unit 150 reads an image of a desired virtual object (eg, avatar) from the first memory 120 in response to a user instruction, and generates and outputs an avatar image processed to be synthesized with the live preview image LV.
[0190] The user can register an avatar image representing the user in the first memory 120. The avatar image representing the user may be a photograph of the user himself or herself, or may be an image of an animation character, a robot, a pet, or the like.
[0191] Furthermore, the user can appropriately select a hairstyle or clothing by operating the operation unit 130, and the virtual object generation unit 150 can generate an avatar image wearing the hairstyle or clothing selected by the user. Furthermore, if the user indicates the position, direction, etc. of the avatar, the virtual object generation unit 150 performs processing such as moving the avatar image to the indicated position, like an avatar in a role-playing game.
[0192] Furthermore, when generating a more realistic avatar image, the virtual object generation unit 150 may adjust the brightness or hue of the avatar image according to the brightness or hue obtained from the live preview image LV or the like.
[0193] The synthesis unit 114 synthesizes the image of the virtual object including the avatar generated by the virtual object generation unit 150 with the live preview image LV supplied via the first communication unit 160 , and outputs the synthesized live preview image LV to the geometric transformation unit 116 .
[0194] In addition, when the virtual object generation unit 150 does not generate a virtual object, the synthesis unit 114 outputs the input live preview image LV as it is to the geometric transformation unit 116 .
[0195] The geometric transformation unit 116 is a part that geometrically transforms the live preview image LV added by the synthesis unit 114 according to the transformation information obtained by the transformation information acquisition unit 112. For example, when the geometric transformation by the geometric transformation unit 116 is an affine transformation, the transformation information acquisition unit 112 obtains the parameters (a to d, t x , t y ) as transformation information, the geometric transformation unit 116 performs affine transformation on the pixel of the coordinate (x, y) of the live preview image LV according to the transformation information, thereby transforming it into the pixel of the coordinate (x', y') of the visual field image IM. In addition, when the geometric transformation by the geometric transformation unit 116 is a projection transformation, the projection transformation parameter (h ) shown in [Formula 2] is obtained. 11 ~h 33 ) as transformation information, the geometric transformation unit 116 performs projective transformation on the pixel at the coordinate (x, y) of the live preview image LV according to the transformation information, thereby transforming it into the pixel at the coordinate (x', y') of the field of view image IM.
[0196] The live preview image LV geometrically transformed by the geometric transformation unit 116 (hereinafter referred to as “live preview image LV1 ”) is output to the XR display 140 .
[0197] The XR display 140 displays the geometrically transformed live preview image LV1 at the corresponding pixel position of the XR display 140 (the pixel position corresponding to the field of view image IM of the built-in camera 101). Fig. 9 In the example shown, the live preview image LV represented by four coordinates P1 to P4 in its XY coordinate system is coordinate-converted to the X'Y' coordinate system of the visual field image IM and displayed as a live preview image LV1 represented by four coordinates P1' to P4'.
[0198] In addition, since there is parallax between the user's field of view when wearing smart glasses and the field of view image IM photographed by the built-in camera 101, the XR display 140 preferably corrects the parallax based on the size (equivalent to the distance) and position of the camera image in the field of view image IM.
[0199] <Second Embodiment of Conversion Information Acquisition Unit>
[0200] Figure 8 The conversion information acquisition unit 112 of the first embodiment shown in the figure analyzes the visual field image IM to detect the quadrilateral outline of the monitor panel 240, and converts the points ( Fig. 9The quadrilateral area surrounded by P1'~P4') is set as the display area A of the camera device 200 and the transformation information is obtained, but it is not limited to this. The transformation information acquisition unit 112 can also obtain the transformation information based on the position of the mark (hereinafter referred to as "AR mark") of the camera device 200 photographed by the built-in camera 101.
[0201] Fig.10 This is a diagram showing the appearance of an imaging device provided with an AR marker.
[0202] exist Fig.10 The imaging device 200 shown in the figure is provided with nine AR markers M1 to M9.
[0203] The AR marker is a marker representing a pre-registered illustration or image, etc., and can be associated with position information. Therefore, the transformation information acquisition unit 112 can obtain the position information of the position displayed by the AR markers M1 to M9 by reading the AR markers M1 to M9 from the field of view image IM, and can also obtain information such as the inclination, position, size, and center coordinates of each surface of the camera device 200.
[0204] Among the AR markers M1 to M9 , the AR marker M1 is an AR marker displayed on the sub-display 242 , and the AR markers M2 to M9 are AR markers provided in advance on the camera body 203 .
[0205] The AR markers M2 to M4 are AR markers indicating the positions of three of the four corners of the monitor panel 240 , and when the area of the monitor panel 240 is used as the display area A of the live preview image LV1 , they become AR markers M that specify the display area A of the live preview image LV1 of the camera device 200 .
[0206] In addition, when the camera device 200 is used in an XR environment, the AR marker M1 can be displayed as a marker for specifying the display area A of the live preview image LV1. Furthermore, when the AR marker M is displayed on the sub-monitor 242, the number of the AR markers M is not limited to one, but a plurality of AR markers M can be displayed. Figure 3 When (B) and (C) are set as the display area of the live preview image LV1, a plurality of AR markers M that specify the area of the primary monitor 241 can be displayed.
[0207] The AR marker M5 is a marker provided on the upper surface of the camera body 203, the AR marker M6 is a marker provided on the upper back surface of the camera body 203, and the AR marker M7 is a marker provided on the side surface of the camera body 203. These AR markers M5 to M7 are provided to identify the respective surfaces of the camera body 203.
[0208] The AR mark M8 is a mark provided on the upper surface of the shutter speed / sensitivity dial 234, and the AR mark M9 is a mark provided on the upper surface of the exposure correction dial 236. These AR marks M8 and M9 are provided to identify the position (including the rotational position) of the dial.
[0209] The AR markers M4 to M9 can be used to identify an area used for attaching a virtual object (skin with a changed appearance) to the camera body 203 .
[0210] In addition, the AR markers M1 to M9 may be replaced by a two-dimensional barcode (e.g., "QR Code" (registered trademark)). When the two-dimensional barcode is displayed, the information of what the camera device 200 displays can be set as a two-dimensional barcode that encodes the data or indicators (numerical information such as shutter speed, aperture value, histogram, etc.) displayed in the live preview image, and the two-dimensional barcode may include information that combines the configuration information of the data display.
[0211] Furthermore, the AR markers M1 to M9 may be replaced by other marks having shapes suitable for determining positions, or the AR markers M and two-dimensional barcodes may be mixed. The types, positions, and numbers of the marks are not limited to the following. Fig.10 The embodiment shown.
[0212] <Third Embodiment of Conversion Information Acquisition Unit>
[0213] The transformation information acquisition unit that obtains the transformation information used when making the live preview image LV overlap with the display area A of the camera device 200 is not limited to the case of using the field of view image IM photographed by the built-in camera 101, but is capable of acquiring the transformation information by acquiring information representing the three-dimensional position and posture of the camera device 200 and information representing the three-dimensional position and posture of the XR display 140, and acquiring information representing the relative three-dimensional position and posture of the camera device 200 with respect to the XR display 140 (first display).
[0214] The transformation information acquisition unit can obtain information representing the three-dimensional position and posture of the XR display 140 based on the sensor outputs of the GPS receiver 170, gyro sensor 172, acceleration sensor 174 and geomagnetic sensor 176 of the virtual image display device 100. Similarly, it can obtain information representing the three-dimensional position and posture of the XR display 140 based on the sensor outputs of the GPS receiver 270, gyro sensor 272, acceleration sensor 274 and geomagnetic sensor 276 of the camera device 200.
[0215] Here, if the virtual image display device 100 (XR display 140) is represented by a global coordinate system and the camera device 200 is represented by a camera coordinate system (local coordinate system), since the size and shape of the camera device 200 are known, the three-dimensional position of each part of the camera device 200 in the local coordinate system can be set in advance.
[0216] On the other hand, the global coordinate system of the XR display 140 can be defined as, for example, the following XYZ three-axis orthogonal coordinate system: the center of the display element of the XR display 140 is the origin, the directions in the planes of the display elements orthogonal to each other passing through the origin of the display element are the X-axis and the Y-axis, and the normal direction passing through the origin of the display element is the Z-axis.
[0217] As described above, by acquiring information representing the three-dimensional position and posture of the XR display 140 and information representing the three-dimensional position and posture of the camera device 200 (i.e., acquiring information representing the three-dimensional position and posture of the camera device 200 based on the XR display 140), it is possible to acquire transformation information used when making the live preview image LV overlap with the display area A of the camera device 200 based on this information.
[0218] For example, based on the information indicating the three-dimensional position of the area A of the camera device 200 (i.e., the three-dimensional position of the live preview image LV) and the posture representation based on the XR display 140, the three-dimensional position of the local coordinate system of the display area A of the camera device 200 is transformed into the coordinates of the coordinate system of the XR display 140, i.e., the global coordinate system. Next, based on known parameters such as the size of the display element of the XR display 140 and the focal length of the optical system, the three-dimensional position of the display area A of the camera device 200 (i.e., the three-dimensional position of the live preview image LV) whose coordinates are transformed into the global coordinate system is transformed into a perspective projection on the display surface of the display element of the XR display 140, thereby obtaining the transformed live preview image LV1.
[0219] In addition, the information indicating the relative three-dimensional position and orientation of the imaging device 200 with respect to the XR display 140 is not limited to being obtained based on the sensor outputs of the GPS receiver 170, gyro sensor 172, acceleration sensor 174, and geomagnetic sensor 176 of the virtual image display device 100, and the sensor outputs of the GPS receiver 270, gyro sensor 272, acceleration sensor 274, and geomagnetic sensor 276 of the imaging device 200. It can also be obtained based on the measurement data measured by an optical sensing measurement device (e.g., LiDAR (Light Detection And Ranging)) provided in the virtual image display device 100 (smart glasses), the distance image acquired by a TOF (Time Of Flight) camera, or the stereo image acquired by the built-in cameras provided on the left and right of the smart glasses, etc.
[0220] <First Embodiment of XR Display>
[0221] Fig.11 It is a diagram showing the first embodiment of the XR display of the virtual image display device.
[0222] Fig.11 The XR display 140 shown is a transmissive field-of-view display and is a glasses-type wearable device (smart glasses) that displays an instant preview image LV1 with geometric transformation to the real-world field of view.
[0223] The XR display 140 includes an optical system 146 including a display element 144, a lens 146A, and a half mirror 146B. The geometrically transformed instant preview image LV1 displayed on the display element 144 is displayed as a virtual image at the display position in front of the user's eyes by the optical system 146.
[0224] The user can visually recognize the real world transmitted through the half mirror 146B and can also visually recognize the instant preview image LV1 reflected on the half mirror 146B and superimposed on the real world. In addition, the instant preview image LV1 is an image of the display area A of the imaging device 200 superimposed on the real world and is a virtual image since it is not the reflected light from the imaging device 200.
[0225] And, Fig.11The display element 144 shown is a surface-emitting element such as an organic EL (electro-luminescence) panel or a liquid crystal panel, but is not limited thereto, and may also be a display element composed of a DMD (Digital Micromirror Device) in which a plurality of movable micromirrors (micromirrors) are arranged in a planar shape on an integrated circuit and a light source that makes red, green, and blue light incident on the DMD.
[0226] <First display example in XR mode>
[0227] Fig.12 It is a diagram showing a first display example such as an instant preview image when the imaging device is set to the XR mode.
[0228] Fig.12 (A) is a diagram showing an image when the imaging device 200 is observed with the naked eye. In Fig.12 (A), the monitor panel 240 is turned off, and photographic information is displayed on the sub-monitor 242. When the XR mode is set, the user cannot observe the instant preview image LV1 without wearing the virtual image display device 100 (smart glasses).
[0229] Fig.12 (B) is a diagram showing an image when the imaging device 200 is observed while wearing the virtual image display device 100. Fig.12 (B) shows a first display example in which the instant preview image LV1 is overlapped and displayed on the area of the closed monitor panel 240 of the imaging device 200.
[0230] In this case, the user wears the virtual image display device 100 and observes the imaging device 200, seemingly as if the instant preview image LV1 is displayed on the area of the monitor panel 240.
[0231] In addition, it is also possible to switch from the instant preview image LV1 to an image synthesized with the real world by setting, thereby changing to another display mode.
[0232] <Second display example in XR mode>
[0233] Fig.13 It is a diagram showing a second display example such as an instant preview image when the imaging device is set to the XR mode.
[0234] Fig.13 The second display example shown is shown for the case where the instant preview image LV1 is overlapped and displayed on the area of the main monitor 241 of the imaging device 200. And, Fig.13The imaging device 200 shows a state where the monitor panel 240 is rotated by 90°. In this case, the main monitor 241 inside the monitor panel 240 can be visually recognized from above, which is suitable for low-angle photography.
[0235] When the imaging device 200 is set to the XR mode, the live preview image is not displayed on the main monitor 241. By wearing the virtual image display device 100, it seems to the user that the live preview image is displayed in the area of the main monitor 241.
[0236] According to Fig.13 In the second display example shown, the generated live preview image LV1 is restricted by the movable range of the main monitor 241 of the imaging device 200. However, since the main monitor 241 cannot rotate freely, it is not the case that the display orientation of the live preview image LV1 is unknown. Since the operation method is the same as that of the actual imaging device 200, it has the advantages of good operability and no hesitation.
[0237] Moreover, a simulated monitor panel can be displayed instead of the actual monitor panel 240, and the live preview image LV1 is displayed on the simulated monitor panel. In this case, the movable range of the simulated monitor panel can be made larger than that of the actual monitor panel 240 (for example, 10 degrees larger). Furthermore, it is preferable to switch the movable range of the simulated monitor panel (freely, step by step,...) by selection, or to change the color or shape of the frame of the monitor panel (such as a real camera and a transparent photo frame shape) so as to know the switching situation.
[0238] <The Third Display Example in XR Mode>
[0239] Fig.14 It is a diagram showing the third display example of the live preview image and the like when the imaging device is set to the XR mode.
[0240] Fig.14 The third display example shown is Fig.12 a modified example of the first display example shown in (B), showing the case where the live preview image LV1 is overlapped and displayed on a simulated monitor panel larger than the closed monitor panel 240 of the imaging device 200.
[0241] The size of the geometrically deformed and overlapped live preview image LV1 can be freely set. In Fig.14 the third display example shown, by wearing the virtual image display device 100, it seems to the user that the live preview image LV1 is displayed on a simulated monitor panel that is larger than the size of the monitor panel 240 and actually does not exist.
[0242] In addition, the size of the simulated monitor panel is not limited to a certain size, and it can also be enlarged and reduced in联动 with operations such as magnifying the corners of the stretched monitor panel 240. In this case, it is preferable that the monitor panel 240 is provided with a touch feeling imparting member that gives a touch feeling as if the picture frame is actually stretched.
[0243] As a method for detecting the magnifying operation of the corners of the stretched monitor panel 240 in this case, the size of the simulated monitor panel can be enlarged and reduced corresponding to the magnifying and reducing actions by using the gesture control principle through image recognition of the built-in camera 101 of the virtual image display device 100, or other methods such as detecting the case where a pinch operation is performed on the touch panel provided on the monitor panel 240 of the imaging device 200 and making it联动 therewith can also be used.
[0244] Moreover, when the corners of the monitor panel 240 are stretched, the aspect ratio of the simulated monitor panel remains unchanged, and it expands towards the direction of stretching the simulated monitor panel and upwards, or expands uniformly in the direction of stretching and up and down, or expands downwards.
[0245] <The Fourth Display Example in XR Mode>
[0246] Fig.15 It is a diagram showing the fourth display example of an instant preview image or the like when the imaging device is set to the XR mode.
[0247] Fig.15 The fourth display example shown is about the case where the instant preview image LV1 is overlapped and displayed on the area of the monitor panel that actually does not exist.
[0248] The monitor panel 240 is as Figure 3 (B) and shown in Figure (C) is provided with a rotation mechanism that can pitch relative to the camera body 203, but does not have a rotation mechanism that rotates the instant preview image LV1 as Fig.15 shown.
[0249] In Fig.15 In the fourth display example shown, a virtual rotation mechanism is provided on the right side of the back of the camera body 203 to rotate the monitor panel appropriately. As a result, the instant preview image LV1 appears to be approximately the same size as the monitor panel 240, and one end thereof is fixed to the imaging device 200, and it is displayed in a state where the monitor panel is pitched towards the user's direction and appears to be directly facing the line of sight direction.
[0250] In addition, in Fig.15In this case, the display is performed by pitching horizontally with one side on the left or right as the axis. However, this is just an example, and it is also possible to perform pitching display in the vertical direction with one side on the top or bottom as the fixed axis. Moreover, it is also possible to perform an irregular display by combining pitching in two different directions. In this case, the monitor panel can be represented by a virtual image in such a way that it is not seemingly fixed to the imaging device 200 on either side, but is connected by a hinge mechanism.
[0251] <Fifth Display Example in XR Mode>
[0252] Fig.16 This is a diagram showing the fifth display example of an instant preview image or the like when the imaging device is set to the XR mode.
[0253] Fig.16 The fifth display example shown is Fig.15 a modified example of the fourth display example shown, and is shown for the case where the orientation of the camera body 203 is opposite. That is, Fig.16 in the fifth display example shown, a virtual rotation mechanism is provided on the left side of the back of the camera body 203 to appropriately rotate the monitor panel. As a result, the instant preview image LV1 is displayed to have a size approximately the same as that of the monitor panel 240 and is directly facing the line of sight.
[0254] In this way, it is preferable to provide a virtual two-axis or three-axis hinge for rotating the monitor panel, and to automatically rotate and display the monitor panel relative to the camera body so that the instant preview image LV1 is always directly visible. Thereby, an operation feeling close to that of using a real camera can be obtained, without feeling confused, and it will not become an unnatural composite image like the instant preview image LV1 floating in the air, and the quality of the photographing behavior can be improved by performing more realistic photography.
[0255] Fig.17 This is a diagram showing a modified example of the fifth display example of the instant preview image or the like with respect to Fig.16 that shown.
[0256] Fig.17 (A) is a diagram showing the same display state as the fifth display example of the instant preview image or the like shown in Fig.16 that shown. As shown in Fig.17 (A), the imaging device 200 is shown for the case where the entire imaging device 200 enters the field of view of the XR display 140.
[0257] Fig.17 (B) shows the case where the imaging device 200 shown in Fig.17 (A) moves to the left with respect to the field of view of the XR display 140 and the left half of the imaging device 200 is out of the field of view.
[0258] In this case, the virtual image display device 100 can also receive the live preview image LV from the camera device 200, so Fig.17 As shown in (B), the live preview image LV1 can be displayed on the XR display 140. That is, the virtual image display device 100 makes the live preview image LV1 appear relative to the camera device 200. Fig.17 (B) is moved to a position further to the right than the original display position, and the entire live preview image LV1 is displayed.
[0259] Fig.17 (C) About Fig.17 (B) shows a situation where the camera 200 moves further to the left, and the left 2 / 3 of the camera 200 is out of the field of view of the XR display 140. In this case, the virtual image display device 100 makes the live preview image LV1 relative to the camera 200. Fig.17 (C) Upward Fig.17 The display position shown in (B) is moved further to the right, and the entire live preview image LV1 is displayed.
[0260] Fig.17 (D) About Fig.17 (C) shows a case where the camera 200 moves further to the left and the entire camera 200 is out of the field of view of the XR display 140. In this case, the virtual image display device 100 makes the live preview image LV1 appear relative to the camera 200. Fig.17 (D) Upward Fig.17 The display position shown in (C) is moved further to the right, and the entire live preview image LV1 is displayed. In this case, the left end of the live preview image LV1 coincides with the left end of the field of view.
[0261] exist Fig.17 In the display example shown, the case where the camera device 200 moves to the left and out of the field of view is shown, but the present invention is not limited to this. In the case where the camera device 200 moves to the right and out of the field of view, moves upward and out of the field of view, and moves downward and out of the field of view, the instant preview image LV1 can also be moved in the same manner as described above, so that the entire instant preview image LV1 can be displayed.
[0262] And, including Fig.15 In the case of other display examples such as the fourth display example such as the live preview image shown, when the imaging device 200 is out of the field of view, the live preview image LV1 can be moved relative to the imaging device 200 in the same manner as described above.
[0263] Thus, even if the imaging device 200 is out of the field of view, the live preview can be continuously viewed by continuously displaying the live preview image LV1 around the position out of the field of view.
[0264] In addition, in Figures 12 to 17 the first to fifth display examples shown, photographic information can be displayed at the lower end of the live preview image LV1 together with the live preview image LV1 of the composite display.
[0265] <Sixth Display Example in XR Mode>
[0266] Fig.18 FIG. is a diagram showing a sixth display example of a live preview image and the like when the imaging device is set to the XR mode.
[0267] Fig.18 The sixth display example shown shows a case where photographic information INFO is superimposed and displayed in an area wider than the sub-monitor 242 of the closed monitor panel 240 of the imaging device 200 (refer to Figure 1 ).
[0268] In addition to the photographic information (refer to Fig.12 (A)) displayed on the normal sub-monitor 242, the photographic information INFO includes a histogram. This histogram is a luminance histogram representing the luminance distribution of the live preview image during current photography. The histogram is not limited to the luminance histogram, and may also be an RGB histogram for each of R (red), G (green), and B (blue) colors.
[0269] Moreover, the photographic information INFO is not limited to information representing photographic conditions such as shutter speed, aperture value, EV value (exposure value), etc., but also includes various photographic information. For example, metadata representing subject information, ISO sensitivity, detection area information such as a face recognition frame or an AF frame, etc. can be included in the photographic information INFO.
[0270] The virtual image display device 100 can receive text data or an image representing the photographic information INFO from the imaging device 200. And when the photographic information INFO is displayed on the sub-monitor 242, the virtual image display device 100 can read the photographic information INFO through image recognition to obtain it.
[0271] Preferably, the photographic information INFO includes camera identification information for distinguishing data from other imaging devices, but the camera identification information is not displayed when the photographic information INFO is displayed.
[0272] On the other hand, when the virtual image display device 100 compositely displays the photographic information INFO, it preferably superimposes and displays the live preview image LV on the area of the viewfinder 243.
[0273] Thus, when the user wearing the virtual image display device 100 approaches the viewfinder 243 , the user can visually recognize the live preview image LV through the eyepiece portion of the viewfinder 243 .
[0274] <First Appearance Example of Imaging Device>
[0275] Fig.19 This is a diagram showing a first external appearance example of the imaging device when the imaging device is set to the XR mode.
[0276] like Fig.10 As shown, AR markers M1 to M9 are provided on the camera 200 , and the virtual image display device 100 can identify each surface of the camera 200 or a polygon representing the appearance shape of the camera 200 by detecting the positions of the AR markers M1 to M9 from the field of view image IM.
[0277] The virtual image display device 100 combines and synthesizes a virtual object image (skin) representing the appearance of the imaging device 200 based on the AR markers M1 to M9 , thereby making it possible to display an appearance different from the original appearance of the imaging device 200 .
[0278] As a method of setting an appearance different from the appearance of the camera device 200, the above example describes a method of pasting and synthesizing a virtual object image (skin) representing the appearance of the camera device 200, but the virtual object image (skin) may not be pasted flatly on the pasted surface. Fig.19 The lens 200B on the front surface of the image pickup device 200 has a shape that looks like a gas component protruding or other components are installed.
[0279] Furthermore, there may be multiple virtual object images (skins) for one AR marker.
[0280] Furthermore, the lens 200B may be detachably configured. Furthermore, an operation of detaching the lens 200B by grasping the lens 200B (for example, a gesture simulating lens replacement such as rotating and pulling) may be detected by image recognition, and the detachment may be performed in conjunction with the operation.
[0281] And, in Fig.19 In the first external appearance example of the imaging device 200 shown, the monitor on the back side of the camera body 203 is enlarged, and the switches on the back side are changed.
[0282] <Second Appearance Example of Imaging Device>
[0283] Fig. 20 This is a diagram showing a second external appearance example of the imaging device when the imaging device is set to the XR mode.
[0284] and Fig.19Similarly to the first appearance example shown, the virtual image display device 100 superimposes and synthesizes a virtual object image (skin) representing the appearance of the imaging device 200 , thereby changing the appearance of the imaging device 200 .
[0285] exist Fig. 20 In the second appearance example of the imaging device 200 shown in FIG. 1 , the upper surface of the camera body 203 is changed to a simple shape with only the shutter button 232. Fig. 20 In the case of the second appearance example shown, it is not visible Figure 1 The shutter speed / sensitivity dial 234 and the exposure correction dial 236 are shown in the figure.
[0286] Preferably, when the camera device 200 is set to XR mode and a portion of the operating system of the camera device 200 is invisible due to a change in appearance, the camera device 200 disables the invisible operating system (physical shutter speed / sensitivity dial 234 and exposure correction dial 236, etc.).
[0287] <Third Appearance Example of Imaging Device>
[0288] Fig.21 This is a diagram showing a third external appearance example of the imaging device when the imaging device is set to the XR mode.
[0289] and Fig.19 Similarly to the first appearance example shown, the virtual image display device 100 superimposes and synthesizes a virtual object image (skin) representing the appearance of the imaging device 200 , thereby changing the appearance of the imaging device 200 .
[0290] exist Fig.21 In the third external appearance example of the imaging device 200 shown, the position of the viewfinder is changed from the left end to the center, and the monitor and the operating system on the back side of the camera body 203 are deleted.
[0291] <Case in which the appearance of the imaging device is divided into a plurality of regions and changed>
[0292] Fig. 22 This is a conceptual diagram when the appearance of the imaging device is divided into a plurality of areas and changed.
[0293] The first memory 120 of the virtual image display device 100 stores a virtual object image representing the appearance for each of a plurality of divided regions into which the appearance of the imaging device 200 is divided.
[0294] In the example shown in FIG. 0 , the interchangeable lens section 151 , the leather exterior section 152 , the operation dial section 153 , the upper exterior section 154 , the finder section 155 , and the monitor section 156 are stored in the first memory 120 as virtual object images.
[0295] The interchangeable lens unit 151 includes a plurality of interchangeable lens units of different types, such as a wide-angle lens and a telephoto lens. The leather exterior unit 152 includes black leather, brown leather, red leather, crocodile pattern, etc. The upper exterior unit 154 includes exteriors of textures and colors such as silver, black, gold, and two-tone. The viewfinder unit 155 and the monitor unit 156 also include various shapes and sizes.
[0296] The user can select and combine virtual object images representing the appearance of each of these divided areas by operating the operation unit 230 , thereby making the appearance of the imaging device 200 the original appearance.
[0297] That is, the XR processor 110 receives one or more virtual object images representing the appearance through the user operation on the operation unit 230, appropriately geometrically transforms these virtual object images according to the three-dimensional position and posture of the camera device 200, and outputs them to the XR display 140. As a result, the user can recognize the camera device visually recognized via the XR display 140 as a camera device having the appearance of the virtual object image selected by the user.
[0298] Furthermore, when combining multiple virtual object images, multiple valid combinations are prepared in advance, and the user can select from the multiple combinations. In this case, the XR processor 110 receives the virtual object images of the combination selected from the multiple combinations, appropriately geometrically transforms the virtual object images according to the three-dimensional position and posture of the camera device 200, and outputs them to the XR display 140.
[0299] Furthermore, effective combinations of a plurality of virtual object images may be determined based on past learning results (determined by AI (artificial intelligence)) and registered or updated.
[0300] By changing the operability or appearance of the GUI based on past combination data (not limited to AI), an effective combination of virtual object images that matches the user's preferences or user's operability obtained based on past data can be provided. By learning this with AI, the operability can be made higher and more in line with preferences.
[0301] Furthermore, if these technologies are further used to collect past combination data for a large number of users and a large number of user preferences are stored on the data server, customization can be performed as needed to improve the appearance or operability of the user convenience for an unspecified large number of users or a specific user group (left-handed users, female users, children, elderly users, etc.). Furthermore, a method of selectively performing these operations on a menu screen or a setting screen, or automatically performing these operations with the cooperation of AI and the user, can also be used.
[0302] Furthermore, it is also possible to configure such that not only the virtual object image is replaced, but also the virtual object image can be switched. Figure 1 The arrangement of the shutter speed / sensitivity dial 234 and the exposure correction dial 236 shown in the figure is changed.
[0303] Furthermore, although it has been described that the user can select and combine virtual object images representing the appearance of each segmented area through operation on the operation unit 230 to perform an operation to change the appearance of these camera devices 200 to the original appearance, in order to intuitively change the operation to the original appearance, it is also possible to perform gesture control using image recognition technology. That is, based on the field of view image IM of the built-in camera 101 provided in the virtual image display device 100, it is possible to recognize through image recognition that the user has performed a gesture to change the virtual object image representing the appearance, and by controlling the display to move or replace the virtual object image representing the appearance corresponding to the gesture, thereby changing the virtual appearance.
[0304] Moreover, a virtual object image (skin) is pasted on any divided area, but for intuitive understanding, the image of the camera device 200 displayed on the XR display 140 can also be displayed by increasing the brightness of the image, or by displaying a light beam indicating that the inner part is luminous around it, so that the image of the camera device 200 used in the image field of view appears to be luminous.
[0305] In addition, these virtual object images are appropriately geometrically transformed and synthesized according to the three-dimensional position and posture of the camera device 200.
[0306] Furthermore, by replacing the interchangeable lens unit 151 from an actual wide-angle lens to a virtual telephoto lens, for example, the virtual image display device 100 can display the live preview image LV1 in an enlarged manner as if it were photographed by the telephoto lens.
[0307] <Camera device compatible with use in XR environment>
[0308] Fig.23 This is a stereoscopic diagram of a camera device suitable for use in an XR environment.
[0309] Fig.23 The imaging device 280 shown includes a simulation lens 281 , three real lenses 282 , and a simulation monitor 283 .
[0310] The simulation lens 281 is a lens that only constitutes the appearance and has no photographic function.
[0311] The three real lenses 282 have different focal lengths, and the user can select and use them appropriately. The live preview image captured by the real lens 282 is sent to the virtual image display device 100 .
[0312] The simulation monitor 283 is a variable simulation monitor that has no display function and is rotatable relative to the camera body.
[0313] When the aforementioned camera device 200 is used in an XR environment, the real-time preview image is not displayed on the main monitor 241, etc., and the virtual image display device 100 causes the real-time preview image LV1 to be superimposed and displayed on the display area of the camera device 200. However, in the case of the camera device 280 in this example, the virtual image display device 100 causes the real-time preview image to be superimposed and displayed on the simulation monitor 283 of the camera device 280.
[0314] Furthermore, if the simulation monitor 283 is rotated, the virtual image display device 100 can display the real-time preview image on the rotated simulation monitor 283 in a superimposed manner. Fig.13 , Fig.15 and Fig.17 The same situation is shown in the instant preview image LV1.
[0315] <Virtual Camera>
[0316] Fig.24 It is a perspective view showing the back side of an embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0317] Fig.24 The virtual camera 290 shown is a virtual camera having a handle that can be held by at least one hand, and is provided with operation switches. In this case, although the term "virtual camera" is used, the handle has an actual form and does not refer to a camera based on a virtual display.
[0318] The virtual camera 290 is a simulated camera that does not have an imaging unit such as a lens and an imaging element and has no camera function. The virtual camera 290 in this example has an appearance with only a handle.
[0319] An operating member including at least a shutter button is provided on the handle portion of the virtual camera 290 . The operating member is a simulated switch that is movable by a user's operation but does not generate an operation signal.
[0320] The virtual camera 290 does not require a lens or an imaging element, and the communication unit can be greatly simplified and lightweight, and can be easily miniaturized by applying folding, etc. Furthermore, even if the virtual camera 290 does not have a camera function, the user can get the same sense of operation of framing or operating components as a normal camera.
[0321] The operating components of the virtual camera 290 in this embodiment are simulated switches that are movable by user operation but do not generate operation signals. However, as other embodiments, the operating components of the virtual camera 290 can be operated in a manner that actually obtains a sense of operation. By having an operation detection unit that detects user operations and a communication unit that sends operation signals (excluding the sending of instant preview images), the operation content performed by the user on the operating components of the virtual camera 290 is sent to the virtual image display device 100 (or the virtual image display system 10 or the XR server 300 described later) via the communication unit, thereby reflecting the operations performed by the user in real time.
[0322] Furthermore, the method of reflecting the operation performed by the user in real time is not limited to this method. As another variation, each operating component of the virtual camera 290 can be operated in a manner that actually obtains the sense of operation, but even if there is no operation detection unit or communication unit for detecting the operation, the quality of the sense of operation can be improved. In this case, the camera device 200 and the camera viewfinder for image recognition of the virtual image display device 100 and the virtual image display system 10 are used to detect the movement of the user's finger or the operation status of the operating component through image recognition, thereby reflecting the operation performed by the user in real time. The time required for reflection or the error rate of reflection in this case is usually inferior to the performance of the device with the operation detection hand and the communication unit, but according to the performance of image recognition, it can be made to be no less than that in actual application.
[0323] Fig.25 It is a perspective view showing the front side of an embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0324] Fig.25 The virtual camera 290 shown in (A) is viewed from the front side. Fig.24 A stereoscopic view of the virtual camera 290 is shown from the back side.
[0325] As described above, the virtual camera 290 is a simulated camera that does not include an imaging unit such as a photographic lens and an imaging element and does not have a camera function.
[0326] Fig.25 The virtual camera 290B shown in (B) has a small camera 293B mounted on a smartphone built into the front surface of the handle so as not to be hidden by the grasping hand.
[0327] In this case, in addition to the operation detection unit that detects the user's operation, there is also a communication unit that includes image transmission, so that the user's operation content on each operating component of the camera device 290B and the real-time preview image obtained by the small camera 293B can be sent to the virtual image display device 100 (or the virtual image display system 10 or the XR server 300 described later) via the communication unit. The following process is the same as the embodiment using the smartphone described later.
[0328] Furthermore, the virtual camera 290B can be used as a real camera device using the camera function of the camera handle. Furthermore, when the camera function of the camera handle is used, the instant preview image photographed by the camera handle can be supplemented or resolution-enhanced, or whitened or blackened, etc. can be corrected as a final image using the field of view image IM of the built-in camera 101 of the virtual image display device 100 or the instant preview image obtained from the XR server 300, as described later in the embodiment using a smartphone.
[0329] Fig.25 The virtual camera 290C shown in (C) has a small camera 293C for image recognition built into the front surface of the handle and not hidden by the holding hand. The resolution of the small camera 293C can also be lower than Fig.25 (B) shows a small camera 293B.
[0330] In addition to the operation detection unit for detecting the operation, the virtual camera 290C also has a communication unit, but the communication unit at this time may not send images. In the case where the communication unit does not include image sending, instead of sending the user's operation content on each operating component of the virtual camera 290C and the image data acquired by the small camera 293C, it can use image recognition and the like to digitize or characterize the information of the subject facing the virtual camera 290C or the range information of the set field of view angle from the image data acquired by the small camera 293, and send it to the virtual image display device 100 (or virtual image display system 10 or XR server 300) through the communication unit together with the operation content of each operating component. Thus, compared with the communication unit including image sending, it can be constructed to be small and low-cost. The process of the following processing can be achieved by combining the processing based on the implementation method of the simulation camera without camera function and the implementation method using a smart phone described later, so its description is omitted.
[0331] Fig.26 It is a perspective view showing another embodiment of a virtual camera constituting the virtual image display system according to the present invention.
[0332] Fig.26 The virtual camera 292 shown is a device that allows the smartphone 291 to communicate with the Fig.24The virtual camera 290 shown is a combined virtual camera.
[0333] The virtual camera 292 can be used as a virtual camera without a camera function, or can be used as a real camera device using the camera function of the smartphone 291. In addition, when the camera function of the smartphone 291 is used, the instant preview image photographed by the smartphone 291 can be supplemented or resolution-enhanced, and whitening and blackening corrections can be performed as the final image using the field of view image IM of the built-in camera 101 of the virtual image display device 100 or the instant preview image obtained from the XR server 300, as described later.
[0334] Fig.24 The virtual camera 290 shown and the virtual camera 292 when the camera function of the smartphone 291 is not used cannot take instant preview images, but by setting a virtual camera unit, images (instant preview images) corresponding to the field of view of the virtual cameras 290 and 292 can be obtained.
[0335] Fig. 27 It is a diagram showing an implementation method of a method for acquiring a live preview image photographed by a virtual camera.
[0336] As described above, the built-in camera 101 of the virtual image display device 100 captures the visual field image IM corresponding to the visual field of the user including the virtual camera 290, and acquires the captured visual field image IM. Fig. 27 , L1 is the optical axis of the built-in camera 101.
[0337] When the virtual camera 290 has a camera function, the live preview image of the photography can be determined according to the three-dimensional position, posture (photography direction) and field of view of the virtual camera 290. Fig. 27 , L2 is the optical axis assumed in the virtual camera 290 and can be obtained based on the three-dimensional position and posture of the virtual camera 290.
[0338] The virtual image display device 100 can acquire information indicating the three-dimensional position and posture of the virtual camera 290 with respect to the XR display 140 , similarly to the imaging device 200 .
[0339] Therefore, if the field of view of the virtual camera 290 is set, it is possible to determine to which range of the field of view image IM photographed by the built-in camera 101 the live preview image photographed according to the three-dimensional position, posture and field of view of the virtual camera 290 corresponds.
[0340] That is, the virtual image display device 100 can cut out an image corresponding to the field of view angle of the virtual camera 290 from the field of view image IM photographed by the built-in camera 101, and obtain it as an instant preview image. In addition, the field of view angle of the built-in camera 101 is wider than the field of view angle set for the virtual camera 290, and the field of view image IM includes the instant preview image photographed by the virtual camera 290.
[0341] In this way, the virtual image display device 100 can obtain an instant preview image from the field of view image IM photographed by the built-in camera 101, which the virtual camera 290 could photograph if it had an imaging unit.
[0342] The virtual image display device 100 can geometrically transform the obtained instant preview image and display it in a display area set relative to the virtual camera 290 (for example, an area adjacent to the virtual camera 290, i.e., the handle portion).
[0343] And, in Fig.26 In the case of the virtual camera 292 shown, the virtual image display device 100 can geometrically transform the obtained instant preview image and display it superimposed on the display area of the smartphone 291 of the virtual camera 292.
[0344] Moreover, the virtual image display device 100 can detect user operations (gestures) of switches in the virtual camera 290 by analyzing the field of view image IM photographed by the built-in camera 101, and can perform recording photography, saving, and other processes according to the user operations of the switches.
[0345] And, the virtual camera is not limited to the virtual camera with only a handle portion, and various virtual cameras with different appearance shapes, sizes, configurations of switches, or presence or absence can also be considered.
[0346] In addition, although the virtual camera 290 in this example does not have a camera function, it has real switches and a communication unit, etc. The operation information of the switches can also be sent to the virtual image display device 100 via the simple communication unit provided in the virtual camera 290.
[0347] <The Second Embodiment of the XR Display>
[0348] Fig.28 It is a diagram showing the second embodiment of the XR display of the virtual image display device.
[0349] Fig.28 The XR display 141 shown is an immersive display that covers the field of view, is a wearable device with a goggle-type virtual image display function, and is composed of a built-in camera 101, an image synthesis unit 143, a display element 145, and an optical system 147.
[0350] The image (field of view image IM) captured by the built-in camera 101 is output to the image synthesis unit 143 .
[0351] The image synthesis unit 143 adds the geometrically transformed live preview image LV1 to another input, synthesizes (pastes) the live preview image LV1 to the display area of the camera image included in the field of view image IM, and outputs the result to the display element 145 .
[0352] The display device 145 displays the visual field image IM synthesized with the live preview image LV1 . The visual field image IM is displayed as a virtual image at a display position in front of the user's eyes by the optical system 147 .
[0353] The user wearing the goggle-type XR display 141 can visually recognize the virtual image of the visual field image IM displayed on the display element 145 via the optical system 147. Since the user's visual field is covered by the goggle-type XR display 141, he cannot see the part outside the visual field image IM and can immerse himself in the virtual world.
[0354] In addition, Fig.28 In the figure, only one XR display 141 is shown, but in the case of the goggles type, a pair of left and right XR displays 141 corresponding to both eyes are provided. Therefore, the built-in camera 101 and the like are also respectively provided on the left and right, and the live preview image LV1 also generates live preview images for the left and right.
[0355] [Second Embodiment of Virtual Image Display System]
[0356] Fig.29 It is a block diagram showing a second embodiment of the virtual image display system according to the present invention.
[0357] Fig.29 The virtual image display system 11 shown is composed of a virtual image display device 180 , a virtual camera 290 , and a server (XR server) 300 .
[0358] The virtual image display device 180 and Figure 6 The virtual image display device 100 shown is substantially the same as the virtual image display device 100 , but is different from the virtual image display device 100 in that necessary information is transmitted and received with the XR server 300 .
[0359] exist Fig.29 In the example, the XR server 300 includes a database (DB: database) 302 that manages a three-dimensional image (XR space construction data) having three-dimensional information of a real space.
[0360] The XR server 300 collects images taken by a multi-view camera 320, a surveillance camera 330, a driving recorder 340, an unmanned camera 310, or multiple cameras including a mobile terminal camera configured in the real space, generates the latest XR space construction data and manages it in the database 302.
[0361] The database 302 digitizes blocks, cities, shopping streets, tourist spots, etc., and manages them as XR space construction data. In addition, it is preferable to make multiple XR servers 300 cooperate and manage XR space construction data in a wider area.
[0362] The XR processor 111 of the virtual image display device 180 has Figure 6 The present invention has the same function as the XR processor 110 of the virtual image display device 100 shown in the figure, but also cuts out the XR space construction data corresponding to the photographic area of the virtual camera 290 from the XR space construction data managed by the XR server 300 based on the information (the first information) representing the three-dimensional position, posture and field of view of the virtual camera 290, transforms the cut three-dimensional XR space construction data into a two-dimensional image, and uses the two-dimensional image as an instant preview image.
[0363] The XR processor 111 uses the camera internal parameters of the virtual camera 290 (in this case, the pre-set field of view angle or focal length) to perform perspective projection transformation on the three-dimensional XR space construction data read from the database 302 according to the three-dimensional position and posture (photographic direction) of the virtual camera 290, and transforms it into a two-dimensional image, and uses it as the real-time preview image LV.
[0364] Furthermore, the XR processor 111 analyzes the field of view image IM photographed by the built-in camera 101 to determine the display area of the virtual camera 290, obtains the transformation information used when the instant preview image LV is overlapped with the display area of the virtual camera 290, and performs a geometric transformation on the instant preview image LV according to the obtained transformation information. However, the transformation information is obtained according to the relative three-dimensional position and posture of the XR display 140 of the virtual image display device 180 and the virtual camera 290, and when the field of view image IM is not used in the XR display, the built-in camera 101 can be omitted. The XR processor 111 can also use image data sent from the camera devices of other people around to generate a better instant preview image.
[0365] Furthermore, preferably, when generating the latest XR space construction data, the XR server 300 uses not only the current image but also the images recorded in the past as images such as the multi-view camera 320, and constructs XR space construction data with higher image quality than a single image by synthesizing multiple images.
[0366] The number of pixels in an image is increased by using multiple images and processing and combining them with a slight pixel deviation.
[0367] Preferably, when creating a complement image using a plurality of images, mutual complementation of whitishness, blackness, etc. is performed to complement the portion where no image data is obtained or the quality is poor, and further, when the poor quality portion is complemented, the brightness of the complement image is corrected so that the up-down relationship of light and dark generated by the intensity of light is not reversed. Furthermore, when correcting the brightness of the complement image, it is preferred to correct the gamma curve in a manner that makes it converge to the overall display latitude (printing latitude), or correct the printing parameter curve in a manner that makes it converge to the overall printing latitude.
[0368] When synthesizing with the main subject image, priority is given to correcting the brightness, light direction, color temperature, and gamma curve (printing parameters) to correspond to the image of the multi-view camera 320, etc. Furthermore, when synthesizing images of parts other than the main subject based on the past images and the current image of the multi-view camera 320, it is preferred to detect the changing conditions such as weather and time from the current image, and transform and synthesize the past images and XR space construction data based on the detected changing conditions such as weather and time.
[0369] Furthermore, the XR server 300 may also synthesize the virtual object image into the XR space construction data. For example, when the XR space construction data includes a character image, the XR server 300 may transform it into an avatar image that simulates the character, and replace the character image with the avatar image. Furthermore, when the XR server 300 identifies a character based on the character image and the identified character registers its own avatar image in the XR server 300, the XR server 300 may change the character image to the registered avatar image.
[0370] Thereby, the virtual image display device 180 can acquire the live preview image LV including the avatar image from the XR server 300 .
[0371] In addition, it is conceivable that an image including oneself is used by others, but in this case, it is preferable to notify the situation or issue a warning through the virtual image display device 100. In addition, it is possible to allow / disallow the image including oneself, and when it is set to allow, the notification can be omitted.
[0372] Furthermore, if the image of oneself is used by another person, and the setting is not permitted, the other party can be notified of the content and warned, or a signal to lock the other party's photography can be sent, or a signal to mosaic the image part of oneself can be sent, or a generation prohibition signal to not generate the image of the other party can be sent. If the other party's device receives the generation prohibition signal, it will prioritize image processing to avoid generating an image of the prohibited subject (deleting, smearing in the background, etc.).
[0373] Moreover, in this embodiment, the virtual image display device 180 obtains the instant preview image LV from the XR server 300, but is not limited to this. The virtual image display device 180 may also send the information required for extracting the instant preview image LV to the XR server 300. The XR server 300 obtains the instant preview image LV and sends it to the virtual image display device 180, and may also perform a geometric transformation on the instant preview image LV and send the instant preview image LV1 after the geometric transformation to the virtual image display device 100.
[0374] Furthermore, the XR server 30 may also obtain the field of view image IM by obtaining information indicating the field of view range of the XR display 140 in the same manner as the method of obtaining the live preview image LV, and send it to the virtual image display device 100 .
[0375] Furthermore, in this embodiment, the virtual camera 290 is used, but it is also applicable when the camera 200 is used. In this case, when the camera 200 is used in the XR environment, the camera 200 does not shoot a live preview image, but obtains it from the XR server 300.
[0376] Furthermore, the latest XR space construction data is registered in the database 302 of the XR server 300 , but it is not necessarily real-time XR space construction data, so the live preview image LV generated from the latest XR space construction data is not necessarily a real-time image.
[0377] Therefore, the virtual image display device 180 preferably adjusts the brightness or color tone of the live preview image LV according to the brightness or color tone of the visual field image IM photographed by the built-in camera 101 , for example.
[0378] In addition, the posture of the virtual camera 290 can be arbitrarily operated by the user holding the virtual camera 290, but the three-dimensional position of the virtual camera 290 is not a real position and can be changed to any position through user operation, so that the user can enjoy photography of tourist attractions at home.
[0379] [First Embodiment of Virtual Image Display Method]
[0380] Fig.30This is a flowchart showing a first embodiment of a virtual image display method according to the present invention.
[0381] Fig.30 The virtual image display method of the first embodiment shown in the figure is performed by the virtual image display device 100 and mainly includes: Figure 6 The method is performed by the XR processor 110 of the virtual image display device 100 shown.
[0382] exist Fig.30 In step S100 , the XR processor 110 acquires the field of view image IM photographed by the built-in camera 101 .
[0383] The XR processor 110 analyzes the visual field image IM and extracts a camera image representing the camera device 200 included in the visual field image IM (step S102). In the case where the camera device 200 is handheld and a live preview image is photographed by the camera device 200, as shown in FIG. Fig. 9 As shown, the visual field image IM photographed by the built-in camera 101 generally includes a camera image representing the imaging device 200 .
[0384] The XR processor 110 determines whether the camera device 200 is an XR-compatible camera from the extracted camera image (step S104). This determination can be performed, for example, by comparing the extracted camera image with the appearance data of the XR-compatible camera pre-stored in the first memory 120. In addition, it can be determined by whether the AR marker M set on the camera device 200 is consistent with the pre-registered AR marker M.
[0385] If the XR processor 110 determines that the camera 200 photographed by the built-in camera 101 is an XR-compatible camera (in the case of “Yes”), it then detects whether the camera 200 is in XR mode or non-XR mode (step S106 ).
[0386] When the camera device 200 is set to the XR mode and the AR marker M is displayed on the main monitor 241 or the sub-monitor 242, the XR mode / non-XR mode can be detected by detecting the AR marker M. Furthermore, when the camera device 200 has a dedicated mode switching lever, it can be performed by performing image recognition on the position of the mode switching lever. Moreover, when the camera image is extracted from the field of view image IM, the XR processor 110 can automatically connect to the camera device 200 wirelessly from the virtual image display device 100, and detect the XR mode / non-XR mode based on the communication result with the camera device 200. For example, when there is no response from the camera device 200 to the call from the virtual image display device 100, it can be determined that the camera device 200 is not set to the XR mode.
[0387] In step S108, if it is determined that the camera device 200 is not set to the XR mode (“No”) according to the detection result of the XR mode / non-XR mode, the XR processor 110 sets the superimposition synthesis function of the live preview image in the virtual image display device 100 (smart glasses) to OFF (step S110), and returns to step S100. As a result, the user can only see the real world through the smart glasses.
[0388] On the other hand, in step S108, if it is determined that the camera device 200 is set to the XR mode based on the detection result of the XR mode / non-XR mode (in the case of "yes"), the display area A of the camera device 200 is determined based on the extracted camera image (reference Fig.10 The display area A can be determined by detecting the outline of the quadrilateral of the monitor panel 240 or detecting the AR markers M2 to M4 attached to the imaging device 200 .
[0389] Next, the XR processor 110 obtains transformation information when the live preview image LV is to be overlapped with the display area A of the camera device 200. When the live preview image LV is affine transformed and overlapped with the display area A, the parameters (transformation matrix) of the affine transformation are obtained as the transformation information, and when the live preview image LV is projectively transformed and overlapped with the display area A, the transformation matrix of the projective transformation is obtained as the transformation information. In addition, the transformation matrix can be obtained based on the coordinates of multiple feature points of the live preview image LV and the coordinates of multiple corresponding points in the display area A corresponding to the multiple feature points.
[0390] Next, the XR processor 110 receives the live preview image LV from the camera 200 (step S116). The live preview image LV is continuous data, but in step S116, image data of one frame is received.
[0391] The XR processor 110 geometrically transforms the live preview image LV (image data of one frame) according to the transformation information acquired in step S114, and outputs the live preview image LV1 after the geometric transformation to the XR display 140 (step S118).
[0392] The XR display 140 inputs Fig.11 The geometrically transformed live preview image LV1 is displayed on the display element 144 (step S120 ). The user can visually recognize the live preview image LV1 superimposed on the real world (display area A of the camera device 200 ) through the optical system 146 of the XR display 140 .
[0393] The XR processor 110 determines whether to terminate the image display of the live preview image LV1 based on the XR display 140 (step S122). If the image display is not terminated, the process proceeds to step S100, and the process from step S100 to step S122 is repeated for each frame of the live preview image LV. If the image display is not terminated, the process based on the virtual image display device 100 is terminated.
[0394] [Display method based on camera device]
[0395] Fig.31 This is a flowchart showing a display method based on the imaging device involved in the present invention.
[0396] in addition, Fig.30 The camera-based display method shown is mainly composed of Figure 7 The camera processor 210 of the camera device 200 shown in FIG.
[0397] exist Fig.31 In step S200 , the camera processor 210 first determines whether the XR mode is set by the mode switching unit.
[0398] When the user cannot easily see the main monitor 241 in a bright place, or when the user wants to display the live preview image LV1 on the virtual monitor larger than the actual main monitor 241, or when the user wants to photograph various virtual objects including avatars, the XR mode is set to give priority to more natural ease of use. When the user wants to observe the live preview image displayed on the actual main monitor 241, the camera device 200 can be used separately according to the advantages / disadvantages of the XR mode and the non-XR mode by setting it to the non-XR mode.
[0399] In step S200, if it is determined that the XR mode is set (in the case of "yes"), the camera processor 210 displays the AR marker M on the sub-monitor 242 of the closed monitor panel 240. The AR marker M can be used as a marker to identify the display area A of the live preview image LV1 and / or as camera identification information for distinguishing from other camera devices.
[0400] Next, the camera processor 210 transmits the live preview image LV captured by the imaging unit 201 and subjected to RAW development processing to the virtual image display device 100 via the second communication unit 260 (step S220 ).
[0401] On the other hand, in step S200, if it is determined that the XR mode is not set (non-XR mode), the camera processor 210 determines whether the tiltable monitor panel 240 is open (step S230). If it is determined that the monitor panel 240 is open, the camera processor 210 outputs the live preview image LV captured by the camera unit 201 and processed by RAW development to the main monitor 241, and displays the live preview image LV1 on the main monitor 241. Thus, the user can confirm the current shooting range and the like from the live preview image LV displayed on the main monitor 241.
[0402] On the other hand, if it is determined that the monitor panel 240 is not open ("No"), the camera processor 210 then determines whether the viewfinder 243 as a hybrid viewfinder is switched to EVF or OVF (step S250). This determination can be made based on the user's operation of the viewfinder switching lever.
[0403] When the viewfinder 243 is switched to EVF, the camera processor 210 displays the live preview image LV1 on the EVF (step S260). On the other hand, when the viewfinder 243 is switched to OVF, the optical image (viewfinder image) is displayed on the OVF (step S270). In this case, the process of generating the live preview image LV1 can be stopped.
[0404] The user can check the live preview image LV1 displayed on the EVF or the live view image displayed on the OVF by looking into the eyepiece portion of the viewfinder 243 .
[0405] [Second Embodiment of Virtual Image Display Method]
[0406] Fig.32 This is a flowchart showing a second embodiment of the virtual image display method according to the present invention.
[0407] Fig.32 The virtual image display method of the second embodiment shown is Fig.30 The first embodiment shown in the figure is also mainly composed of Figure 6 The method is performed by the XR processor 110 of the virtual image display device 100 shown.
[0408] exist Fig.32 In the process, the XR processor 110 first confirms the camera device 200 (step S300).
[0409] Fig.33 Yes means to proceed Fig.32 The illustrated embodiment is a subroutine of the process of step S300 for confirming the imaging device.
[0410] exist Fig.33In the process, the XR processor 110 acquires the visual field image IM photographed by the built-in camera 101 (step S301), and extracts the camera image of the imaging device 200 present in the visual field image IM (step S302).
[0411] Next, the XR processor 110 analyzes the camera image and extracts the AR marker M (step S303). Since the AR marker M is attached to the XR-compatible camera, that is, the camera 200 in this example, the presence of the AR marker M can be used to confirm the presence of the camera 200.
[0412] If it is determined in step S304 that the AR marker M is present (in the case of YES), the process proceeds to step S305 , where the XR processor 110 starts communicating with the target device having the AR marker M.
[0413] Next, the XR processor 110 confirms whether the target device is the user's camera 200 through communication (step S306). The XR processor 110 confirms whether the target device is the user's camera 200 by, for example, confirming once whether it is a paired device. In addition, the virtual image display device 100 and the user's camera 200 can be paired by a known method.
[0414] In step S307 , if it is determined that the target device is the user's imaging device 200 (in the case of YES), the XR processor 110 confirms that the camera image included in the visual field image IM is the user's imaging device 200 (step S308 ).
[0415] On the other hand, in step S304, if it is determined that there is no AR marker M in the field of view image IM (camera image) (the "No" case), or in step S307, if it is determined that the target device is not the user's camera device 200 (the "No" case), the XR processor 110 fails to confirm (does not confirm) the user's camera device 200 (step S309).
[0416] Back to Fig.32 In step S310, if it is confirmed that the user's camera device 200 exists (in the case of "yes"), the XR processor 110 determines whether the camera device 200 is in XR mode (step S312). Since the determination method of the XR mode can be performed in the same manner as the above method, its description is omitted here.
[0417] When it is determined in step S312 that the camera device 200 is set to the XR mode (“Yes” case), the XR processor 110 confirms whether the instant preview image should be displayed superimposed on the display area A of the camera device 200, whether the playback image should be displayed superimposed on the display, or whether the photographic information IN F0 should be displayed superimposed on the display (step S314).
[0418] When the camera device 200 is used in an XR environment, it is capable of accepting user instructions as to which of the instant preview image, playback image or photographic information INFO should be displayed in the display area A of the camera device 200, and sending the information corresponding to the accepted user instructions (instant preview image, playback image or photographic information INF0) to the virtual image display device 100.
[0419] Therefore, based on the information (such as the instant preview image) received from the camera device 200, the XR processor 110 can confirm whether the instant preview image should be displayed in superimposition on the display area A of the camera device 200, whether the playback image should be displayed in superimposition, or whether the photographic information INF0 should be displayed in superimposition.
[0420] In step S318 , if it is determined that the real-time preview image is to be displayed, the XR processor 110 transitions to step S320 , and displays the real-time preview image received from the camera device 200 on the XR display 140 in an overlapping manner.
[0421] Fig.34 Yes means to proceed Fig.32 The subroutine of the embodiment of the process of step S3200 of overlapping display of the live preview image LV1 is shown.
[0422] exist Fig.34 In step S321 , the XR processor 110 receives a live preview image LV from the camera 200 .
[0423] Next, the XR processor 110 obtains the photographic information INF0 corresponding to the received live preview image LV (step S322). Here, the photographic information INF0 is, for example, text information, which is supplementary information associated with each frame or frames of the live preview image LV.
[0424] If it is determined in step S323 that the photographic information INFO exists (in the case of YES), the XR processor 110 proceeds to step S324 , where the photographic information INFO is synthesized on the peripheral area of the live preview image LV (step S324 ).
[0425] On the other hand, in step S323 , if it is determined that there is “no” photographic information INFO (in the case of “No”), the XR processor 110 skips step S324 and proceeds to step S325 .
[0426] In addition, even if the XR processor 110 obtains the photographic information INFO, if the handover information of the photographic information INFO includes information indicating "non-display", it is determined that there is "no" photographic information INFO, and step S324 is skipped to transition to step S325. Moreover, in the case where an instant preview image is produced by the operation of the camera 200 based on the data obtained from the XR server 300, even if the photographic information INFO of the instant preview image is obtained, in the case of the photographic information INFO of the instant preview image generated by the operation of the camera 200, it is determined that there is "no" photographic information INFO, and step S324 is skipped to transition to step S325.
[0427] In step S325 , the XR processor 110 acquires the field of view image IM from the built-in camera 101 , and detects the positions of the plurality of AR markers M on the imaging device 200 based on the acquired field of view image IM.
[0428] Then, the XR processor 110 performs the processing of steps S326 to S329 to overlap and display the live preview image LV1 on the display area A of the camera device 200 observed by the XR display 140. Fig.30 The processes of steps S112 , S114 , S118 , and S120 in the flowchart shown are the same, and thus detailed description thereof is omitted.
[0429] Back to Fig.32 In step S318 , if it is determined that the photographic information is displayed, the XR processor 110 transitions to step S330 , so that the photographic information INFO received from the camera device 200 is overlapped and displayed on the XR display 140 .
[0430] Fig.35 Yes means to proceed Fig.32 The above-mentioned subroutine is a subroutine of an embodiment of the process of step S330 of superimposed display of the photographic information INFO.
[0431] exist Fig.35 In step S331, the XR processor 110 sets the overlap display function of the live preview image LV1 in the virtual image display device 100 to OFF, and sets the overlap display function of the photographic information INFO to ON (step S331).
[0432] Next, the XR processor 110 receives photographic information INFO of text data from the camera 200 (step S332 ).
[0433] The XR processor 110 generates a photographic information image (raster image or vector image) representing the photographic information INFO based on the received photographic information INFO (step S333 ).
[0434] Then, the XR processor 110 performs the processing of steps S334 to S338 to cause the photographic information image representing the photographic information INF0 to be superimposed and displayed on the display area A of the camera device 200 observed by the XR display 140. Fig.34 The processing of steps S325 to S329 of the flowchart shown is different in that a photographic information image is displayed instead of the live preview image LV1 (ie, the displayed image is different), but the processing other than this is the same, and thus detailed description thereof is omitted.
[0435] Back to Fig.32 In step S318 , if it is determined that the playback image is displayed, the XR processor 110 transitions to step S340 , so that the playback image received from the camera device 200 is overlapped and displayed on the XR display 140 .
[0436] Fig.36 Yes means to proceed Fig.32 The illustrated embodiment is a subroutine of the process of step S340 of overlapping display of the playback images.
[0437] exist Fig.36 In the process, the camera processor 210 of the camera 200 identifies whether the playback image selected in the camera 200 is an image of the camera 200 (a photographic image recorded in the second memory 220 of the camera 200) or an image on a network (an image of a user managed by a server on the network) (step S280). In addition, the camera 200 at this time can be connected to the network, and can store the photographic image in the server on the network, or can read the photographic image from the server.
[0438] In step S281, if it is determined that the image is in the camera device 200, the camera processor 210 reads the playback image from the second memory 220 of the camera device 200 according to the user operation (step S282). In addition, the playback image may be read from a memory card that is removable from the camera device 200 instead of the second memory 220 as a built-in memory.
[0439] On the other hand, if it is determined in step S281 that the image is on the network, the camera processor 210 reads out the playback image from the user's image group stored on the network according to the user operation (step S283).
[0440] The camera processor 210 transmits the playback image read out in step S282 or S283 to the virtual image display device 100 (step S284 ).
[0441] The processes of steps S280 to S284 described above are processes in the imaging device 200 , and the following processes are processes by the virtual image display device 100 .
[0442] The XR processor 110 of the virtual image display device 100 receives the playback image transmitted from the camera 200 (step S341 ).
[0443] Then, the XR processor 110 performs the processing of steps S342 to S346 to cause the playback image to be superimposed and displayed on the display area A of the camera device 200 observed by the XR display 140. Fig.34 The processing of steps S325 to S329 of the flowchart shown is different in that a playback image is displayed instead of the live preview image LV1 (ie, the displayed image is different), but the processing other than this is the same, so the detailed description thereof is omitted.
[0444] In this way, the virtual image display device 100 can display the real-time preview image LV1, photographic information INFO or playback image in an overlapping manner on the display area A of the camera device 200 observed by the XR display 140, and the user wearing the virtual image display device 100 (smart glasses) can visually recognize the real-time preview image LV1, etc. superimposed on the real world through the smart glasses.
[0445] <Method for Changing the Appearance of an Imaging Device>
[0446] Fig.37 This is a flowchart showing an embodiment of a method for changing the appearance of an imaging device.
[0447] Fig.37 The camera device 200 shown in the figure is mainly modified by Figure 6 The method is performed by the XR processor 110 of the virtual image display device 100 shown.
[0448] In addition, the method of changing the appearance of the imaging device 200 in this example does not physically change the appearance of the imaging device 200, but changes the appearance of the imaging device 200 observed by the virtual image display device 100 (smart glasses) in the image.
[0449] exist Fig.37 In the process, the XR processor 110 acquires the visual field image IM photographed by the built-in camera 101 (step S350), and extracts the camera image of the imaging device 200 present in the visual field image IM (step S352).
[0450] The XR processor 110 determines whether the imaging device 200 is an XR-compatible camera from the extracted camera image (step S354 ). This determination can be performed by, for example, comparing the extracted camera image with the appearance data of the XR-compatible camera stored in advance in the first memory 120 .
[0451] In step S354, if it is determined that the camera 200 photographed by the built-in camera 101 is an XR-compatible camera (in the case of "yes"), the XR processor 110 obtains the XR-compatible data of the camera 200 (step S356). The XR-compatible data includes data indicating the appearance of the interchangeable lens unit, leather exterior unit, operation dial unit, upper exterior unit, viewfinder unit, monitor unit, etc. of the camera 200.
[0452] On the other hand, in step S354, if it is determined that the camera device 200 photographed by the built-in camera 101 is not an XR-compatible camera ("No" case), the XR processor 110 sets the superimposition synthesis function of the live preview image in the virtual image display device 100 (smart glasses) to OFF (step S358), and returns to step S350. As a result, the user can only see the real world (real camera device 200) through the smart glasses.
[0453] Next, the XR processor 110 determines the multiple areas after the camera image is divided as display areas based on the camera image extracted in step S352 and the XR corresponding data obtained in step S356 (step S360). That is, the areas such as the interchangeable lens unit, the leather exterior unit, the operation dial unit, the upper exterior unit, the viewfinder unit, and the monitor unit corresponding to the XR corresponding data are determined as display areas.
[0454] Next, the XR processor 110 obtains a component image synthesized with one or more display areas of the plurality of display areas in the camera 200 according to the user's selection operation (step S362). In the first memory 120 of the virtual image display device 100, a component image (virtual object image) representing the appearance is stored for each of the plurality of divided areas into which the appearance of the camera 200 is divided, and the user can display the plurality of component images stored in the first memory 120 on the main monitor 241 and select a component image synthesized with the display area by the operation unit 230. In addition, the component image can have an image attached to the surface of the three-dimensional model of the component.
[0455] The XR processor 110 obtains transformation information used when superimposing the three-dimensional component image on the two-dimensional display area corresponding to the component image (step S364). The three-dimensional data of the component image is known, and since the two-dimensional data of the display area can also be detected, it is possible to obtain transformation information for projecting the three-dimensional component image in a manner consistent with the corresponding display area.
[0456] The XR processor 110 performs geometric transformation on the component image according to the transformation information acquired in step S364 , and outputs the geometrically transformed component image to the XR display 140 (step S366 ).
[0457] The XR display 140 inputs the geometrically transformed component image and displays it on the display element 144, so that the component image is superimposed and displayed on the display area of the camera 200 (step S368). The user can visually recognize the component image (the skin with a changed appearance) superimposed on the real world (the display area of the camera 200) through the optical system 146 of the XR display 140. That is, the user can set it to the original camera with the appearance that he or she likes. In addition, one camera can be used separately with multiple appearances.
[0458] Furthermore, it is of course possible to display the live preview image LV1 and the like superimposed on the display area of the monitor section of the imaging device while changing the appearance of the imaging device.
[0459] [other]
[0460] The camera device 200 of this embodiment can perform Selfie photography when the XR mode is set. In this case, the live preview image LV1 can be displayed on the display area set on the front surface of the camera device 200, or on the display area A on the sub-monitor 242 side of the monitor panel 240 rotated 180 degrees. In addition, the user wears smart glasses or VR goggles, but it is preferred to synthesize the facial image of the avatar on the user's facial area.
[0461] Furthermore, the camera device 200 has, in a non-XR environment, a display mode in which the photographic information INF0 is synthesized with the instant preview image LV and displayed on the main monitor 241, a display mode in which only the instant preview image LV is displayed on the main monitor 241, a display mode in which only the photographic information INF0 is displayed, and a function of switching each time the display / return button 247 is pressed. However, when the virtual image display device 100 uses the camera device 200 in an XR environment, it is preferred to display the instant preview image LV1, etc. according to the display mode set in the non-XR environment.
[0462] Preferably, the XR processor 110 of the virtual image display device 100 changes the brightness of the synthesized instant preview image LV1 according to the brightness information of the field of view image IM taken in by the built-in camera 101, so that the display area A of the camera device 200 becomes brighter when it is bright and becomes darker when it is dark, and preferably controls the brightness of the instant preview image LV1 within a range that does not exceed a preset threshold so as not to cause whitish or black brightness when whitish or black is generated.
[0463] In addition, in the present embodiment, for example, the hardware structure of a processing unit (processing unit) such as a CPU (Central Processing Unit) that performs various processes is various processors (processors) as shown below. The various processors include a general-purpose processor that executes software (programs) to function as various processing units, namely a CPU, a processor whose circuit structure can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), a processor having a circuit structure specially designed to perform specific processing, such as a programmable logic device (PLD), and an ASIC (Application Specific Integrated Circuit), such as a dedicated circuit.
[0464] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). In addition, multiple processing units can be composed of one processor. As an example of multiple processing units composed of one processor, first, as represented by computers such as clients and servers, there is a method of forming a processor composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Second, as represented by a system on a chip (System On Chip: SoC), there is a method of using a processor that uses an IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units. In this way, various processing units are composed of more than one of the above-mentioned various processors as a hardware structure.
[0465] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements.
[0466] Furthermore, the present invention includes a plurality of the above-described embodiments, and as long as the imaging device or system of the present invention can be applied, modifications can be made so as to selectively or concurrently act.
[0467] Furthermore, the "first display" of the virtual image display device is a glasses-type or goggle-type wearable device with a virtual image display function, which uses an HMD device or a transmissive smart glasses to provide images including virtual images, but other methods (for example, a retinal projection device built into smart glasses, etc., a method of displaying a virtual image according to the projection mapping principle by using a projector as a display method of the first display, etc.) can be any display unit as long as it can be applied to the photographic device or system of the present invention and can realize the essence of the present invention, namely, display.
[0468] Furthermore, in the present embodiment, the virtual image display device 100 performs geometric transformation on the real-time preview image LV and outputs the real-time preview image LV1 after the geometric transformation to the XR display 140, but the virtual image display device 100 may also obtain the real-time preview image LV1 after the geometric transformation from the XR server 300 as described above, or may obtain the real-time preview image LV1 after the geometric transformation from the camera device 200. In this case, in order to perform the process of geometrically transforming the real-time preview image LV, the camera device 200 needs to obtain the field of view image IM from the virtual image display device 100, or obtain the information of the relative three-dimensional position and posture of the virtual image display device 100 and the camera device 200.
[0469] Furthermore, the virtual image display device 100 can overlap and display the real-time preview image LV1 on the display area of the camera device 100 on the XR display 140 when the camera device 200 is used in an XR environment (when the XR mode is set), but can also set the overlap synthesis function of the real-time preview image LV1 in the virtual image display device 100 to ON when the camera device 200 is used in a non-XR environment (normal environment) (when the non-XR mode is set). Thus, when the camera device 200 is used in a normal environment, there is an advantage that the real-time preview image LV1 can be well observed even when the display of the camera device 200 is difficult to see due to the irradiation of direct light.
[0470] Moreover, the following structure can be configured: even if the situation where the technology is used is set to "used in an XR environment", a mode can be set in which the live preview image LV is not automatically sent to the virtual image display device 100, so that the live preview image can be selectively displayed on the display of the camera device 100.
[0471] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made without departing from the spirit of the present invention.
[0472] Explanation of symbols
[0473] 10, 11-virtual image display system, 100-virtual image display device, 101-built-in camera, 102-photographic lens, 104-image sensor, 110, 111-XR processor, 112-transformation information acquisition unit, 114-synthesis unit, 116-geometric transformation unit, 120-first memory, 130, 230-operation unit, 140, 141-XR display, 142-display control unit, 143-image synthesis unit, 144, 145-display element, 146, 147-optical system, 146A-lens, 146 B-half-mirror, 150-virtual object generating unit, 151-interchangeable lens unit, 152-leather exterior unit, 153-operating dial unit, 154-upper exterior unit, 155-viewfinder unit, 156-monitor unit, 160-first communication unit, 170, 270-GPS receiver, 172, 272-gyro sensor, 174, 274-acceleration sensor, 176, 276-geomagnetic sensor, 180-virtual image display device, 200, 280-camera device, 201-camera unit, 202-interchangeable lens, 203-camera Camera body, 204-image sensor, 210-camera processor, 220-second memory, 232-shutter button, 234-sensitivity dial, 236-exposure correction dial, 240-monitor panel, 241-main display, 242-sub-display, 243-viewfinder, 244-trigger switch, 245-execution button, 246-playback button, 247-button, 248-display control unit, 250-image processing unit, 260-second communication unit, 281-simulation lens, 282-actual lens, 283-simulation monitor, 2 90, 290B, 290C, 292-virtual camera, 291-smartphone, 300-XR server, 302-database, 310-unmanned camera, 320-multi-view camera, 330-surveillance camera, 340-driving recorder, A-display area, IM-field of view image, INFO-photographic information, LV, LV1-instant preview image, M, M1~M9-AR markers, P1~P4-feature points, P1'~P4'-corresponding points, S100~S122, S200~S284, S300~S368-steps.
Claims
1. A virtual image display device comprising a processor and a first display, The virtual image display device receives the instant preview image from the first camera device, The processor performs the following processing: acquiring information used when causing the live preview image to overlap with a display area of the first camera device; and transforming the instant preview image according to the information, The first display displays the transformed live preview image.
2. The virtual image display device according to claim 1, wherein: The first display is a glasses-type or goggle-type wearable device having a virtual image display function.
3. The virtual image display device according to claim 1 or 2, comprising a second camera for photographing a user's field of view. The processor analyzes the photographic image taken by the second camera device to determine the display area and acquire the information.
4. The virtual image display device according to claim 3, wherein: The processor acquires the information based on the position of the mark of the first imaging device photographed by the second imaging device.
5. The virtual image display device according to claim 1 or 2, wherein: The processor acquires information indicating a three-dimensional position and posture of the first imaging device with respect to the first display, and acquires the information based on the information indicating the three-dimensional position and posture.
6. The virtual image display device according to any one of claims 1 to 5, wherein: The first camera device has a second display for displaying the live preview image. The display area is the display area of the second display.
7. The virtual image display device according to claim 6, wherein: The second display has a rotation mechanism capable of changing a relative posture relationship with a device body of the first camera device by rotation. The display area is the display area of the second display after rotation.
8. The virtual image display device according to any one of claims 1 to 5, wherein: The first camera device has a viewfinder. The display area is the display area of the viewfinder.
9. The virtual image display device according to any one of claims 1 to 8, wherein: The first display is a display that transmits the visual field and displays the live preview image transformed into the visual field of the real world.
10. The virtual image display device according to any one of claims 1 to 8, comprising a second camera for photographing a user's field of view. The first display is an immersive display covering a field of view, and displays a composite image in which the live preview image after the transformation is superimposed on the photographic image photographed by the second camera device.
11. The virtual image display device according to any one of claims 1 to 10, wherein: The processor synthesizes the virtual object image into the instant preview image.
12. A virtual image display device comprising a processor, a first display, and a second camera device for photographing a user's field of view, The processor performs the following processing: Cutting out an image corresponding to the field angle of the virtual camera having at least a handle from the image photographed by the second camera device and acquiring the image as a live preview image; Acquire information used when causing the instant preview image to overlap with a display area of the virtual camera; transforming the instant preview image according to the information; and The first display displays the transformed live preview image.
13. The virtual image display device according to any one of claims 1 to 12, wherein: The transformation is a geometric transformation.
14. A camera device comprising: Camera department, photography instant preview image; A mode switching unit for switching between the first mode and the second mode; a second display or viewfinder, which displays the live preview image when switched to the first mode; and The communication unit transmits the live preview image to the virtual image display device according to any one of claims 1 to 11 when switching to the second mode.
15. The imaging device according to claim 14, wherein: When switching to the second mode, a mark is displayed on the second display or the viewfinder. 16 . A virtual image display system, comprising the virtual image display device according to claim 1 and the imaging device according to claim 14 or 15 .
17. The virtual image display system according to claim 16, wherein: The processor of the virtual image display device performs the following processing: Synthesizing a virtual object image into the instant preview image; and When recording photography is performed in the state switched to the second mode, the virtual object image is sent to the camera device, When the camera device performs photography for recording in the state switched to the second mode, the photographed image is recorded in association with the virtual object image received from the virtual image display device, or the virtual object image is synthesized with the photographed image and recorded.
18. A virtual image display system, comprising the virtual image display device according to any one of claims 1 to 11 and a server communicating with the virtual image display device, The server manages a three-dimensional image having three-dimensional information of a real space, The processor performs the following processing: cutting out a three-dimensional image corresponding to a photographing area of the first camera from the three-dimensional image managed by the server based on first information indicating a three-dimensional position and posture of the first camera; and The cut-out three-dimensional image is converted into a two-dimensional image, and the two-dimensional image is used as the live preview image.
19. A virtual image display system, comprising the virtual image display device according to claim 12 and a server communicating with the virtual image display device, The server manages a three-dimensional image having three-dimensional information of a real space, The processor performs the following processing: cutting out a three-dimensional image corresponding to a photographing area of the virtual camera from the three-dimensional image managed by the server based on first information indicating the three-dimensional position and posture of the virtual camera; and The cut-out three-dimensional image is converted into a two-dimensional image, and the two-dimensional image is used as the live preview image.
20. The virtual image display system according to claim 18 or 19, wherein: The server generates and manages the latest three-dimensional image based on images taken by multiple cameras including a multi-viewpoint camera, a monitoring camera, a driving recorder, an unmanned camera or a mobile terminal camera arranged in a real space.
21. The virtual image display system according to any one of claims 18 to 20, wherein: The server synthesizes a virtual object image into the three-dimensional image.
22. A virtual image display method, which is performed by a virtual image display device having a processor and a first display, The virtual image display method comprises the following steps: The processor receives a real-time preview image from the first camera device; The processor acquires information used when causing the live preview image to overlap with a display area of the first camera device; The processor transforms the instant preview image according to the information; and The first display displays the transformed live preview image.
Citation Information
Patent Citations
Program executed by computer providing virtual space, method, and information processing device executing the program
JP2019021236A
