Virtual image display device, imaging device, virtual image display system, and method

Abstract

Provided are a virtual image display device, imaging device, virtual image display system, and method for superimposing a virtual image in the real world and displaying an image that cannot be displayed by a single imaging device. A virtual image display device (100) comprising a processor and a first display receives a live view image from an imaging device (200). The processor acquires transformation information to be used when superimposing the live view image on a display region of the imaging device (200), and geometrically transforms the live view image on the basis of the transformation information. The first display displays the geometrically-transformed live view image. A user can visually recognize, through an optical system (146), the virtual live view image superimposed on the display region of the imaging device (200).

Description

Virtual image display device, imaging device, virtual image display system and method

[0001] The present invention relates to a virtual image display device, an imaging device, a virtual image display system and a method, and more particularly to a technique for displaying a virtual image superimposed on the real world.

[0002] Conventionally, a method has been proposed in which a user can take a photograph in a virtual space and easily check a photograph (image) generated by the photograph (Patent Document 1).

[0003] The method described in Patent Literature 1 places a virtual camera at the center of a virtual space provided using a head-mounted device (HMD), generates an image corresponding to the virtual camera's shooting range from the position and orientation of the virtual camera, and displays the generated image on a preview screen of the virtual camera.When a user takes a photo in the virtual space in response to a shooting instruction, a photographic image is generated and saved in memory, and the generated photographic image is placed at a predetermined position in the virtual space, and the position where the photographic image is placed is notified to the user wearing the HMD.

[0004] The virtual camera described in Patent Document 1 is a camera object generated based on object information stored in a memory module and placed in a virtual space. When a button on the camera object is pressed by a hand object corresponding to the hand of an avatar object, the camera object captures an image in the virtual space. The hand object corresponding to the hand of the avatar object moves in conjunction with the position of the user's hand. While Patent Document 1 uses an HMD device to provide the image captured by the virtual camera, other methods, such as see-through smart glasses or retinal projection, are also possible.

[0005] Furthermore, Patent Document 2 proposes an information processing device that effectively utilizes augmented reality (AR) technology, which combines the real world and the virtual world, for games and the like.

[0006] The information processing device described in Patent Document 2 acquires a captured image from a rear camera that captures real space, and when it detects a marker included in the captured image, it starts information processing (a game) corresponding to the marker, and places a virtual object such as a character or icon corresponding to the marker in a three-dimensional coordinate system corresponding to the captured space, draws it on the captured image, and displays it on a display device on the surface. The icon is expressed as if it were placed on the same plane as the marker, and can be operated by either touching a touchpad on the display screen or by pointing to a corresponding position in the captured space with a finger.

[0007] JP 2019-21236 A JP 2015-41126 A

[0008] One embodiment of the technology disclosed herein provides a virtual image display device, an imaging device, a virtual image display system, and a method for overlaying a virtual image on the real world and displaying an image that cannot be displayed by a conventional imaging device alone.

[0009] A first aspect of the invention is a virtual image display device having a processor and a first display, wherein the virtual image display device receives a live view image from a first imaging device, the processor acquires information used to overlay the live view image on a display area of ​​the first imaging device, converts the live view image based on the information, and the first display displays the converted live view image.

[0010] In the virtual image display device according to the second aspect of the present invention, the first display is preferably a glasses-type or goggle-type wearable device having a virtual image display function.

[0011] In a virtual image display device according to a third aspect of the present invention, a second imaging device is provided for capturing an image of the user's field of view, and a processor analyzes the captured image captured by the second imaging device to identify the display area and acquire information.

[0012] In the virtual image display device according to the fourth aspect of the present invention, the processor acquires information based on the position of the marker of the first imaging device photographed by the second imaging device.

[0013] In a virtual image display device according to a fifth aspect of the present invention, a processor acquires information indicating the three-dimensional position and orientation of a first imaging device relative to a first display, and acquires information based on the information indicating the three-dimensional position and orientation.

[0014] In the virtual image display device according to the sixth aspect of the present invention, it is preferable that the first imaging device has a second display that displays a live view image, and the display area is the display area of ​​the second display.

[0015] In the virtual image display device according to the seventh aspect of the present invention, it is preferable that the second display is provided with a rotation mechanism that can change the relative attitude of the first imaging device with respect to the device body by rotation, and that the display area is the display area of ​​the rotated second display.

[0016] In the virtual image display device according to the eighth aspect of the present invention, it is preferable that the first imaging device has a viewfinder, and the display area is the display area of ​​the viewfinder.

[0017] In the virtual image display device according to the ninth aspect of the present invention, it is preferable that the first display is a display that transmits the field of view and displays a live view image converted into a field of view of the real world.

[0018] In the virtual image display device according to the tenth aspect of the present invention, it is preferable that the device is provided with a second imaging device that captures the user's field of view, and the first display is an immersive display that covers the field of view, and displays a composite image in which the converted live view image is superimposed on the captured image captured by the second imaging device.

[0019] In the virtual image display device according to the eleventh aspect of the present invention, it is preferable that the processor composites the virtual object image with the live view image.

[0020] A twelfth aspect of the present invention is a virtual image display device having a processor, a first display, and a second imaging device that captures an image of a user's field of view, wherein the processor cuts out an image corresponding to at least the angle of view of a virtual camera having a grip portion from an image captured by the second imaging device, acquires the image as a live view image, acquires information to be used when overlaying the live view image on the display area of ​​the virtual camera, and converts the live view image based on the information, and the first display displays the converted live view image.

[0021] In the virtual image display device according to the thirteenth aspect of the present invention, the transformation is preferably a geometric transformation.

[0022] The invention of a fourteenth aspect is an imaging device comprising an imaging unit that captures a live view image, a mode switching unit that switches between a first mode and a second mode, a second display or viewfinder that displays the live view image when switched to the first mode, and a communication unit that transmits the live view image to a virtual image display device of any of the first to eleventh aspects when switched to the second mode.

[0023] In the imaging device according to the fifteenth aspect of the present invention, when the mode is switched to the second mode, it is preferable that a marker be displayed on the second display or the viewfinder.

[0024] A sixteenth aspect of the invention is a virtual image display system comprising the virtual image display device of any one of the first to thirteenth aspects and the imaging device of the fourteenth or fifteenth aspect.

[0025] In the virtual image display system according to the seventeenth aspect of the present invention, the processor of the virtual image display device preferably combines a virtual object image with a live view image, and when a recording shot is taken with the mode switched to the second mode, transmits the virtual object image to the imaging device, and when a recording shot is taken with the mode switched to the second mode, the imaging device preferably links the captured image with the virtual object image received from the virtual image display device and records it, or combines the virtual object image with the captured image and records it.

[0026] An 18th aspect of the invention is a virtual image display system comprising a virtual image display device according to any one of the first to 11th aspects and a server that communicates with the virtual image display device, wherein the server manages three-dimensional images having three-dimensional information of real space, and the processor cuts out a three-dimensional image corresponding to the shooting area of ​​the first imaging device from the three-dimensional images managed by the server based on first information indicating the three-dimensional position and attitude of the first imaging device, converts the cut-out three-dimensional image into a two-dimensional image, and uses the two-dimensional image as a live view image.

[0027] A nineteenth aspect of the invention is a virtual image display system comprising the virtual image display device of the twelfth aspect and a server that communicates with the virtual image display device, in which the server manages three-dimensional images having three-dimensional information of real space, and the processor cuts out a three-dimensional image corresponding to the shooting area of ​​the virtual camera from the three-dimensional images managed by the server based on first information indicating the three-dimensional position and attitude of the virtual camera, converts the cut-out three-dimensional image into a two-dimensional image, and uses the two-dimensional image as a live view image.

[0028] In the virtual image display system according to the twentieth aspect of the present invention, it is preferable that the server generates and manages the latest three-dimensional images from images taken by a number of cameras, including multi-viewpoint cameras, surveillance cameras, drive recorders, drone cameras, or mobile device cameras, arranged in real space.

[0029] In the virtual image display system according to the twenty-first aspect of the present invention, it is preferable that the server composites the virtual object image with the three-dimensional image.

[0030] The invention according to a 22nd aspect is a virtual image display method executed by a virtual image display device having a processor and a first display, the virtual image display method including the steps of: the processor receiving a live view image from a first imaging device; the processor acquiring information to be used when overlaying the live view image on the display area of ​​the first imaging device; the processor converting the live view image based on the information; and the first display displaying the converted live view image.

[0031] FIG. 1 is a diagram showing the system configuration of a virtual image display system according to the present invention. FIG. 2 is a front perspective view of the imaging device shown in FIG. 1. FIG. 3 is a diagram showing three open / closed states of a monitor panel of the imaging device. FIG. 4 is a diagram showing how a field of view image including the imaging device is captured by a second imaging device of the virtual image display device. FIG. 5 is a diagram showing how a live view image is superimposed on a specific display area of ​​the imaging device. FIG. 6 is a block diagram showing an embodiment of a virtual image display device constituting the virtual image display system according to the present invention. FIG. 7 is a block diagram showing an embodiment of the imaging device according to the present invention. FIG. 8 is a block diagram of essential parts showing a first embodiment of the virtual image display system according to the present invention. FIG. 9 is a diagram showing the geometric relationship between a live view image, a field of view image, and the display area of ​​the imaging device. FIG. 10 is a diagram showing the appearance of an imaging device provided with an AR marker. FIG. 11 is a diagram showing a first embodiment of an XR display of the virtual image display device. FIG. 12 is a diagram showing a first display example of a live view image, etc. when the imaging device is set to XR mode. FIG. 13 is a diagram showing a second display example of a live view image, etc. when the imaging device is set to XR mode. FIG. 14 is a diagram showing a third display example of a live view image or the like when the imaging device is set to XR mode. FIG. 15 is a diagram showing a fourth display example of a live view image or the like when the imaging device is set to XR mode. FIG. 16 is a diagram showing a fifth display example of a live view image or the like when the imaging device is set to XR mode. FIG. 17 is a diagram showing a fifth display example of a live view image or the like when the imaging device is set to XR mode. FIG. 18 is a diagram showing a sixth display example of a live view image or the like when the imaging device is set to XR mode. FIG. 19 is a diagram showing a first external appearance example of an imaging device when the imaging device is set to XR mode. FIG. 20 is a diagram showing a second external appearance example of an imaging device when the imaging device is set to XR mode. FIG. 21 is a diagram showing a third external appearance example of an imaging device when the imaging device is set to XR mode. FIG. 22 is a conceptual diagram of a case where the external appearance of an imaging device is changed by dividing it into multiple regions. FIG. 23 is a perspective view of an imaging device adapted for use in an XR environment.FIG. 24 is a rear perspective view of an embodiment of a virtual camera constituting a virtual image display system according to the present invention. FIG. 25 is a front perspective view of an embodiment of a virtual camera constituting a virtual image display system according to the present invention. FIG. 26 is a perspective view of another embodiment of a virtual camera constituting a virtual image display system according to the present invention. FIG. 27 is a diagram illustrating an embodiment of a method for acquiring a live view image captured by a virtual camera. FIG. 28 is a diagram illustrating a second embodiment of an XR display of a virtual image display device. FIG. 29 is a block diagram illustrating a second embodiment of a virtual image display system according to the present invention. FIG. 30 is a flowchart illustrating a first embodiment of a virtual image display method according to the present invention. FIG. 31 is a flowchart illustrating a display method using an imaging device according to the present invention. FIG. 32 is a flowchart illustrating a second embodiment of a virtual image display method according to the present invention. FIG. 33 is a subroutine illustrating an embodiment of the processing of step S300 shown in FIG. 32 for confirming the imaging device. FIG. 34 is a subroutine illustrating an embodiment of the processing of step S3200 for superimposing and displaying the live view image LV1 shown in FIG. 32. FIG. 35 is a subroutine illustrating an embodiment of the processing of step S330 for superimposing and displaying the shooting information INFO shown in FIG. 32. Fig. 36 is a subroutine showing an embodiment of the processing of step S340 for superimposing and displaying the reproduced images shown in Fig. 32. Fig. 37 is a flowchart showing an embodiment of a method for changing the appearance of an imaging device.

[0032] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a virtual image display device, an imaging device, a virtual image display system, and a method thereof according to the present invention will now be described with reference to the accompanying drawings.

[0033] [Outline of Virtual Image Display System] FIG. 1 is a diagram showing the system configuration of a virtual image display system according to the present invention.

[0034] 1 is composed of a virtual image display device 100 and a first imaging device (hereinafter simply referred to as "imaging device") 200, and the virtual image display device 100 and the imaging device 200 communicate with each other. The communication is preferably short-range wireless communication according to standards such as Bluetooth (registered trademark) and Wi-Fi (Wireless Fidelity) (registered trademark), but wired communication is also acceptable.

[0035] The virtual image display device 100 of this embodiment is a glasses-type wearable device having a virtual image display function, and includes a processor and a first display (not shown), and a second imaging device (hereinafter referred to as a "built-in camera").

[0036] The first display is a display for realizing augmented reality (AR), virtual reality (VR), mixed reality (MR), or cross reality (XR), which is a general term for AR, VR, and MR.

[0037] AR is a technology that "virtually augments" the real world in front of the user's eyes by overlaying a virtual world (virtual images, etc.) on the real world seen through glasses, while VR displays images captured by a camera on VR goggles or a VR headset, blocking out the real world through the VR goggles, etc., allowing the user to become deeply immersed in a non-real world. MR is a technology that displays virtual objects and information overlaid on the real world, and is similar to AR but is a further development of AR, allowing digital information to be displayed in accordance with the user's position and movement, and for the user to directly touch and manipulate the digital information.

[0038] The first display (hereinafter referred to as "XR display") of the virtual image display device 100 shown in FIG. 1 has an AR or MR display function that superimposes a virtual world onto the real world seen through glasses (eyeglasses).

[0039] 1, reference numeral 146 denotes an optical system including a half mirror of the virtual image display device 100 (hereinafter also referred to as "smart glasses"), and an image (virtual image) displayed on the display element of the smart glasses is superimposed on the real world via the optical system 146. Details of the smart glasses will be described later.

[0040] FIG. 2 is a perspective view of the imaging device shown in FIG. 1 as seen from the front.

[0041] 1 and 2 is a mirrorless digital single-lens camera that includes an interchangeable lens 202 and a camera body (device body) 203 to which the interchangeable lens 202 can be attached or detached. Note that the imaging device 200 is not limited to a mirrorless digital single-lens camera, and may also be a digital single-lens reflex camera, a digital camera with an integrated lens that does not allow the lens to be replaced, or a smartphone or PDA (Personal Digital Assistant) that has a camera function.

[0042] In FIG. 1, a shutter button 232, a shutter speed / sensitivity dial 234, an exposure compensation dial 236, and the like are provided on the top surface of the camera body 203.

[0043] A monitor panel 240 is disposed on the rear surface, which is one wall surface of the camera body 203, so as to be rotatable via a rotation mechanism (not shown).

[0044] FIG. 3 shows three open and closed states of the monitor panel of the imaging device.

[0045] As shown in FIG. 3, the monitor panel 240 is a tilt-type panel whose relative position with respect to the camera body 203 can be changed by rotation.

[0046] FIG. 3A shows the imaging device 200 with the monitor panel 240 closed, and FIGS. 3B and 3C show the imaging device 200 with the monitor panel 240 rotated 90° and 180°.

[0047] A main monitor (second display) 241 is disposed on the inner surface (rear surface) of the monitor panel 240 which is hidden when the monitor panel 240 is closed, and a sub-monitor 242 is disposed on the surface (front surface) which is substantially flush with the exterior of the camera when the monitor panel 240 is closed.

[0048] The main monitor 241 is operable when the monitor panel 240 is open as shown in Figures 3(B) and (C), and displays live view images, recorded images (playback images), etc., while the sub-monitor 242 is operable when the monitor panel 240 is closed as shown in Figures 1 and 3(A), and displays shutter speed, aperture value, ISO sensitivity, and other information related to shooting (shooting information).

[0049] 1, an eyepiece of a viewfinder 243 is disposed on the rear surface of the camera body 203. This viewfinder 243 is a hybrid viewfinder that can be switched between an electronic viewfinder (EVF) and an optical viewfinder (OVF).

[0050] Furthermore, when using the OVF display, you can choose to display or hide the Electronic Range Finder (ERF) (a small window that displays the EVF) as part of the OVF. The ERF can be displayed in three different ways: 100% field of view, 2.5x magnification, or 6x magnification, and you can check the focus, angle of view, exposure, and white balance when shooting with the OVF.

[0051] As shown in FIG. 2, a viewfinder switching lever 249 for switching the viewfinder 243 between the EVF and OVF is provided on the front surface of the camera.

[0052] 2, the viewfinder can be switched between EVF and OVF by rotating the viewfinder switch lever 249 clockwise. Also, when the viewfinder is in OVF display, the viewfinder can be switched between displaying and hiding the ERF by rotating the viewfinder switch lever 249 counterclockwise.

[0053] Furthermore, on the rear surface of the camera body 203, there are provided a toggle switch 244 for indicating multiple directions such as up, down, left, and right, a menu / execute button 245, a play button 246, a display / back button 247, and the like.

[0054] Furthermore, a plurality of function buttons are provided to which various functions can be assigned.

[0055] [Outline of Virtual Image Display System] FIG. 4 is a diagram showing how a field of view image including the imaging device is captured by the second imaging device of the virtual image display device.

[0056] 4, the built-in camera 101 of the virtual image display device 100 captures a visual field image corresponding to the visual field of the user including the imaging device 200. The built-in camera 101 captures a visual field image corresponding to the visual field of the user when the user wears smart glasses and has their eyes facing forward.

[0057] The user holds the imaging device 200 in his / her hand and performs various operations such as framing.

[0058] When the imaging device 200 is being used in an XR environment, the imaging device 200 transmits live view images generated by operating the imaging device 200 to the virtual image display device 100 as continuous data.

[0059] The processor of the virtual image display device 100 identifies the display area of ​​the imaging device 200 based on the image (camera image) of the imaging device 200 present in the field of view image. This display area is not limited to a display area where an image can actually be displayed, such as the main monitor 241 or sub-monitor 242 of the imaging device 200 shown in Fig. 3, but may also be, for example, an area set in advance on the back surface, which is one wall surface of the camera body 203.

[0060] The user can view the imaging device 200 in the real world through the smart glasses, but the processor transforms (geometrically transforms) the live view image received from the imaging device 200 and displays it on the smart glasses, thereby superimposing the live view image (virtual image) on a specific display area of ​​the real imaging device 200 via the optical system 146 of the smart glasses.

[0061] FIG. 5 is a diagram showing how a live view image is superimposed on a specific display area of ​​the imaging device.

[0062] A live view image LV1 that has been geometrically transformed via the optical system 146 of the smart glasses is displayed at a display position in front of the eyes of a user wearing the smart glasses. This allows the user to view the real imaging device 200 through the smart glasses as shown in Fig. 5 , and to observe the live view image LV1 as if an image of an XR environment that does not actually exist, synthesized (pasted) in a specific display area of ​​the imaging device 200, is being displayed on the imaging device 200.

[0063] The imaging device 200 shown in Figures 1 and 3 does not have a real monitor that displays a live view image LV1 of the size shown in Figure 5, but the virtual image display device 100 displays the live view image LV1, allowing the user to visually recognize the imaging device 200 as if it were equipped with a display that displays the live view image LV1 shown in Figure 5.

[0064] According to this virtual image display system 10, the user does not look at a specific display area (the real world) on the back of the imaging device 200, but rather observes a live view image LV1 synthesized into the specific display area using smart glasses. Therefore, even if the specific real display area is difficult to see due to direct sunlight, the user can clearly observe the live view image LV1.

[0065] Furthermore, because the imaging device 200 is a real-world camera product, it is easy to operate for creative photography and can maintain a high quality photography experience. Furthermore, even when the position or angle of the imaging device 200 is changed for framing or other purposes, the live view image LV1 can be displayed on the smart glasses while tracking the imaging device 200, making it possible to avoid missing a photo opportunity even in fast-moving photography that requires quick operation of the imaging device 200 (such as sports photography, panning, or photography of children or pets).

[0066] In addition, the field of view image captured by the built-in camera 101 of the smart glasses is used to obtain information (conversion information) used to determine a specific display area (coordinates) of the camera image in the field of view image and to superimpose the live view image on a specific display area of ​​the imaging device 200, and if an immersive display (VR goggles) that covers the field of view (visual field) is used instead of smart glasses, it is used as the field of view image to be displayed on the VR goggles.

[0067] <Embodiment of Virtual Image Display Device> FIG. 6 is a block diagram showing an embodiment of a virtual image display device constituting a virtual image display system according to the present invention.

[0068] FIG. 6 is a block diagram showing the internal configuration of the virtual image display device 100 having the external appearance shown in FIG.

[0069] The virtual image display device 100 includes a built-in camera 101, a processor (XR processor) 110, a first memory 120, an operation unit 130, an XR display 140 which is a first display having a virtual image display function, a display control unit 142, a virtual object generation unit 150, a first communication unit 160, a GPS (Global Positioning System) receiver 170, a gyro sensor 172, an acceleration sensor 174, and a geomagnetic sensor 176.

[0070] The built-in camera 101 includes a photographing lens 102 , an image sensor 104 , and an AFE (Analog Front End) 106 .

[0071] When the imaging device 200 is held in the hand and used to take a photograph, the built-in camera 101 captures a field of view image including the imaging device 200 (camera image), and it is preferable that the photographing lens 102 has a field of view angle equal to or greater than that of a human being.

[0072] The image sensor 104 can be configured as a CMOS (Complementary Metal-Oxide Semiconductor) color image sensor. Note that the image sensor 104 is not limited to a CMOS type, and may also be a CCD (Charge Coupled Device) type image sensor.

[0073] An optical image of a subject formed on the light receiving surface of the image sensor 104 by the photographic lens 102 is converted into an electrical signal by the image sensor 104. Charges corresponding to the amount of incident light are accumulated in each pixel of the image sensor 104, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 104 as an image signal.

[0074] The AFE 106 performs various types of analog signal processing on the analog image signal output from the image sensor 104. The AFE 106 includes a correlated double sampling circuit, an AGC (Automatic Gain Control) circuit, and an analog-to-digital conversion circuit (A / D conversion circuit) (all not shown). The correlated double sampling circuit performs correlated double sampling on the analog signal from the image sensor 104 to remove noise caused by resetting the signal charge. The AGC circuit amplifies the analog signal from which noise has been removed by the correlated double sampling circuit, so that the signal level of the analog signal falls within an appropriate range. The A / D conversion circuit converts the image signal, whose gain has been adjusted by the AGC circuit, into a digital signal.

[0075] When the image sensor 104 is a CMOS image sensor, the AFE 106 is often built into the image sensor 104 .

[0076] The XR processor 110 is composed of a CPU (Central Processing Unit) and the like, and controls each unit in an integrated manner, and also performs various processes such as acquiring information (transformation information) used when superimposing a live view image received from the imaging device 200 via the first communication unit 160 on a specific display area of ​​the imaging device 200, and converting (geometrically transforming) the live view image based on that information and outputting it to the XR display 140. Details of the processes performed by the XR processor 110 will be described later.

[0077] The first memory 120 includes a flash memory, a read-only memory (ROM), a random access memory (RAM), etc. The flash memory and the ROM are non-volatile memories that store various programs including firmware, parameters, images of various virtual objects including avatars (virtual object images), etc.

[0078] The RAM functions as a working area for processing by the XR processor 110, and also temporarily stores firmware and the like stored in the non-volatile memory. Note that the XR processor 110 may incorporate a part of the first memory 120 (RAM).

[0079] The operation unit 130 is a part that issues various operation commands, sets parameters, selects virtual objects including avatars, controls avatars, etc., and may include a voice input device equipped with a microphone that inputs operations by voice, a gesture input unit that reads body movements such as those of the hands and fingers from images captured by the built-in camera 101 and uses them as operation inputs, a touch panel provided on the frame of the smart glasses, a GUI (Graphical User Interface) screen displayed on the XR display 140, and a dedicated controller that moves a cursor on the GUI screen, etc.

[0080] In addition, the gyro sensor 272, acceleration sensor 274, geomagnetic sensor 276, etc. of the imaging device 200 can detect shaking, vibration, or light angle of the imaging device 200, and the detected information can be used as various operation commands, etc.

[0081] Since both hands are used during the shooting operation of the imaging device 200, a voice input device is preferable.

[0082] When the display control unit 142 receives a live view image, a playback image, shooting information, etc. from the XR processor 110, it converts them into a signal format for display and outputs them to the XR display 140.

[0083] The XR display 140 has a display element (not shown) and displays a live view image or the like on the display element driven by a display signal from the display control unit 142. The live view image displayed on the display element is superimposed on the real world through the glasses via the optical system 146 of the smart glasses (see FIG. 1).

[0084] The XR display 140 displays a live view image or the like that has been geometrically transformed so as to be superimposed on a specific display area of ​​the imaging device 200 at a display position in front of the user, and the user can observe the live view image or the like superimposed on the real world.

[0085] The virtual object generation unit 150 reads out a desired virtual object image designated by the user from various virtual object images, including avatars, stored in the first memory 120, and generates a virtual object image processed for compositing with a live view image. For example, when an avatar representing the user is selected and a hairstyle or outfit is also selected, the virtual object generation unit 150 generates an avatar wearing the selected hairstyle or outfit, and when the position, orientation, etc. of the avatar are designated by the operation unit 130, the virtual object generation unit 150 performs processing to move the avatar to that position, etc.

[0086] Furthermore, the virtual object generation unit 150 receives an instruction to select a virtual object indicating a part (exterior part, switches, etc.) of the imaging device 200, and generates a virtual object image by processing an image of the selected part (part image) for compositing with the imaging device 200. By compositing the part image with the imaging device 200, it is possible to change the appearance of the imaging device 200 observed through the XR display 140.

[0087] The first communication unit 160 is a unit that transmits and receives necessary information to and from the imaging device 200, and when the imaging device 200 is capturing a live view image in an XR environment, the first communication unit 160 receives the live view image from the imaging device 200. Furthermore, when the shutter button 232 (see FIG. 1 ) is pressed in the imaging device 200 in an XR environment to capture an image for recording, the first communication unit 160 receives a shutter release signal. If an avatar image is being superimposed on the live view image when the shutter release signal is received, the first communication unit 160 transmits the avatar image to the imaging device 200.

[0088] The GPS receiver 170, the gyro sensor 172, the acceleration sensor 174, and the geomagnetic sensor 176 are sensors for detecting information indicating the three-dimensional position and orientation of the XR display 140 (built-in camera 101). The information indicating the three-dimensional position and orientation of the XR display 140 is used to acquire information indicating the three-dimensional position and orientation of the imaging device 200 relative to the XR display 140. The information indicating the three-dimensional position and orientation of the imaging device 200 relative to the XR display 140 can be used to calculate transformation information for geometrically transforming the live view image.

[0089] <Embodiment of Imaging Apparatus> FIG. 7 is a block diagram showing an embodiment of an imaging apparatus according to the present invention.

[0090] FIG. 7 is a block diagram showing the internal configuration of the imaging device 200 that constitutes the virtual image display system 10 shown in FIG.

[0091] The imaging device 200 includes an imaging unit 201, a camera processor 210, a second memory 220, an operation unit 230, a display control unit 248, a main monitor 241, a sub-monitor 242, a viewfinder 243, an image processing unit 250, a second communication unit (communication unit) 260, a GPS receiver 270, a gyro sensor 272, an acceleration sensor 274, and a geomagnetic sensor 276, etc.

[0092] The imaging unit 201 includes an interchangeable lens 202 , an image sensor 204 , and an AFE 206 .

[0093] The interchangeable lens 202 is attached to the camera body 203 and can perform two-way communication with the camera body 203 via multiple signal terminals provided on the lens mount of the camera body 203. For example, in response to a request signal from the camera body 203, the interchangeable lens 202 transmits lens information (information such as focal length, focus position, aperture value, etc.) of the interchangeable lens 202 to the camera body 203, and also drives the focus lens, aperture, etc. in accordance with commands from the camera body 203.

[0094] The image sensor 204 is configured as a CMOS color image sensor, but is not limited to the CMOS type and may be a CCD type image sensor.

[0095] The AFE 206 performs various types of analog signal processing on the analog image signal output from the image sensor 204, converts it into a digital image signal, and outputs it as RAW data.

[0096] The camera processor 210 is composed of a CPU and the like, and controls each section in an integrated manner, and also controls each section in an integrated manner in accordance with user operations using the operation section 230, and performs various processes including AF (Auto Focus) control and AE (Automatic Exposure) control.

[0097] It goes without saying that AF control and AE control are performed automatically when the auto mode is set by the operation unit 230, and that AF control and AE control are not performed when the manual mode is set.

[0098] As shown in FIG. 1, the operation unit 230 includes a shutter button 232, a shutter speed / sensitivity dial 234, an exposure compensation dial 236, as well as a toggle switch 244, a menu / execute button 245, a play button 246, a display / back button 247, etc., and various instructions and settings can be made by operating these buttons, dials, etc.

[0099] The operation unit 230 of the imaging device 200 of this example differs from the operation units of conventional imaging devices in that it has a mode switching unit that switches between a first mode (non-XR mode) in which the imaging device 200 is used in a non-XR environment (normal environment) and a second mode (XR mode) in which the imaging device 200 is used in an XR environment.

[0100] The mode switching unit for XR mode / non-XR mode can use a function button to which setting / cancelling of XR mode is assigned, but a dedicated mode switching lever may also be provided. Also, a toggle switch 244 and a menu / execute button 245 may be used to set / cancel XR mode on the menu screen.

[0101] As a dedicated mode switching lever, for example, a mode switching lever having a configuration similar to the viewfinder switching lever 249 shown in FIG. 2 can be provided on the front, back or top of the camera body 203.

[0102] The operation of the imaging device 200 when the XR mode is set and the device is used in an XR environment will be described in detail later.

[0103] The second memory 220 includes a flash memory, a ROM, a RAM, etc. The flash memory and the ROM are non-volatile memories that store various programs including firmware, parameters, captured images (still images, video images), and the like.

[0104] The RAM functions as a working area for processing by the camera processor 210, and also temporarily stores firmware and the like stored in the non-volatile memory. Note that the camera processor 210 may incorporate part of the second memory 220 (RAM).

[0105] When the display control unit 248 inputs live view images, playback images, shooting information, etc. from the camera processor 210 or the image processing unit 250, it converts them into a signal format for display and outputs them to the main monitor 241, the sub-monitor 242, and the viewfinder 243 (EVF).

[0106] The main monitor 241 is operable when the monitor panel 240 is open as shown in Figures 3(B) and (C), and displays live view images, recorded images (playback images), etc. in accordance with instructions from the camera processor 210 in the non-XR mode.

[0107] The sub-monitor 242 is operable when the monitor panel 240 is closed as shown in FIG. 3A, and displays shooting information indicating the shutter speed, aperture value, ISO sensitivity, etc. in accordance with instructions from the camera processor 210 in the non-XR mode.

[0108] The viewfinder 243 can be switched between EVF and OVF by operating a viewfinder switching lever, and when switched to EVF and in non-XR mode, displays a live view image in accordance with instructions from the camera processor 210.

[0109] Therefore, when the monitor panel 240 is closed and in the non-XR mode, the user can frame the image while checking the shooting range through the viewfinder 243.

[0110] Even if the monitor panel 240 is open in non-XR mode, when you bring your eye close to the viewfinder 243, an eye sensor (not shown) automatically switches the display to the viewfinder 243, and when you move your eye away, the display switches to the main monitor 241.

[0111] On the other hand, in XR mode, the main monitor 241 and the sub-monitor 242 do not display the live view image and shooting information, respectively, but can display a marker (e.g., an AR marker) that specifies the display area of ​​the main monitor 241 or the sub-monitor 242 in accordance with instructions from the camera processor 210, for example.

[0112] The image processing unit 250 performs RAW development processing such as offset processing, white balance correction, gain control processing including sensitivity correction, gamma correction processing, and demosaic processing (demosaicing processing) on ​​the RAW data output from the imaging unit 201. Furthermore, for RAW data acquired upon pressing the shutter button 232, the image processing unit 250 performs RAW development processing followed by compression processing, and records the data as an image file in the second memory 220 or a memory card (not shown).

[0113] In addition, the continuous RAW data output from the imaging unit 201 is subjected to RAW development processing by the image processing unit 250, and the continuous data is output as a live view image LV to the main monitor 241 or the second communication unit 260 via the display control unit 248.

[0114] The second communication unit 260 is a part that transmits and receives necessary information to and from the virtual image display device 100, and when the imaging device 200 is set to the XR mode and is capturing a live view image in an XR environment, the second communication unit 260 transmits the live view image LV to the virtual image display device 100.

[0115] Furthermore, when the shutter button 232 (see FIG. 1 ) of the imaging device 200 is pressed in an XR environment and an image is captured for recording, the second communication unit 260 transmits a shutter release signal to the virtual image display device 100. The virtual image display device 100 receives the shutter release signal via the first communication unit 160, and if an avatar image is being composited with the live view image at the time of reception, the first communication unit 160 of the virtual image display device 100 transmits the avatar image to the imaging device 200.

[0116] The GPS receiver 270, the gyro sensor 272, the acceleration sensor 274, and the geomagnetic sensor 276 are sensors for detecting information indicating the three-dimensional position and orientation of the imaging device 200. The information indicating the three-dimensional position and orientation of the imaging device 200 is used to acquire information indicating the three-dimensional position and orientation of the imaging device 200 relative to the XR display 140. The information indicating the three-dimensional position and orientation of the imaging device 200 relative to the XR display 140 can be used to calculate transformation information for geometrically transforming the live view image LV.

[0117] [First Embodiment of Virtual Image Display System] FIG. 8 is a block diagram showing a main part of a first embodiment of a virtual image display system according to the present invention, and is a functional block diagram showing the functions of the XR processor 110 of the virtual image display device 100 shown in FIG. 6.

[0118] In FIG. 8, the XR processor 110 functions as a transformation information acquisition unit 112, a synthesis unit 114, and a geometric transformation unit 116.

[0119] The built-in camera 101 of the smart glasses captures a visual field image IM corresponding to the user's visual field including the imaging device 200, and outputs the captured visual field image IM to the conversion information acquisition unit 112.

[0120] When the imaging device 200 is being used in an XR environment, the imaging device 200 transmits a live view image LV generated by operating the imaging device 200 as continuous data, and the first communication unit 160 receives the live view image LV from the imaging device 200.

[0121] The conversion information acquisition unit 112 is a part that acquires conversion information used when superimposing the live view image LV on the display area of ​​the imaging device 200, and inputs the captured image (field of view image IM) from the built-in camera 101.

[0122] The conversion information acquisition unit 112 analyzes the input visual field image IM to acquire area information indicating the display area of ​​the imaging device 200, and acquires conversion information based on the area information.

[0123] FIG. 9 is a diagram showing the geometric relationship between a live view image, a field of view image, and a display area of ​​an imaging device.

[0124] 9, the image coordinate system of the live view image LV is an XY coordinate system, and the image coordinate system of the field of view image IM is an X'Y' coordinate system. The vertical and horizontal image sizes of the live view image LV are h1 and w1, and the vertical and horizontal image sizes of the field of view image IM are h2 and w2. The coordinates of the four corner points (feature points) P1 to P4 of the live view image LV in the XY coordinate system are known based on the image size of the live view image LV.

[0125] As shown in FIG. 9, the field of view image IM includes a camera image showing the imaging device 200 .

[0126] The conversion information acquisition unit 112 acquires, for example, the coordinates of the four corner points of the display area (corresponding points P1' to P4' corresponding to feature points P1 to P4) as area information indicating a specific display area of ​​the imaging device 200 in the field of view image IM.

[0127] The conversion information acquisition unit 112 of the first embodiment analyzes the field of view image IM to detect the outline of a rectangle on the monitor panel 240 (see FIG. 1 ), and sets the four corner points of that outline as corresponding points P1' to P4' corresponding to the feature points P1 to P4 at the four corners of the live view image LV. In this example, the area surrounded by the outline of the rectangle on the closed monitor panel 240 is set as the display area, but the display area can be defined as appropriate.

[0128] Next, the conversion information acquisition unit 112 acquires conversion information to be used when superimposing the live view image LV on the display area on the camera image of the field of view image IM (the area of ​​the monitor panel 240 identified by the corresponding points P1' to P4').

[0129] Now, when a live view image LV is geometrically transformed by affine transformation and superimposed on a display area on a camera image, it is necessary to obtain parameters (transformation information) of the affine transformation.

[0130] The affine transformation can be performed by the following formula.

[0131]

[0132] In [Equation 1], (x, y) indicate the coordinates of the live view image LV (coordinates in the XY coordinate system), and (x', y') indicate the coordinates after affine transformation (coordinates in the X'Y' coordinate system of the field of view image IM).

[0133] In addition, in [Equation 1], parameters a to d are parameters that indicate the enlargement / reduction, rotation, and skew (shear) of the image, and parameter t x , t y is a parameter indicating translation.

[0134] In the affine transformation, three or more sets of matching points (feature points and corresponding points) of two images (in this example, the live view image LV and the visual field image IM) must be given to estimate the parameters. However, four sets of matching points are obtained: the feature points P1 to P4 of the live view image LV and the corresponding points P1' to P4' of the visual field image IM. Therefore, the parameters a to d and t of the affine transformation shown in [Equation 1] are x , t y can be obtained.

[0135] In this way, the transformation information acquisition unit 112 obtains the affine transformation parameters a to d and t x , t y can be obtained as conversion information.

[0136] Another method for performing geometric transformation between two images is, for example, projective transformation.

[0137] The projective transformation can be performed by the following formula:

[0138]

[0139] In the projective transformation, nine parameters (h 11 ~h 33 ) and therefore more matching points between two images must be found than in the affine transformation, but it is possible to perform a geometric transformation with higher accuracy than the affine transformation.

[0140] 8 , the live view image LV is added to the composition unit 114 via the first communication unit 160. The composition unit 114 is a part that combines a virtual object with the live view image LV, and an image of a virtual object generated by the virtual object generation unit 150 is added to another input of the composition unit 114.

[0141] The virtual object generation unit 150 reads an image of a desired virtual object (for example, an avatar) from the first memory 120 in response to a user instruction, and generates and outputs an image of the avatar that has been processed for synthesis with the live view image LV.

[0142] A user can register an avatar image representing the user in the first memory 120. The avatar image representing the user may be a photograph of the user, or an image of an anime character, a robot, a pet, or the like.

[0143] The user can also select a hairstyle or outfit as appropriate by operating the operation unit 130, and the virtual object generation unit 150 can generate an image of an avatar wearing the hairstyle or outfit selected by the user. Furthermore, when the position, orientation, etc. of the avatar is specified by the user, the virtual object generation unit 150 performs processing to move the image of the avatar to the specified position, etc., like an avatar in a role-playing game.

[0144] Furthermore, when generating a more realistic avatar image, the virtual object generation unit 150 may adjust the brightness and color of the avatar image to match the brightness and color obtained from the live view image LV, etc.

[0145] The synthesis unit 114 synthesizes an image of a virtual object including an avatar generated by the virtual object generation unit 150 with the live view image LV added via the first communication unit 160, and outputs the synthesized live view image LV to the geometric transformation unit 116.

[0146] If the virtual object generation unit 150 does not generate a virtual object, the synthesis unit 114 outputs the input live view image LV as is to the geometric transformation unit 116 as a live view image LV.

[0147] The geometric transformation unit 116 is a part that performs geometric transformation on the live view image LV added via the synthesis unit 114, based on the transformation information acquired by the transformation information acquisition unit 112. For example, when the geometric transformation by the geometric transformation unit 116 is an affine transformation, the transformation information acquisition unit 112 acquires the parameters (a to d, t x , t y ) as transformation information, and the geometric transformation unit 116 performs affine transformation on the pixel at coordinates (x, y) of the live view image LV to transform it into a pixel at coordinates (x', y') of the visual field image IM based on the transformation information. In addition, when the geometric transformation performed by the geometric transformation unit 116 is a projective transformation, the parameter (h 11 ~h 33 ) is acquired as conversion information, and the geometric transformation unit 116 converts the pixel at coordinates (x, y) of the live view image LV into the pixel at coordinates (x', y') of the field of view image IM by performing projective transformation based on the conversion information.

[0148] The live view image LV that has been geometrically transformed by the geometric transformation unit 116 (hereinafter referred to as the “live view image LV1”) is output to the XR display 140.

[0149] The XR display 140 displays the geometrically transformed live view image LV1 at the corresponding pixel position on the XR display 140 (the pixel position corresponding to the field of view image IM of the built-in camera 101). In the example shown in Fig. 9 , the live view image LV represented by four coordinates P1 to P4 in the XY coordinate system of the live view image LV is coordinate-transformed into the X'Y' coordinate system of the field of view image IM and displayed as the live view image LV1 represented by four coordinates P1' to P4'.

[0150] In addition, since there is parallax between the user's field of view when wearing smart glasses and the field of view image IM captured by the built-in camera 101, it is preferable that the XR display 140 corrects the parallax according to the size (corresponding to the distance) of the camera image in the field of view image IM and the position of the camera image.

[0151] <Second embodiment of conversion information acquisition unit> The conversion information acquisition unit 112 of the first embodiment shown in FIG. 8 analyzes the field of view image IM to detect the rectangular outline of the monitor panel 240, sets the rectangular area surrounded by the four corner points of the outline (P1′ to P4′ in FIG. 9) as the display area A of the image capture device 200, and acquires the conversion information, but is not limited to this, and the conversion information acquisition unit 112 may acquire the conversion information based on the position of a marker (hereinafter referred to as an “AR marker”) of the image capture device 200 captured by the built-in camera 101.

[0152] FIG. 10 is a diagram showing the appearance of an imaging device provided with an AR marker.

[0153] The imaging device 200 shown in FIG. 10 is provided with nine AR markers M1 to M9.

[0154] The AR markers are markers that represent pre-registered illustrations, images, etc., and can be linked to position information. Therefore, by reading the AR markers M1 to M9 from the field of view image IM, the conversion information acquisition unit 112 can acquire position information on the positions where the AR markers M1 to M9 are displayed, and can acquire information such as the tilt, position, size, and center coordinates of each surface of the imaging device 200.

[0155] Of the AR markers M1 to M9, the AR marker M1 is an AR marker displayed on the sub-monitor 242, and the AR markers M2 to M9 are AR markers provided in advance on the camera body 203.

[0156] AR markers M2 to M4 are AR markers that indicate the positions of three of the four corners of the monitor panel 240, and when the area of ​​the monitor panel 240 is the display area A of the live view image LV1, they are AR markers M that identify the display area A of the live view image LV1 of the imaging device 200.

[0157] When the imaging device 200 is used in an XR environment, the AR marker M1 can be displayed as a marker that identifies the display area A of the live-view image LV1. When the AR marker M is displayed on the sub-monitor 242, the number of AR markers M is not limited to one, and multiple AR markers M can be displayed. Furthermore, when the main monitor 241 (see FIGS. 3B and 3C) is used as the display area for the live-view image LV1, multiple AR markers M that identify the area of ​​the main monitor 241 can be displayed.

[0158] AR marker M5 is a marker provided on the top surface of camera body 203, AR marker M6 is a marker provided on the upper back surface of camera body 203, and AR marker M7 is a marker provided on the side surface of camera body 203. These AR markers M5 to M7 are markers provided to identify each surface of camera body 203.

[0159] Furthermore, the AR marker M8 is a marker provided on the upper surface of the shutter speed / sensitivity dial 234, and the AR marker M9 is a marker provided on the upper surface of the exposure compensation dial 236. These AR markers M8 and M9 are markers provided to identify the position of the dial (including the rotation position).

[0160] The AR markers M4 to M9 can be used to specify an area on the camera body 203 where a virtual object (a skin that changes the appearance) is to be attached.

[0161] Instead of the AR markers M1 to M9, the markers may be configured as two-dimensional barcodes (for example, "QR Code" (registered trademark)). When a two-dimensional barcode is displayed, it may be a two-dimensional barcode that encodes information about what the image capture device 200 is displaying, or data or indicators (numerical information such as shutter speed, aperture value, histogram, etc.) that are displayed in the live view image, and the two-dimensional barcode may also include information incorporating information about the layout of the data display.

[0162] In addition, instead of AR markers M1 to M9, markers having other shapes suitable for identifying a position may be used, and AR markers M may be mixed with two-dimensional barcodes, and the type, position, and number of markers are not limited to the embodiment shown in Figure 10.

[0163] <Third embodiment of conversion information acquisition unit> The conversion information acquisition unit that acquires conversion information used when superimposing the live view image LV on the display area A of the image capture device 200 is not limited to cases where the field of view image IM captured by the built-in camera 101 is used. It can acquire the conversion information by acquiring information indicating the three-dimensional position and orientation of the image capture device 200 and information indicating the three-dimensional position and orientation of the XR display 140, and acquiring information indicating the relative three-dimensional position and orientation of the image capture device 200 with respect to the XR display 140 (first display).

[0164] The conversion information acquisition unit can acquire information indicating the three-dimensional position and attitude of the XR display 140 based on the sensor outputs of the GPS receiver 170, gyro sensor 172, acceleration sensor 174, and geomagnetic sensor 176 of the virtual image display device 100, and similarly can acquire information indicating the three-dimensional position and attitude of the XR display 140 based on the sensor outputs of the GPS receiver 270, gyro sensor 272, acceleration sensor 274, and geomagnetic sensor 276 of the imaging device 200.

[0165] Here, assuming that the virtual image display device 100 (XR display 140) is represented in a global coordinate system and the imaging device 200 is represented in a camera coordinate system (local coordinate system), the size and shape of the imaging device 200 are known, and therefore the three-dimensional positions (for example, ) of each part of the imaging device 200 in the local coordinate system can be set in advance.

[0166] On the other hand, the global coordinate system of the XR display 140 can be defined as an XYZ three-axis Cartesian coordinate system, for example, with the center of the display element of the XR display 140 as the origin, the X-axis and Y-axis being directions in the mutually orthogonal planes of the display element that pass through the origin of the display element, and the Z-axis being a normal direction that passes through the origin of the display element.

[0167] As described above, by acquiring information indicating the three-dimensional position and orientation of the XR display 140 and information indicating the three-dimensional position and orientation of the imaging device 200 (i.e., acquiring information indicating the three-dimensional position and orientation of the imaging device 200 relative to the XR display 140), it is possible to acquire transformation information to be used when superimposing the live view image LV on the display area A of the imaging device 200 based on this information.

[0168] For example, the three-dimensional position in the local coordinate system of display area A of image capture device 200 (i.e., the three-dimensional position of live view image LV) is converted into coordinates in the global coordinate system, which is the coordinate system of XR display 140, based on information indicating the three-dimensional position and orientation of image capture device 200 relative to XR display 140. Next, the three-dimensional position of display area A of image capture device 200 (i.e., the three-dimensional position of live view image LV) that has been coordinate-converted into the global coordinate system is perspectively projected onto the display surface of the display element of XR display 140 based on known parameters such as the size of the display element of XR display 140 and the focal length of the optical system, thereby obtaining a converted live view image LV1.

[0169] Note that the information indicating the relative three-dimensional position and attitude of the imaging device 200 with respect to the XR display 140 is not limited to being obtained based on the sensor outputs of the GPS receiver 170, gyro sensor 172, acceleration sensor 174, and geomagnetic sensor 176 of the virtual image display device 100, and the sensor outputs of the GPS receiver 270, gyro sensor 272, acceleration sensor 274, and geomagnetic sensor 276 of the imaging device 200, but may also be obtained from measurement data measured by an optical sensing measurement device (for example, LiDAR (Light Detection And Ranging)) provided in the virtual image display device 100 (smart glasses), a distance image acquired by a TOF (Time Of Flight) camera, or stereo images acquired by built-in cameras provided on the left and right sides of the smart glasses.

[0170] <First Embodiment of XR Display> FIG. 11 is a diagram showing a first embodiment of an XR display of a virtual image display device.

[0171] The XR display 140 shown in FIG. 11 is a display that transmits the visual field, and is a glasses-type wearable device (smart glasses) that displays a live view image LV1 that has been geometrically transformed into the visual field of the real world.

[0172] The XR display 140 includes a display element 144 and an optical system 146 including a lens 146A and a half mirror 146B. The live view image LV1 after geometric transformation displayed on the display element 144 is displayed as a virtual image by the optical system 146 at a display position in front of the user's eyes.

[0173] The user can view the real world passing through the half mirror 146B, and can also view a live view image LV1 reflected by the half mirror 146B and superimposed on the real world. Note that the live view image LV1 is an image superimposed on the display area A of the image capture device 200 in the real world, and is a virtual image because it is not a reflected light beam from the image capture device 200.

[0174] Furthermore, the display element 144 shown in FIG. 11 is a surface-emitting element such as an organic electroluminescence (EL) panel or a liquid crystal panel, but is not limited to this. It may also be a display element composed of a DMD (Digital Micromirror Device) in which a large number of movable micromirrors are arranged in a planar manner on an integrated circuit, and a light source that emits red, green, and blue light onto the DMD.

[0175] <First Display Example in XR Mode> FIG. 12 is a diagram showing a first display example of a live view image or the like when the imaging device is set to the XR mode.

[0176] Fig. 12A is a diagram showing an image of the imaging device 200 viewed with the naked eye. In Fig. 12A, the monitor panel 240 is closed, and shooting information is displayed on the sub-monitor 242. When the XR mode is set and the user is not wearing the virtual image display device 100 (smart glasses), the user cannot observe the live view image LV1.

[0177] Fig. 12(B) is a diagram showing an image when the imaging device 200 is viewed while wearing the virtual image display device 100. The first display example shown in Fig. 12(B) shows a case where a live view image LV1 is displayed superimposed on the area of ​​the closed monitor panel 240 of the imaging device 200.

[0178] In this case, when the user wears the virtual image display device 100 and looks at the imaging device 200, it appears as if the live view image LV1 is being displayed in the area of ​​the monitor panel 240.

[0179] It is also possible to change the image to be combined with the real world from the live view image LV1 by changing the settings, and to change to a different display format.

[0180] <Second Display Example in XR Mode> FIG. 13 is a diagram showing a second display example of a live view image or the like when the imaging device is set to the XR mode.

[0181] The second display example shown in Fig. 13 shows a case where a live view image LV1 is displayed superimposed on the area of ​​the main monitor 241 of the imaging device 200. Fig. 13 also shows the imaging device 200 with the monitor panel 240 rotated 90 degrees. In this case, the main monitor 241 inside the monitor panel 240 can be seen from above, which is suitable for low-angle shooting.

[0182] When the imaging device 200 is set to XR mode, a live view image is not displayed on the main monitor 241, and by wearing the virtual image display device 100, the user sees as if a live view image were being displayed in the area of ​​the main monitor 241.

[0183] According to the second display example shown in FIG. 13, the live view image LV1 that is generated is subject to restrictions according to the movable range of the main monitor 241 of the imaging device 200, but because the main monitor 241 cannot rotate freely, the orientation of the display of the live view image LV1 is not lost, and since the operation method is the same as that of the actual imaging device 200, there is an advantage that the operability is good and there is no confusion.

[0184] Alternatively, a dummy monitor panel can be displayed in place of the actual monitor panel 240, and the live view image LV1 can be displayed on the dummy monitor panel. In this case, the movable range of the dummy monitor panel can be made larger (for example, by +10 degrees) than the movable range of the actual monitor panel 240. Furthermore, it is preferable to selectively switch the movable range of the dummy monitor panel (freely, stepwise, etc.), and to change the color or shape of the monitor panel frame (e.g., between a real camera-like and a transparent frame) so that the change is noticeable.

[0185] <Third Display Example in XR Mode> FIG. 14 is a diagram showing a third display example of a live view image or the like when the imaging device is set to the XR mode.

[0186] The third display example shown in Figure 14 is a modified example of the first display example shown in Figure 12 (B), and shows a case in which a live view image LV1 is superimposed and displayed on a dummy monitor panel that is larger than the closed monitor panel 240 of the imaging device 200.

[0187] The size of the live view image LV1 that is displayed by geometric deformation and superimposition can be freely set, and in the third display example shown in Figure 14, by wearing the virtual image display device 100, the user sees the live view image LV1 displayed on a dummy monitor panel of a non-existent size that is larger than the size of the monitor panel 240.

[0188] The size of the dummy monitor panel is not limited to a fixed size, but may be made resizable in response to an enlargement operation such as pulling on a corner of the monitor panel 240. In this case, it is preferable to provide the monitor panel 240 with a tactile member that allows the user to feel the sensation of actually pulling on the image frame.

[0189] In this case, a method for detecting an enlargement operation such as pulling a corner of the monitor panel 240 may be to use the principle of gesture control through image recognition by the built-in camera 101 of the virtual image display device 100 to enlarge or reduce the size of the dummy monitor panel in accordance with the enlargement or reduction operation, or other methods may be used, such as detecting a pinch operation on a touch panel provided on the monitor panel 240 of the imaging device 200 and linking it to that.

[0190] Furthermore, when a corner of the monitor panel 240 is pulled, the aspect ratio of the dummy monitor panel remains fixed, but the dummy monitor panel may expand in the direction of pulling and upward, or may expand equally in the direction of pulling and upward and downward, or may expand downward.

[0191] <Fourth Display Example in XR Mode> FIG. 15 is a diagram showing a fourth display example of a live view image or the like when the imaging device is set to the XR mode.

[0192] The fourth display example shown in FIG. 15 shows a case where a live view image LV1 is displayed superimposed on a non-existent area of ​​a monitor panel.

[0193] The monitor panel 240 has a rotation mechanism that allows it to tilt relative to the camera body 203 as shown in Figures 3(B) and (C), but does not have a rotation mechanism that rotates the live view image LV1 as shown in Figure 15.

[0194] 15, a virtual rotation mechanism is provided on the right side of the rear surface of the camera body 203, and the monitor panel is rotated as needed. As a result, the live view image LV1 appears to be approximately the same size as the monitor panel 240, with one end of the image being fixed to the imaging device 200, and is displayed so as to face directly in the line of sight, as if the monitor panel had been tilted and was facing the user.

[0195] 15 shows the monitor panel tilted horizontally around one of the left and right sides as an axis, but this is just one example, and it may be tilted vertically around one of the top and bottom sides as a fixed axis, or an undefined display combining tilts in two different directions may be used. In this case, the monitor panel does not appear to be fixed to the imaging device 200 at any of its sides, and may be expressed as a virtual image as if it were connected by a hinge mechanism.

[0196] <Fifth Display Example in XR Mode> FIG. 16 is a diagram showing a fifth display example of a live view image or the like when the imaging device is set to the XR mode.

[0197] The fifth display example shown in Fig. 16 is a modified example of the fourth display example shown in Fig. 15, and shows the case where the orientation of the camera body 203 is reversed. That is, in the fifth display example shown in Fig. 16, a virtual rotation mechanism is provided on the left side of the rear of the camera body 203, and the monitor panel is rotated as appropriate. As a result, the live view image LV1 is displayed at approximately the same size as the monitor panel 240 and appears to face directly in the line of sight.

[0198] In this way, it is preferable to provide a virtual two-axis or three-axis hinge that rotates the monitor panel, and to automatically rotate the monitor panel relative to the camera body so that the live view image LV1 is always displayed facing forward. This provides an operational feel similar to that of a real camera, eliminating confusion and preventing an unnatural composite image in which the live view image LV1 appears to be floating in the air, thereby improving the quality of the photographic act by allowing for more realistic photographing.

[0199] FIG. 17 is a diagram showing a modification of the fifth display example of the live view image etc. shown in FIG.

[0200] Fig. 17(A) is a diagram showing the same display state as the fifth display example of the live view image etc. shown in Fig. 16. As shown in Fig. 17(A), the imaging device 200 is shown in its entirety within the field of view of the XR display 140.

[0201] FIG. 17B illustrates a case where the imaging device 200 illustrated in FIG. 17A has moved to the left with respect to the field of view of the XR display 140, causing the left half of the imaging device 200 to fall outside the field of view.

[0202] Even in this case, virtual image display device 100 can receive live view image LV from imaging device 200, and can therefore display live view image LV1 on XR display 140 as shown in Fig. 17(B) . That is, virtual image display device 100 moves live view image LV1 to the right of the original display position in Fig. 17(B) relative to imaging device 200, and displays the entire live view image LV1.

[0203] 17(C) shows a case where the imaging device 200 shown in FIG. 17(B) is moved further to the left, and the left two-thirds of the imaging device 200 is outside the field of view of the XR display 140. In this case, the virtual image display device 100 moves the live view image LV1 for the imaging device 200 further to the right in FIG. 17(C) from the display position shown in FIG. 17(B), and displays the entire live view image LV1.

[0204] FIG. 17(D) illustrates a case where the imaging device 200 shown in FIG. 17(C) has moved further to the left, and the entire imaging device 200 has moved out of the field of view of the XR display 140. In this case, the virtual image display device 100 moves the live view image LV1 for the imaging device 200 further to the right in FIG. 17(D) from the display position shown in FIG. 17(C), and displays the entire live view image LV1. In this case, the left edge of the live view image LV1 coincides with the left edge of the field of view.

[0205] The display example shown in Figure 17 shows a case where the imaging device 200 moves to the left and out of the field of view, but this is not limited to this. In cases where the imaging device 200 moves to the right and out of the field of view, moves upward and out of the field of view, or moves downward and out of the field of view, the live view image LV1 can be moved in the same manner as above, and the entire live view image LV1 can be displayed.

[0206] In addition, in the case of other display examples including the fourth display example such as the live view image shown in Figure 15, if the imaging device 200 goes out of the field of view, the live view image LV1 can be moved relative to the imaging device 200 in the same manner as described above.

[0207] As a result, even if the image capturing device 200 moves out of the field of view, the live view image LV1 continues to be displayed around the position where the image capturing device 200 moves out of the field of view, allowing the user to continue viewing the live view.

[0208] In addition, in the first to fifth display examples shown in Figures 12 to 17, it is possible to display shooting information at the bottom end of the live view image LV1 together with the compositely displayed live view image LV1.

[0209] <Sixth Display Example in XR Mode> FIG. 18 is a diagram showing a sixth display example of a live view image or the like when the imaging device is set to the XR mode.

[0210] The sixth display example shown in FIG. 18 shows a case where shooting information INFO is displayed in an overlapping manner in an area larger than the sub-monitor 242 (see FIG. 1) of the closed monitor panel 240 of the imaging device 200 .

[0211] This shooting information INFO includes a histogram in addition to the normal shooting information (see FIG. 12A) displayed on the sub-monitor 242. This histogram is a brightness histogram that shows the distribution of brightness in the live view image currently being captured. The histogram is not limited to a brightness histogram, and may be an RGB histogram for each of the colors R (red), G (green), and B (blue).

[0212] Furthermore, the shooting information INFO is not limited to information indicating shooting conditions such as shutter speed, aperture value, EV value (exposure value), etc., but also includes various types of shooting information. For example, the shooting information INFO can include metadata indicating subject information, ISO sensitivity, detection area information such as face recognition frames and AF frames, etc.

[0213] The virtual image display device 100 can receive text data or an image indicating the shooting information INFO from the imaging device 200. Furthermore, when the shooting information INFO is displayed on the sub-monitor 242, the virtual image display device 100 can acquire the shooting information INFO by reading it using image recognition.

[0214] The photographing information INFO preferably includes camera identification information for distinguishing it from data of other image capture devices, but when the photographing information INFO is displayed, the camera identification information is not displayed.

[0215] On the other hand, when the virtual image display device 100 combines and displays the shooting information INFO, it is preferable that the live view image LV be superimposed on the area of ​​the viewfinder 243 and displayed.

[0216] As a result, when a user wearing the virtual image display device 100 brings their eye close to the viewfinder 243 , they can view a live view image LV through the eyepiece of the viewfinder 243 .

[0217] <First Appearance Example of Imaging Device> FIG. 19 is a diagram showing a first appearance example of the imaging device when the imaging device is set to the XR mode.

[0218] As shown in Figure 10, the imaging device 200 is provided with AR markers M1 to M9, and the virtual image display device 100 can identify each surface of the imaging device 200 or a polygon indicating the external shape of the imaging device 200 by detecting the positions of the AR markers M1 to M9 from the field of view image IM.

[0219] The virtual image display device 100 can display an appearance that is different from the appearance of the original imaging device 200 by pasting and synthesizing a virtual object image (skin) that shows the appearance of the imaging device 200 in accordance with the AR markers M1 to M9.

[0220] In the above example, a method of pasting and synthesizing a virtual object image (skin) representing the appearance of the imaging device 200 has been described as a method of creating an appearance different from the actual appearance of the imaging device 200. However, the virtual object image (skin) does not have to be pasted flatly onto the surface to which it is pasted. For example, like the lens 200B attached to the front surface of the imaging device 200 in Fig. 19, the virtual object image (skin) may have a shape that makes it appear as if a separate component is protruding or that a separate component is attached.

[0221] Furthermore, a plurality of virtual object images (skins) may be associated with one AR marker.

[0222] The lens 200B may be configured to be detachable. Furthermore, an operation of holding and removing the lens 200B (for example, a gesture that imitates lens replacement, such as rotating and pulling away) may be detected by image recognition, and the lens 200B may be attached or detached in conjunction with that operation.

[0223] In addition, in the first external appearance example of the imaging device 200 shown in FIG. 19, the monitor on the back of the camera body 203 is enlarged and the switches on the back are changed.

[0224] <Second Appearance Example of Imaging Device> FIG. 20 is a diagram showing a second appearance example of the imaging device when the imaging device is set to the XR mode.

[0225] As with the first appearance example shown in FIG. 19, the virtual image display device 100 changes the appearance of the imaging device 200 by pasting and synthesizing a virtual object image (skin) that represents the appearance of the imaging device 200.

[0226] In the second external appearance example of the imaging device 200 shown in Fig. 20, the top surface of the camera body 203 has been changed to a simple shape with only a shutter button 232. In the second external appearance example shown in Fig. 20, the shutter speed / sensitivity dial 234 and exposure compensation dial 236 shown in Fig. 1 etc. are not visible.

[0227] When the imaging device 200 is set to the XR mode and some of the operating systems of the imaging device 200 become invisible due to a change in appearance, it is preferable that the imaging device 200 disables the operating systems that become invisible (such as the actual shutter speed / sensitivity dial 234 and exposure compensation dial 236).

[0228] <Third Appearance Example of Imaging Device> FIG. 21 is a diagram showing a third appearance example of the imaging device when the imaging device is set to the XR mode.

[0229] As with the first appearance example shown in FIG. 19, the virtual image display device 100 changes the appearance of the imaging device 200 by pasting and synthesizing a virtual object image (skin) that represents the appearance of the imaging device 200.

[0230] In the third external appearance example of the imaging device 200 shown in FIG. 21, the position of the viewfinder has been changed from the left end to the center, and the monitor and operation system on the back of the camera body 203 have been removed.

[0231] <Third Appearance Example of Imaging Device> FIG. 22 is a conceptual diagram showing a case where the appearance of an imaging device is changed by dividing it into a plurality of regions.

[0232] The first memory 120 of the virtual image display device 100 stores virtual object images that show the appearance of the imaging device 200 for each of the divided areas obtained by dividing the appearance of the imaging device 200 into a plurality of areas.

[0233] In the example shown in FIG. 0, an interchangeable lens unit 151, a leather exterior unit 152, an operation dial unit 153, an upper exterior unit 154, a viewfinder unit 155, and a monitor unit 156 are stored in the first memory 120 as virtual object images.

[0234] The interchangeable lens unit 151 includes a plurality of interchangeable lens units of different types, such as wide-angle lenses and telephoto lenses. The leather exterior unit 152 includes black leather, brown leather, red leather, crocodile pattern leather, etc. The upper exterior unit 154 includes exterior units with textures and colors, such as silver, black, gold, and two-tone. The viewfinder unit 155 and monitor unit 156 also include those of various shapes and sizes.

[0235] The user can create an original appearance for the imaging device 200 by selecting and combining virtual object images showing the appearance of each of these divided areas through operations on the operation unit 230 .

[0236] That is, the XR processor 110 receives one or more virtual object images showing appearances through user operation on the operation unit 230, appropriately performs geometric transformation on these virtual object images based on the three-dimensional position and orientation of the imaging device 200, and outputs the images to the XR display 140. This allows the user to recognize the imaging device viewed through the XR display 140 as having the appearance of the virtual object image selected by the user.

[0237] Furthermore, when combining multiple types of virtual object images, multiple effective combinations may be prepared in advance, and the user may select from the multiple combinations. In this case, the XR processor 110 receives virtual object images selected from the multiple combinations, appropriately performs geometric transformation on these virtual object images based on the three-dimensional position and orientation of the imaging device 200, and outputs the results to the XR display 140.

[0238] Furthermore, effective combinations of multiple types of virtual object images may be determined based on past learning results (determined by AI (artificial intelligence)) and registered or updated.

[0239] By changing the operability and appearance of the GUI based on past combination data (not limited to AI), it is possible to provide an effective combination of virtual object images that matches the user's preferences and operability obtained from the past data. By using AI to learn this, it is possible to achieve a system that is more operable and more suited to the user's preferences.

[0240] Furthermore, by collecting past combination data for a larger number of users and storing the preferences of many users on a data server, these technologies can be customized as needed to improve the appearance and operability for an unspecified number of users or for specific user groups (left-handed users, female users, children, elderly users, etc.). These can also be done selectively on a menu screen or settings screen, or automatically by AI according to the user.

[0241] Furthermore, in addition to exchanging virtual object images, it may be possible to change the arrangement, for example, by swapping the shutter speed / sensitivity dial 234 and exposure compensation dial 236 shown in FIG. 1 and the like.

[0242] Although it has been described above that the user can change the appearance of the imaging device 200 to an original appearance by selecting and combining virtual object images showing the appearance of each divided area through an operation on the operation unit 230, the operation of changing the appearance to an original appearance may be performed through gesture control using image recognition technology in order to intuitively perform the operation of changing the appearance. That is, the virtual appearance may be changed by recognizing, through image recognition from the field of view image IM of the built-in camera 101 provided in the virtual image display device 100, that the user has made a gesture to change the virtual object image showing the appearance, and controlling the display so as to move or replace the virtual object image showing the appearance corresponding to the gesture.

[0243] Furthermore, to enable intuitive recognition of which divided area a virtual object image (skin) is pasted onto, the image of the imaging device 200 displayed on the XR display 140 may be made to appear to glow by increasing the brightness of the image, or by displaying a light beam around the image to indicate that the inner part is glowing.

[0244] These virtual object images are synthesized by appropriately performing geometric transformation based on the three-dimensional position and orientation of the image capturing device 200 .

[0245] Furthermore, by replacing the interchangeable lens unit 151, for example, from an actual wide-angle lens to a virtual telephoto lens, the virtual image display device 100 can enlarge and display the live view image LV1 as if it had been captured with a telephoto lens.

[0246] <Imaging Device Compatible with Use in XR Environment> FIG. 23 is a perspective view of an imaging device compatible with use in an XR environment.

[0247] The imaging device 280 shown in FIG. 23 includes a dummy lens 281 , three real lenses 282 , and a dummy monitor 283 .

[0248] The dummy lens 281 is a lens that only constitutes the exterior and has no photographing function.

[0249] The three real lenses 282 have different focal lengths, and the user can select one to use as appropriate. The live view images captured by the real lenses 282 are transmitted to the virtual image display device 100.

[0250] The dummy monitor 283 is a variable dummy monitor that does not have a display function and is rotatable relative to the camera body.

[0251] When the imaging device 200 described above is used in an XR environment, a live view image is not displayed on the main monitor 241 or the like, and the virtual image display device 100 displays a live view image LV1 superimposed on the display area of ​​the imaging device 200. However, in the case of the imaging device 280 of this example, the virtual image display device 100 displays a live view image superimposed on the dummy monitor 283 of the imaging device 280.

[0252] Furthermore, when the dummy monitor 283 rotates, the virtual image display device 100 can display a live view image superimposed on the rotated dummy monitor 283. This makes it possible to display a live view image LV1 similar to the cases shown in FIGS.

[0253] <Virtual Camera> FIG. 24 is a perspective view from the rear side showing an embodiment of a virtual camera that constitutes a virtual image display system according to the present invention.

[0254] 24 is a virtual camera equipped with at least a grip portion that can be held by hand and is provided with operation switches. In this case, although the term "virtual camera" is used, the grip portion is a physical object, and does not refer to a camera with a virtual display.

[0255] The virtual camera 290 is a dummy camera that does not have an imaging section such as a lens or an imaging element and does not have a camera function, and the virtual camera 290 in this example has an external shape consisting of only a grip section.

[0256] The grip portion of the virtual camera 290 is provided with operation members including at least a shutter button, but the operation members are dummy switches that are movable by user operation but do not generate operation signals.

[0257] Virtual camera 290 does not require a lens or an image sensor, and the communication unit can be significantly simplified, can be lightweight, and can easily be made smaller by folding, etc. Furthermore, even with virtual camera 290 that does not have a camera function, the user can still get the same feel as with a regular camera when framing and operating the operation members.

[0258] In this embodiment, the operation members of the virtual camera 290 are dummy switches that are movable by user operation but do not generate operation signals. However, in another embodiment, each operation member of the virtual camera 290 can be operated so that the user can actually feel the operation, and the virtual camera 290 may have an operation detection unit that detects user operations and a communication unit that transmits operation signals (not including transmission of live view images). By having the operation content of each operation member of the virtual camera 290 performed by the user transmitted via the communication unit to the virtual image display device 100 (or the virtual image display system 10 or the XR server 300 described later), the operation performed by the user can be reflected in real time.

[0259] Furthermore, the method of reflecting user operations in real time is not limited to this method. As another variation, each operating member of the virtual camera 290 can be operated to provide a real-time operational feel, but the quality of the operational feel can be improved even if the virtual camera does not have an operation detection unit or communication unit to detect the operations. In this case, the image capture device 200 or a camera eye for image recognition included in the virtual image display device 100 or virtual image display system 10 can be used to detect the user's finger movements and the operation status of the operating members through image recognition, thereby reflecting the user's operations in real time. In this case, the time required for reflection and the reflection error rate are generally inferior to those that have an operation detection unit and communication unit, but depending on the image recognition performance, it can be made comparable in practical terms.

[0260] FIG. 25 is a perspective view from the front side showing an embodiment of a virtual camera that constitutes a virtual image display system according to the present invention.

[0261] The virtual camera 290 shown in FIG. 25A is a perspective view of the virtual camera 290 shown in FIG. 24 viewed from the rear side, as viewed from the front side.

[0262] As described above, the virtual camera 290 is a dummy camera that does not have an imaging unit such as a photographing lens or an imaging element and does not have any camera function.

[0263] A virtual camera 290B shown in FIG. 25B has a small camera 293B, such as that found in a smartphone, built into the front surface of the grip in a location that is not hidden by the hand holding the grip.

[0264] In this case, by having a communication unit including an image transmission unit in addition to an operation detection unit that detects user operations, it is possible to transmit the operation details of each operation member of the image capture device 290B performed by the user and the live view image captured by the small camera 293B via the communication unit to the virtual image display device 100 (or the virtual image display system 10 or the XR server 300 described later). The subsequent flow is the same as in the embodiment in which a smartphone is also used, described later.

[0265] Furthermore, the virtual camera 290B can be used as a real imaging device using the camera function of the grip unit with camera. When using the camera function of the grip unit with camera, the live view image captured by the grip unit with camera may be complemented, the resolution enhanced, and corrected for overexposure and underexposure, as in the embodiment using a smartphone described later, using a field of view image IM of the built-in camera 101 of the virtual image display device 100 or a live view image acquired from the XR server 300 to create a final image.

[0266] 25(C) has a small camera 293C for image recognition built into the front of the grip, in a portion that is not hidden by the hand holding the grip. This small camera 293C may have a lower resolution than the small camera 293B shown in FIG. 25(B).

[0267] The virtual camera 290C includes a communication unit in addition to an operation detection unit that detects operations, but in this case, the communication unit does not need to transmit images. If the communication unit does not include image transmission, instead of transmitting the user's operations on each operation member of the virtual camera 290C and image data acquired by the miniature camera 293C, the communication unit can use image recognition or the like to convert information about the subject at which the virtual camera 290C is aimed and information about the range of the set angle of view, etc., from the image data acquired by the miniature camera 293C into numerical or text form, and transmit this information along with the operations on each operation member to the virtual image display device 100 (or the virtual image display system 10 or the XR server 300) via the communication unit. This allows for a smaller, lower-cost configuration than a communication unit that includes image transmission. The subsequent processing flow can be achieved by combining the processing of an embodiment using a dummy camera without camera functionality and the processing of an embodiment using a smartphone, which will be described later, and therefore will not be described here.

[0268] FIG. 26 is a perspective view showing another embodiment of the virtual camera constituting the virtual image display system according to the present invention.

[0269] A virtual camera 292 shown in FIG. 26 is a combination of the virtual camera 290 shown in FIG. 24 and a smartphone 291 .

[0270] This virtual camera 292 can be used as a virtual camera without a camera function, or as a real imaging device using the camera function of the smartphone 291. When using the camera function of the smartphone 291, the live view image captured by the smartphone 291 may be subjected to complementation, resolution enhancement, and correction of blown-out highlights and crushed shadows, etc., using a field of view image IM of the built-in camera 101 of the virtual image display device 100 or a live view image acquired from the XR server 300, as described below, to obtain a final image.

[0271] The virtual camera 290 shown in FIG. 24 and the virtual camera 292 when the camera function of the smartphone 291 is not used cannot capture live view images, but by setting a virtual imaging unit, it is possible to obtain images (live view images) corresponding to the angle of view of the virtual cameras 290 and 292.

[0272] FIG. 27 is a diagram showing an embodiment of a method for acquiring a live view image captured by a virtual camera.

[0273] As described above, the built-in camera 101 of the virtual image display device 100 captures a visual field image IM corresponding to the user's visual field including the virtual camera 290, and acquires the captured visual field image IM. In FIG. 27 , L1 is the optical axis of the built-in camera 101.

[0274] A live view image captured by virtual camera 290 when it has a camera function can be specified by the three-dimensional position, orientation (capturing direction), and angle of view of virtual camera 290. In Fig. 27 , L2 is an optical axis assumed in virtual camera 290, and can be determined from the three-dimensional position and orientation of virtual camera 290.

[0275] Similar to the imaging device 200, the virtual image display device 100 can acquire information indicating the three-dimensional position and orientation of the virtual camera 290 with respect to the XR display 140.

[0276] Therefore, by setting the angle of view of the virtual camera 290, it is possible to determine which range of the field of view image IM captured by the built-in camera 101 corresponds to the live view image captured using the three-dimensional position, posture, and angle of view of the virtual camera 290.

[0277] That is, the virtual image display device 100 can extract an image corresponding to the angle of view of the virtual camera 290 from the field of view image IM captured by the built-in camera 101 and acquire it as a live view image. Note that the angle of view of the built-in camera 101 is wider than the angle of view set in the virtual camera 290, and the field of view image IM includes the live view image captured by the virtual camera 290.

[0278] In this way, the virtual image display device 100 can acquire a live view image that can be captured if the virtual camera 290 had an imaging unit, from the field of view image IM captured by the built-in camera 101.

[0279] The virtual image display device 100 can perform geometric transformation on the acquired live view image and display it in a display area set for the virtual camera 290 (for example, an area adjacent to the grip portion, which is the virtual camera 290).

[0280] In addition, in the case of the virtual camera 292 shown in FIG. 26 , the virtual image display device 100 can perform geometric transformation on the acquired live view image and display it superimposed on the display area of ​​the smartphone 291 of the virtual camera 292 .

[0281] Furthermore, the virtual image display device 100 can detect user operation (gestures) of switches on the virtual camera 290 by analyzing the field of view image IM captured by the built-in camera 101, and can capture, save, and perform other processing for recording based on the user operation of the switches.

[0282] Furthermore, the virtual camera is not limited to a grip portion, and various types with different external shapes, sizes, and the presence or absence of switches are conceivable.

[0283] In addition, the virtual camera 290 in this example does not have a camera function, but it does have real switches and a communication unit, etc., and operation information of the switches may be transmitted to the virtual image display device 100 via a simple communication unit provided in the virtual camera 290.

[0284] Second Embodiment of XR Display FIG. 28 is a diagram showing a second embodiment of the XR display of the virtual image display device.

[0285] The XR display 141 shown in FIG. 28 is an immersive display that covers the field of view, and is a wearable device with a goggle-type virtual image display function, and is composed of a built-in camera 101, an image synthesis unit 143, a display element 145, and an optical system 147.

[0286] The image (field of view image IM) captured by the built-in camera 101 is output to the image synthesis unit 143.

[0287] The other input of the image synthesis unit 143 is the geometrically transformed live view image LV1, and the image synthesis unit 143 synthesizes (pastes) the live view image LV1 into the display area of ​​the camera image contained in the field of view image IM and outputs it to the display element 145.

[0288] The display element 145 displays the visual field image IM combined with the live view image LV1. The visual field image IM is displayed as a virtual image by the optical system 147 at a display position in front of the user's eyes.

[0289] A user wearing the goggle-type XR display 141 can view a virtual image of the visual field image IM displayed on the display element 145 through the optical system 147. Because the user's visual field is covered by the goggle-type XR display 141, the user cannot see anything other than the visual field image IM, and can become deeply immersed in a non-real world.

[0290] 28 shows only one XR display 141, but in the case of a goggle type, a pair of left and right XR displays 141 corresponding to both eyes are provided. Therefore, the built-in cameras 101 and the like are also provided on the left and right, and live view images LV1 for the left and right are generated.

[0291] [Second Embodiment of Virtual Image Display System] FIG. 29 is a block diagram showing a second embodiment of the virtual image display system according to the present invention.

[0292] The virtual image display system 11 shown in FIG. 29 is composed of a virtual image display device 180, a virtual camera 290, and a server (XR server) 300.

[0293] The virtual image display device 180 has almost the same components as the virtual image display device 100 shown in FIG. 6, but differs from the virtual image display device 100 in that it transmits and receives necessary information to and from the XR server 300.

[0294] In FIG. 29, the XR server 300 includes a database (DB) 302 that manages three-dimensional images (XR space construction data) having three-dimensional information of the real space.

[0295] The XR server 300 collects images taken by a number of cameras, including a multi-view camera 320, a surveillance camera 330, a drive recorder 340, a drone camera 310, or a mobile device camera, placed in the real space, generates the latest XR space construction data, and manages it in a database 302.

[0296] The database 302 digitizes town blocks, cities, shopping streets, tourist spots, etc. and manages them as XR space construction data. It is also preferable to link multiple XR servers 300 together to manage XR space construction data for a wider area.

[0297] The XR processor 111 of the virtual image display device 180 has the same functions as the XR processor 110 of the virtual image display device 100 shown in FIG. 6, but furthermore, based on information (first information) indicating the three-dimensional position, attitude, and angle of view of the virtual camera 290, cuts out XR space construction data corresponding to the shooting area of ​​the virtual camera 290 from the XR space construction data managed by the XR server 300, converts the cut-out three-dimensional XR space construction data into a two-dimensional image, and uses this two-dimensional image as a live view image.

[0298] The XR processor 111 converts the three-dimensional XR space construction data read from the database 302 into a two-dimensional image by performing perspective projection transformation using the internal camera parameters of the virtual camera 290 (in this example, a preset angle of view or focal length) based on the three-dimensional position and orientation (shooting direction) of the virtual camera 290, and this is used as a live view image LV.

[0299] Furthermore, the XR processor 111 analyzes the field of view image IM captured by the built-in camera 101 to identify the display area of ​​the virtual camera 290, acquires transformation information to be used when superimposing the live view image LV on the display area of ​​the virtual camera 290, and performs geometric transformation on the live view image LV based on the acquired transformation information; however, if the transformation information is acquired based on the relative three-dimensional position and orientation of the XR display 140 of the virtual image display device 180 and the virtual camera 290 and the field of view image IM is not used on the XR display, the built-in camera 101 can be omitted. The XR processor 111 may also generate a better live view image using image data transmitted from the imaging devices of other people in the vicinity.

[0300] Furthermore, when generating the latest XR space construction data, the XR server 300 preferably uses not only current images but also previously recorded images as images from the multi-viewpoint camera 320, etc., to construct XR space construction data of higher image quality than a single image by compositing multiple images. By using multiple images and processing and compositing them by taking advantage of slight pixel shifts, the number of pixels in the image can be increased.

[0301] When creating a complementary image using multiple images, it is preferable to complement each other with blown-out highlights, blocked-up shadows, etc., to complement areas where image data is unavailable or of poor quality, and when complementing areas of poor quality, it is preferable to correct the brightness of the complementary image so that the relationship between light and dark caused by the intensity of light is not reversed. Furthermore, when correcting the brightness of the complementary image, it is preferable to correct the gamma curve so that it fits within the overall display latitude (within the printing latitude), or to correct the printing parameter curve so that it fits within the overall printing latitude.

[0302] When combining with the main subject image, it is preferable to perform corrections to match the brightness, light direction, color temperature, and gamma curve (printing parameters) to the image of the multi-viewpoint camera 320. Furthermore, when combining images other than the main subject from past and current images of the multi-viewpoint camera 320, it is preferable to detect changing conditions such as weather and time from the current image, and convert and combine the past images and XR space construction data according to the detected changing conditions such as weather and time.

[0303] The XR server 300 may also combine a virtual object image with the XR space construction data. For example, if the XR space construction data includes an image of a person, the XR server 300 can convert the image into an avatar that resembles that person and change the image of the avatar to replace the image of the person. Furthermore, if the XR server 300 identifies a person from the image of the person and the identified person has registered their own avatar image in the XR server 300, the XR server 300 can change the image of the person to the image of the registered avatar.

[0304] This allows the virtual image display device 180 to acquire a live view image LV including an image of the avatar from the XR server 300.

[0305] It is conceivable that an image in which oneself appears may be used by another person, and in this case, it is preferable to notify or warn of this fact on the virtual image display device 100. Furthermore, it is possible to set whether to allow or not allow the image in which oneself appears, and if it is set to allow, notification etc. can be omitted.

[0306] Furthermore, if an image of the user is used by another person, or if the image is set to be prohibited, the device can notify the other person of this fact, warn them, or send a signal to lock the other person's photography, or a signal to blur the image of the user, or a generation prohibition signal to prevent the other person from generating an image of the other person. When the other person's device receives the generation prohibition signal, it is preferable to process the image so that an image of the prohibited subject is not generated (by erasing it, painting it over with a background, etc.).

[0307] Furthermore, in this embodiment, the virtual image display device 180 acquires the live view image LV from the XR server 300, but this is not limiting. The virtual image display device 180 may transmit information necessary for extracting the live view image LV to the XR server 300, and the XR server 300 may acquire the live view image LV and transmit it to the virtual image display device 180, or the live view image LV may be geometrically transformed and the geometrically transformed live view image LV1 may be transmitted to the virtual image display device 100.

[0308] In addition, the XR server 30 may acquire information indicating the field of view range of the XR display 140, acquire a field of view image IM in the same manner as acquiring a live view image LV, and transmit the image to the virtual image display device 100.

[0309] Furthermore, although this embodiment uses the virtual camera 290, it can also be applied to a case where the imaging device 200 is used. In this case, when the imaging device 200 is used in an XR environment, the imaging device 200 does not capture a live view image, but acquires the image from the XR server 300.

[0310] In addition, the latest XR space construction data is registered in the database 302 of the XR server 300, but since this is not necessarily real-time XR space construction data, the live view image LV generated from the latest XR space construction data is not necessarily a real-time image.

[0311] Therefore, it is preferable that the virtual image display device 180 adjusts the brightness and color of the live view image LV to match the brightness and color of the field of view image IM captured by the built-in camera 101, for example.

[0312] The user can freely manipulate the posture of the virtual camera 290 by holding the virtual camera 290 in his / her hand, and the three-dimensional position of the virtual camera 290 is not a real position, but can be changed to any position by user operation, allowing the user to enjoy taking pictures of tourist spots and the like from the comfort of their own home.

[0313] [First Embodiment of Virtual Image Display Method] FIG. 30 is a flowchart showing a first embodiment of a virtual image display method according to the present invention.

[0314] The virtual image display method of the first embodiment shown in FIG. 30 is performed by the virtual image display device 100, and is a method mainly performed by the XR processor 110 of the virtual image display device 100 shown in FIG.

[0315] In FIG. 30, the XR processor 110 acquires a field of view image IM captured by the built-in camera 101 (step S100).

[0316] The XR processor 110 analyzes the visual field image IM and extracts a camera image showing the imaging device 200 included in the visual field image IM (step S102). When a live view image is captured by the imaging device 200 while the imaging device 200 is held in hand, the visual field image IM captured by the built-in camera 101 typically includes a camera image showing the imaging device 200, as shown in FIG.

[0317] The XR processor 110 determines whether the imaging device 200 is an XR-compatible camera from the extracted camera image (step S104). This determination can be made, for example, by comparing the extracted camera image with appearance data of the XR-compatible camera pre-stored in the first memory 120. In addition, the determination can be made based on whether the AR marker M provided on the imaging device 200 matches a pre-registered AR marker M.

[0318] When the XR processor 110 determines that the imaging device 200 captured by the built-in camera 101 is an XR-compatible camera (if "Yes"), it then detects whether the imaging device 200 is in XR mode or non-XR mode (step S106).

[0319] When the imaging device 200 is set to XR mode and displays an AR marker M on the main monitor 241 or the sub-monitor 242, the XR mode / non-XR mode can be detected by detecting the AR marker M. Furthermore, when the imaging device 200 has a dedicated mode switching lever, the XR mode / non-XR mode can be detected by image recognition of the position of the mode switching lever. Furthermore, when a camera image is extracted from the field of view image IM, the XR processor 110 automatically establishes a wireless connection from the virtual image display device 100 to the imaging device 200 and detects the XR mode / non-XR mode based on the communication result with the imaging device 200. For example, when the imaging device 200 does not respond to a call from the virtual image display device 100, it can be determined that the imaging device 200 is not set to XR mode.

[0320] In step S108, if it is determined from the XR mode / non-XR mode detection result that the imaging device 200 is not set to the XR mode (in the case of "No"), the XR processor 110 turns off the overlay synthesis function of the live view image in the virtual image display device 100 (smart glasses) (step S110), and returns to step S100. This allows the user to see only the real world transmitted through the smart glasses.

[0321] On the other hand, in step S108, if it is determined from the XR mode / non-XR mode detection result that the image capture device 200 is set to XR mode (if "Yes"), the display area A (see FIG. 10) of the image capture device 200 is identified based on the extracted camera image. The display area A can be identified by detecting the rectangular outline of the monitor panel 240 or by detecting AR markers M2 to M4 attached to the image capture device 200.

[0322] Next, the XR processor 110 acquires transformation information to be used when overlaying the live view image LV on the display area A of the imaging device 200. When affine transforming the live view image LV to overlay it on the display area A, the XR processor 110 acquires affine transformation parameters (transformation matrix) as the transformation information, and when projectively transforming the live view image LV to overlay it on the display area A, the XR processor 110 acquires the projective transformation transformation matrix as the transformation information. Note that the transformation matrix can be found from the coordinates of multiple feature points in the live view image LV and the coordinates of multiple corresponding points in the display area A that correspond to the multiple feature points.

[0323] Next, the XR processor 110 receives a live view image LV from the imaging device 200 (step S116). The live view image LV is continuous data, but in step S116, image data for one frame is received.

[0324] The XR processor 110 performs geometric transformation on the live view image LV (one frame of image data) based on the transformation information acquired in step S114, and outputs the geometrically transformed live view image LV1 to the XR display 140 (step S118).

[0325] 11 , the XR display 140 receives the geometrically transformed live view image LV1 and displays it on the display element 144 (step S120). The user can view the live view image LV1 superimposed on the real world (display area A of the imaging device 200) through the optical system 146 of the XR display 140.

[0326] The XR processor 110 determines whether to end the image display of the live view image LV1 on the XR display 140 (step S122). If the image display is not to be ended, the process proceeds to step S100, and the processes from step S100 to step S122 are repeated for each frame of the live view image LV. If the image display is not to be ended, the process by the virtual image display device 100 is ended.

[0327] [Display Method by Imaging Device] FIG. 31 is a flowchart showing a display method by the imaging device according to the present invention.

[0328] The display method by the imaging device shown in FIG. 30 is a method that is mainly performed by the camera processor 210 of the imaging device 200 shown in FIG.

[0329] In FIG. 31, the camera processor 210 first determines whether or not the XR mode has been set by the mode switching unit (step S200).

[0330] The user can set the XR mode when the main monitor 241 is difficult to see in bright places, when the user wants to display the live view image LV1 on the virtual monitor larger than the actual main monitor 241, or when the user wants to photograph various virtual objects including avatars, and can set the non-XR mode when the user wants to prioritize more natural usability and want to observe the live view image, etc. displayed on the actual main monitor 241, thereby allowing the user to use the imaging device 200 appropriately depending on the advantages and disadvantages of the XR mode and non-XR mode.

[0331] If it is determined in step S200 that the XR mode is set (if "Yes"), the camera processor 210 displays an AR marker M on the sub-monitor 242 of the closed monitor panel 240. This AR marker M can be used as a marker that identifies the display area A of the live view image LV1, and / or can be used as camera identification information for distinguishing the camera from other imaging devices.

[0332] Next, the camera processor 210 transmits the live view image LV, which has been captured by the imaging unit 201 and subjected to RAW development processing, to the virtual image display device 100 via the second communication unit 260 (step S220).

[0333] On the other hand, if it is determined in step S200 that XR mode is not set (in the case of non-XR mode), the camera processor 210 determines whether the tiltable monitor panel 240 is open (step S230). If it is determined that the monitor panel 240 is open, the camera processor 210 outputs a live view image LV that has been captured by the imaging unit 201 and subjected to RAW development processing to the main monitor 241, and causes the main monitor 241 to display a live view image LV1. This allows the user to check the current shooting range, etc., from the live view image LV displayed on the main monitor 241.

[0334] On the other hand, if it is determined that the monitor panel 240 is not open ("No"), the camera processor 210 then determines whether the viewfinder 243, which is a hybrid finder, has been switched to an EVF or an OVF (step S250). This determination can be made based on the user's operation of the finder switching lever.

[0335] If the viewfinder 243 has been switched to the EVF, the camera processor 210 causes the EVF to display the live view image LV1 (step S260). On the other hand, if the viewfinder 243 has been switched to the OVF, the camera processor 210 causes the OVF to display an optical image (finder image) (step S270). In this case, the process of generating the live view image LV1 can be stopped.

[0336] By looking through the eyepiece of the viewfinder 243, the user can check the live view image LV1 displayed on the EVF or the finder image displayed on the OVF.

[0337] [Second Embodiment of Virtual Image Display Method] FIG. 32 is a flowchart showing a second embodiment of the virtual image display method according to the present invention.

[0338] The virtual image display method of the second embodiment shown in FIG. 32 is a method that is mainly performed by the XR processor 110 of the virtual image display device 100 shown in FIG. 6, similar to the first embodiment shown in FIG.

[0339] In FIG. 32, the XR processor 110 first checks the imaging device 200 (step S300).

[0340] FIG. 33 shows a subroutine illustrating an embodiment of the process of step S300 shown in FIG. 32 for checking the image pickup device.

[0341] In FIG. 33, the XR processor 110 acquires a visual field image IM captured by the built-in camera 101 (step S301), and extracts the camera image of the imaging device 200 present in the visual field image IM (step S302).

[0342] Next, the XR processor 110 performs image analysis on the camera image and performs processing to extract the AR marker M (step S303). Since the imaging device 200, which is an XR-compatible camera in this example, is provided with the AR marker M, it is possible to confirm the presence or absence of the imaging device 200 from the presence or absence of the AR marker M.

[0343] In step S304, if it is determined that the AR marker M is present (if "Yes"), the process proceeds to step S305, where the XR processor 110 starts communication with the target device having the AR marker M.

[0344] Next, the XR processor 110 confirms through communication whether the target device is the user's imaging device 200 (step S306). The XR processor 110 can confirm whether the target device is the user's imaging device 200 by, for example, checking whether the devices have been paired once. Note that the virtual image display device 100 and the user's imaging device 200 can be paired by a known method.

[0345] In step S307, if it is determined that the target device is the user's imaging device 200 (if "Yes"), the XR processor 110 confirms that the camera image included in the field of view image IM is the user's imaging device 200 (step S308).

[0346] On the other hand, if it is determined in step S304 that the AR marker M is not present in the field of view image IM (camera image) (if "No"), or if it is determined in step S307 that the target device is not the user's imaging device 200 (if "No"), the XR processor 110 determines that it was unable to confirm the user's imaging device 200 (unconfirmed) (step S309).

[0347] 32, when the presence of the user's imaging device 200 is confirmed in step S310 (if "Yes"), the XR processor 110 determines whether the imaging device 200 is in the XR mode (step S312). The method for determining the XR mode can be performed in the same manner as described above, and therefore, a description thereof will be omitted here.

[0348] If it is determined in step S312 that the imaging device 200 is set to the XR mode (if "Yes"), the XR processor 110 checks whether a live view image, a playback image, or shooting information INFO should be superimposed and displayed in the display area A of the imaging device 200 (step S314).

[0349] When used in an XR environment, the imaging device 200 can accept a user instruction as to whether to display a live view image, a playback image, or shooting information INFO in the display area A of the imaging device 200, and transmits information corresponding to the accepted user instruction (live view image, playback image, or shooting information INFO) to the virtual image display device 100.

[0350] Therefore, based on the information (such as a live view image) received from the imaging device 200, the XR processor 110 can confirm whether to superimpose and display a live view image, a playback image, or shooting information INFO on the display area A of the imaging device 200.

[0351] If it is determined in step S318 that a live view image is to be displayed, the XR processor 110 transitions to step S320, and displays the live view image received from the imaging device 200 on the XR display 140 in a superimposed manner.

[0352] FIG. 34 shows a subroutine illustrating an embodiment of the process of step S3200 for superimposing and displaying the live view image LV1 shown in FIG.

[0353] In FIG. 34, the XR processor 110 receives a live view image LV from the imaging device 200 (step S321).

[0354] Next, the XR processor 110 acquires shooting information INFO corresponding to the received live view image LV (step S322). Here, the shooting information INFO is, for example, text information, and is attached information associated with one frame or each of multiple frames of the live view image LV.

[0355] In step S323, if it is determined that the shooting information INFO is "present" (if "Yes"), the XR processor 110 transitions to step S324, where it combines the shooting information INFO with the peripheral area of ​​the live view image LV (step S324).

[0356] On the other hand, if it is determined in step S323 that the shooting information INFO is "absent" (in the case of "No"), the XR processor 110 skips step S324 and transitions to step S325.

[0357] Note that even if the XR processor 110 acquires the shooting information INFO, if the transfer information of the shooting information INFO includes information indicating "not displayed," the XR processor 110 determines that the shooting information INFO is "absent," skips step S324, and transitions to step S325. Also, in the case where a live view image is created by operating the imaging device 200 based on data acquired from the XR server 300, even if the shooting information INFO of the live view image is acquired, if it is the shooting information INFO of the live view image created by operating the imaging device 200, the XR processor 110 determines that the shooting information INFO is "absent," skips step S324, and transitions to step S325.

[0358] In step S325, the XR processor 110 acquires a field of view image IM from the built-in camera 101, and detects the positions of multiple AR markers M on the imaging device 200 based on the acquired field of view image IM.

[0359] Thereafter, the XR processor 110 performs the processes of steps S326 to S329 to display the live view image LV1 superimposed on the display area A of the imaging device 200 observed on the XR display 140. Note that the processes of steps S326 to S329 are similar to the processes of steps S112, S114, S118, and S120 in the flowchart shown in Fig. 30, and therefore detailed description thereof will be omitted.

[0360] Returning to FIG. 32, when it is determined in step S318 that the imaging information is to be displayed, the XR processor 110 transitions to step S330, and displays the imaging information INFO received from the imaging device 200 on the XR display 140 in a superimposed manner.

[0361] FIG. 35 shows a subroutine illustrating an embodiment of the process of step S330 for superimposing and displaying the shooting information INFO shown in FIG.

[0362] In FIG. 35, the XR processor 110 turns off the superimposed display function of the live view image LV1 in the virtual image display device 100 and turns on the superimposed display function of the shooting information INFO (step S331).

[0363] Next, the XR processor 110 receives the shooting information INFO of text data from the imaging device 200 (step S332).

[0364] The XR processor 110 generates a shooting information image (raster image or vector image) indicating the shooting information INFO from the received shooting information INFO (step S333).

[0365] Thereafter, the XR processor 110 performs the processing of steps S334 to S338 to superimpose and display a shooting information image indicating the shooting information INFO on the display area A of the imaging device 200 observed on the XR display 140. Note that the processing of steps S334 to S338 differs from the processing of steps S325 to S329 in the flowchart shown in Fig. 34 in that a shooting information image is displayed instead of the live view image LV1 (i.e., the image to be displayed is different), but since the other processing is common, detailed description thereof will be omitted.

[0366] Returning to FIG. 32, if it is determined in step S318 that a playback image is to be displayed, the XR processor 110 transitions to step S340, and displays the playback image received from the imaging device 200 on the XR display 140 in a superimposed manner.

[0367] FIG. 36 shows a subroutine illustrating an embodiment of the process of step S340 for displaying the reproduced images in a superimposed manner as shown in FIG.

[0368] 36, the camera processor 210 of the imaging device 200 identifies whether the playback image selected in the imaging device 200 is an image of the imaging device 200 (a captured image recorded in the second memory 220 of the imaging device 200) or an image on a network (not shown) (a user's image managed by a server on the network) (step S280). Note that the imaging device 200 in this case is connectable to a network and can store captured images in a server on the network or read captured images from the server.

[0369] If it is determined in step S281 that the image is stored in the image capture device 200, the camera processor 210 reads the reproduced image from the second memory 220 of the image capture device 200 in accordance with a user operation (step S282). Note that the reproduced image may be read from a memory card that is detachable from the image capture device 200 instead of the second memory 220, which is a built-in memory.

[0370] On the other hand, if it is determined in step S281 that the image is on the network, the camera processor 210 reads out a reproduced image from the user's image group stored on the network in accordance with the user's operation (step S283).

[0371] The camera processor 210 transmits the reproduced image read out in step S282 or S283 to the virtual image display device 100 (step S284).

[0372] The processes in steps S280 to S284 are performed by the image capture device 200, and the following processes are performed by the virtual image display device 100.

[0373] The XR processor 110 of the virtual image display device 100 receives the reproduced image transmitted from the imaging device 200 (step S341).

[0374] Thereafter, the XR processor 110 performs the processing of steps S342 to S346 to display the playback image superimposed on the display area A of the imaging device 200 observed on the XR display 140. Note that the processing of steps S342 to S346 differs from the processing of steps S325 to S329 in the flowchart shown in Fig. 34 in that a playback image is displayed instead of the live view image LV1 (i.e., the image to be displayed is different), but since the other processing is common, detailed description thereof will be omitted.

[0375] In this way, the virtual image display device 100 can superimpose a live view image LV1, shooting information INFO, or a playback image on the display area A of the imaging device 200 observed by the XR display 140, and a user wearing the virtual image display device 100 (smart glasses) can view the live view image LV1, etc. superimposed on the real world transmitted through the smart glasses.

[0376] <Method for Changing Appearance of Imaging Device> FIG. 37 is a flowchart showing an embodiment of a method for changing the appearance of an imaging device.

[0377] The method of changing the appearance of the imaging device 200 shown in FIG. 37 is a method that is mainly performed by the XR processor 110 of the virtual image display device 100 shown in FIG.

[0378] Note that the method for changing the appearance of the imaging device 200 in this example does not physically change the appearance of the imaging device 200, but changes the appearance of the imaging device 200 observed by the virtual image display device 100 (smart glasses) on the image.

[0379] In FIG. 37, the XR processor 110 acquires the field of view image IM captured by the built-in camera 101 (step S350), and extracts the camera image of the imaging device 200 present in the field of view image IM (step S352).

[0380] The XR processor 110 determines whether the imaging device 200 is an XR-compatible camera from the extracted camera image (step S354). This determination can be made, for example, by comparing the extracted camera image with the appearance data of the XR-compatible camera stored in advance in the first memory 120.

[0381] If it is determined in step S354 that the image capture device 200 captured by the built-in camera 101 is an XR-compatible camera (if "Yes"), the XR processor 110 acquires XR compatibility data of the image capture device 200 (step S356). The XR compatibility data includes data indicating the external shapes of the interchangeable lens unit, leather exterior unit, operation dial unit, upper exterior unit, viewfinder unit, monitor unit, etc. of the image capture device 200.

[0382] On the other hand, if it is determined in step S354 that the image capturing device 200 captured by the built-in camera 101 is not an XR compatible camera ("No"), the XR processor 110 turns off the overlay synthesis function of the live view image in the virtual image display device 100 (smart glasses) (step S358), and returns to step S350. This allows the user to see only the real world (the actual image capturing device 200) transmitted through the smart glasses.

[0383] Next, the XR processor 110 identifies, as display areas, multiple regions obtained by dividing the camera image based on the camera image extracted in step S352 and the XR-compatible data acquired in step S356 (step S360). That is, the XR processor 110 identifies, as display areas, the areas corresponding to the XR-compatible data, such as the interchangeable lens unit, leather exterior unit, operation dial unit, upper exterior unit, viewfinder unit, and monitor unit.

[0384] Next, the XR processor 110 acquires component images to be composited into one or more of the display areas of the imaging device 200 based on the user's selection operation (step S362). Component images (virtual object images) showing the appearance of each of the divided areas into which the appearance of the imaging device 200 is divided are stored in the first memory 120 of the virtual image display device 100. The user can display the component images stored in the first memory 120 on the main monitor 241 and select a component image to be composited into a display area using the operation unit 230. The component image may be an image pasted on the surface of a three-dimensional model of the component.

[0385] The XR processor 110 acquires transformation information used to overlay the three-dimensional component image onto the two-dimensional display area corresponding to the component image (step S364). Because the three-dimensional data of the component image is known and the two-dimensional data of the display area can also be detected, transformation information for projecting the three-dimensional component image so that it matches the corresponding display area can be acquired.

[0386] The XR processor 110 performs geometric transformation on the part image based on the transformation information acquired in step S364, and outputs the geometrically transformed part image to the XR display 140 (step S366).

[0387] The XR display 140 receives the geometrically transformed component image and displays it on the display element 144, thereby superimposing the component image on the display area of ​​the imaging device 200 (step S368). The user can view the component image (the skin that changes the appearance) superimposed on the real world (the display area of ​​the imaging device 200) through the optical system 146 of the XR display 140. In other words, the user can create an original imaging device with an appearance that suits their preferences. Furthermore, one imaging device can be used with multiple different appearances.

[0388] It goes without saying that the live view image LV1 or the like can be superimposed and displayed on the display area of ​​the monitor unit of the imaging device while changing the appearance of the imaging device.

[0389] [Others] When the XR mode is set, the imaging device 200 of this embodiment can take selfies. In this case, a live view image LV1 can be displayed in a display area set on the front of the imaging device 200, or in a display area A on the sub-monitor 242 side of the monitor panel 240 rotated 180 degrees. In addition, when the user wears smart glasses or VR goggles, it is preferable to superimpose an avatar's facial image onto the user's face area.

[0390] Furthermore, in a non-XR environment, the imaging device 200 has a function of switching between a display mode in which the main monitor 241 combines a live view image LV with shooting information INFO and displays it, a display mode in which the main monitor 241 displays only the live view image LV, and a display mode in which the main monitor 241 displays only the shooting information INFO, each time the display / back button 247 is pressed; however, when the imaging device 200 is used in an XR environment, it is preferable that the virtual image display device 100 displays the live view image LV1, etc. in accordance with the display mode set in the non-XR environment.

[0391] It is preferable that the XR processor 110 of the virtual image display device 100 changes the brightness of the live view image LV1 to be synthesized according to the brightness information of the field of view image IM captured by the built-in camera 101 so that the image is bright when the display area A of the imaging device 200 is bright and dark when it is dark, and it is preferable that the brightness of the live view image LV1 is controlled within a range that does not exceed a preset threshold value so that whiteout or blackout does not occur when the brightness is such that whiteout or blackout occurs.

[0392] Furthermore, in this embodiment, the hardware structure of a processing unit that executes various processes, such as a CPU (Central Processing Unit), is the following various processors: The various processors include a CPU, which is a general-purpose processor that executes software (programs) to function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0393] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0394] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0395] Furthermore, the present invention includes a plurality of the above-described embodiments, and it goes without saying that modifications that act selectively or in combination are possible as long as the photographing device or system of the present invention can be applied.

[0396] Furthermore, the "first display" of the virtual image display device is a wearable device with a virtual image display function in the form of glasses or goggles, and provides images including virtual images using an HMD device or see-through smart glasses. However, any display unit as the first display may be used as long as it can apply the imaging device or system of the present invention and can realize the display that is the essence of the present invention, such as a retinal projection device built into smart glasses or a method of displaying virtual images using the principle of projection mapping by using a projector as the display method of the first display.

[0397] Furthermore, in this embodiment, the virtual image display device 100 performs geometric transformation on the live view image LV and outputs the geometrically transformed live view image LV1 to the XR display 140, but the virtual image display device 100 may acquire the geometrically transformed live view image LV1 from the XR server 300 as described above, or may acquire the geometrically transformed live view image LV1 from the imaging device 200. In this case, the imaging device 200 needs to acquire a field of view image IM from the virtual image display device 100 or acquire information on the relative three-dimensional position and orientation of the virtual image display device 100 and the imaging device 200 in order to perform the process of geometrically transforming the live view image LV.

[0398] Furthermore, when the imaging device 200 is used in an XR environment (when the XR mode is set), the virtual image display device 100 displays the live view image LV1 superimposed on the display area of ​​the imaging device 100 on the XR display 140. However, even when the imaging device 200 is used in a non-XR environment (a normal environment) (when the non-XR mode is set), the superimposition function of the live view image LV1 in the virtual image display device 100 may be turned on. This has the advantage that, when the imaging device 200 is used in a normal environment, the live view image LV1 can be clearly observed even if the display of the imaging device 200 is difficult to see due to exposure to direct sunlight.

[0399] Furthermore, even when using this technology "in use in an XR environment," a mode may be provided in which the live view image LV is not automatically transmitted to the virtual image display device 100, and the live view image may be selectively displayed on the display of the imaging device 100.

[0400] Furthermore, it goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention.

[0401] 10, 11...Virtual image display system 100...Virtual image display device 101...Built-in camera 102...Photographing lens 104...Image sensor 110, 111...XR processor 112...Conversion information acquisition unit 114...Synthesis unit 116...Geometric transformation unit 120...First memory 130, 230...Operation unit 140, 141...XR display 142...Display control unit 143...Image synthesis unit 144, 145...Display element 146, 147...Optical system 146A...Lens 146B...Half mirror 150...Virtual object generation unit 151...Interchangeable lens unit 152...Leather exterior unit 153...Operation dial unit 154...Upper exterior unit 155...Finder unit 156...Monitor unit 160...First communication unit 170, 270...GPS receiver 172, 272... Gyro sensor 174, 274... Acceleration sensor 176, 276... Geomagnetic sensor 180... Virtual image display device 200, 280... Imaging device 201... Imaging unit 202... Interchangeable lens 203... Camera body 204... Image sensor 210... Camera processor 220... Second memory 232... Shutter button 234... Sensitivity dial 236... Exposure compensation dial 240... Monitor panel 241... Main monitor 242... Sub-monitor 243... Viewfinder 244... Toggle switch 245... Execute button 246... Playback button 247... Button 248... Display control unit 250... Image processing unit 260... Second communication unit 281... Dummy lens 282... Real lens 283... Dummy monitor 290, 290B, 290C, 292... Virtual camera 291...Smartphone 300...XR server 302...Database 310...Drone camera 320...Multi-view camera 330...Surveillance camera 340...Driving recorder A...Display area IM...Field of view image INFO...Shooting information LV, LV1...Live view image M, M1 to M9...AR marker P1 to P4...Feature points P1' to P4'...Corresponding points S100 to S122, S200 to S284, S300 to S368...Steps

Claims

1. 1. A virtual image display device having a processor and a first display, The virtual image display device receives a live view image from a first imaging device, The processor, acquiring information used when superimposing the live view image on a display area of ​​the first imaging device; Transforming the live view image based on the information; The first display displays the converted live view image. Virtual image display device.

2. The first display is a wearable device having a glasses-type or goggle-type virtual image display function. The virtual image display device according to claim 1 .

3. a second imaging device that captures an image of a user's visual field; The processor analyzes an image captured by the second imaging device to identify the display area and acquire the information. The virtual image display device according to claim 1 .

4. The processor acquires the information based on a position of the marker of the first imaging device photographed by the second imaging device. The virtual image display device according to claim 3 .

5. acquiring information indicating a three-dimensional position and orientation of the first imaging device with respect to the first display, and acquiring the information based on the information indicating the three-dimensional position and orientation; The virtual image display device according to claim 1 .

6. the first imaging device has a second display that displays the live view image; the display area is the display area of ​​the second display; The virtual image display device according to claim 1 .

7. the second display includes a rotation mechanism that can change a relative attitude of the first imaging device to a device body by rotation; the display area is a rotated display area of ​​the second display, The virtual image display device according to claim 6 .

8. the first imaging device has a viewfinder; the display area is a display area of ​​the viewfinder; A virtual image display device according to any one of claims 1 to 5.

9. the first display is a display that is transparent to the visual field and displays the converted live view image in the visual field of the real world; The virtual image display device according to claim 1 .

10. A second imaging device is provided for capturing an image of a visual field of a user, the first display is an immersive display that covers a field of view and displays a composite image obtained by superimposing the converted live view image on an image captured by the second imaging device. The virtual image display device according to claim 1 .

11. The processor combines a virtual object image with the live view image. The virtual image display device according to claim 1 .

12. A virtual image display device having a processor, a first display, and a second imaging device that captures a user's field of view, The processor, extracting an image corresponding to at least an angle of view of a virtual camera having a grip portion from the image captured by the second imaging device and acquiring the image as a live view image; acquiring information used when overlaying the live view image on a display area of ​​the virtual camera; Transforming the live view image based on the information; The first display displays the converted live view image. Virtual image display device.

13. The transformation is a geometric transformation. The virtual image display device according to any one of claims 1 to 5 and 12.

14. An imaging unit that captures a live view image; a mode switching unit that switches between a first mode and a second mode; a second display or a viewfinder that displays the live view image when the first mode is selected; a communication unit that transmits the live view image to the virtual image display device according to claim 1 when the mode is switched to the second mode; An imaging device comprising:

15. When the mode is switched to the second mode, a marker is displayed on the second display or the viewfinder. The imaging device according to claim 14.

16. A virtual image display device according to claim 1 and an imaging device according to claim 14, Virtual image display system.

17. The processor of the virtual image display device synthesizes a virtual object image onto the live view image, When an image for recording is captured in the second mode, the virtual object image is transmitted to the imaging device; When the imaging device captures an image for recording in a state switched to the second mode, the imaging device records the captured image in association with the virtual object image received from the virtual image display device, or records the captured image in combination with the virtual object image.

17. The virtual image display system of claim 16.

18. A virtual image display system comprising the virtual image display device according to any one of claims 1 to 5 and a server that communicates with the virtual image display device, The server manages a three-dimensional image having three-dimensional information of a real space, The processor, extracting a three-dimensional image corresponding to an imaging area of ​​the first imaging device from the three-dimensional image managed by the server based on first information indicating a three-dimensional position and orientation of the first imaging device; converting the extracted three-dimensional image into a two-dimensional image, and using the two-dimensional image as the live view image; Virtual image display system.

19. A virtual image display system comprising the virtual image display device according to claim 12 and a server that communicates with the virtual image display device, The server manages a three-dimensional image having three-dimensional information of a real space, The processor, extracting a three-dimensional image corresponding to a shooting area of ​​the virtual camera from the three-dimensional image managed by the server based on first information indicating a three-dimensional position and attitude of the virtual camera; converting the extracted three-dimensional image into a two-dimensional image, and using the two-dimensional image as the live view image; Virtual image display system.

20. The server generates and manages the latest three-dimensional images from images taken by a number of cameras, including a multi-view camera, a surveillance camera, a drive recorder, a drone camera, or a mobile terminal camera, arranged in real space.

20. The virtual image display system of claim 18.

21. The server synthesizes a virtual object image with the three-dimensional image.

20. The virtual image display system of claim 18.

22. 1. A method for displaying a virtual image executed by a virtual image display device having a processor and a first display, comprising: receiving a live view image from a first image capture device by the processor; acquiring information used when the processor overlays the live view image on a display area of ​​the first imaging device; the processor transforming the live view image based on the information; displaying the transformed live view image on the first display; A virtual image display method comprising: