Display apparatus, method of controlling the same, and storage

Abstract

To provide an imaging system capable of easily performing work for putting a subject in a frame and work for confirming a picked-up image.SOLUTION: In the image pickup system including the smartphone 100 and the smart glasses 110, the smartphone 100 generates a live view image and transmits the live view image to the smart glasses 110, and transmits operation information in the smartphone 100. The image generation unit 502 of the smart glasses 110 generates a display image to be displayed by the display unit 402 based on the received live view image, and the display control unit 503 causes the display unit 402 to display the display image. The image generation section 502 changes at least one of the transparency, size, and display position of the display image to be generated according to the operation information received from the smartphone 100.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an imaging system and a display device and a control method thereof, and more particularly to an imaging system that allows a user wearing a head-mounted display device to view real scenery through the display device and also view captured images obtained by the imaging device. [Background technology]

[0002] High-angle photography, in which the camera is held higher than eye level, makes it difficult to capture fast-moving subjects. One reason for this is that the line of sight from which the user (photographer) views the live view image displayed on the camera's vari-angle display is different from the line of sight from which the user views the subject. In other words, the user cannot view the live view image and the subject in the same line of sight, making it difficult to fit the subject into the frame.

[0003] One possible solution to this problem is to utilize virtual reality glasses (hereinafter referred to as "AR glasses"), which have become increasingly popular in recent years. For example, one possible method would be to display a live view image from a camera on the display of the AR glasses, allowing the user to capture an image with the camera while checking the live view image on the same axis as the line of sight of the subject.

[0004] A technique for displaying a live view image on AR glasses that superimposes additional information on a transmissive display, such as that described in Patent Document 1, can be applied. For example, a semi-transparent live view image is displayed on the AR glasses as a transmissive display. This allows the user to see the subject of the real scene through the live view image, allowing the user to confirm the subject in the real scene and the subject in the live view image in the same line of sight. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-91055 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when a live-view image is made semi-transparent and displayed on AR glasses, if the transparency of the live-view image is high, it may be difficult for the user to distinguish between the live-view image and the actual scenery image. In this case, for example, the outline of the frame (image capture angle) may become difficult to see, making it difficult to determine whether the subject is captured properly, such as whether the subject is framed as intended. On the other hand, if the transparency of the live-view image is low, the actual scenery behind the live-view image may be hidden. In this case, it may become difficult to visually recognize the subject in the actual scenery, making it difficult to fit the subject within the frame.

[0007] An object of the present invention is to provide an imaging system that allows easy operation of fitting a subject within a frame and easy operation of checking a captured image. [Means for solving the problem]

[0008] The imaging system of the present invention is an imaging system having an imaging device and a head-mounted display device, wherein the imaging device comprises a first image generation means for generating a live view image, an image transmission means for transmitting the live view image to the display device, an operation means for operating the imaging process, and an information transmission means for transmitting operation information of operations performed by the operation means, and the display device comprises a display unit, a receiving means for receiving the live view image and the operation information from the imaging device, a second image generation means for generating a display image to be displayed by the display unit based on the live view image received by the receiving means, and a display control means for displaying the display image generated by the second image generation means on the display unit, and wherein the second image generation means changes the display image to be displayed on the display unit in accordance with the operation information received by the receiving means. [Effects of the Invention]

[0009] According to the present invention, it is possible to realize an imaging system that allows the user to easily perform the tasks of fitting a subject within a frame and checking a captured image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic configuration of an imaging system according to an embodiment. [Figure 2] 1A and 1B are front and rear views of a smartphone that constitutes an imaging system. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of a smartphone. [Figure 4] FIG. 1 is a perspective view showing the appearance of smart glasses that constitute the imaging system. [Figure 5] FIG. 2 is a diagram illustrating functional blocks of smart glasses. [Figure 6] 10 is a flowchart of an imaging process according to the first embodiment, which is performed by a smartphone. [Figure 7] 10 is a flowchart of an imaging process according to the first embodiment performed by smart glasses. [Figure 8]FIG. 2 is a diagram showing an example of an image displayed on the smart glasses during imaging processing according to the first embodiment. [Figure 9] 10 is a flowchart of an imaging process according to the second embodiment performed by smart glasses. [Figure 10] FIG. 10 is a diagram showing an example of an image displayed on the smart glasses during imaging processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] 1 is a diagram showing a schematic configuration of an imaging system according to an embodiment of the present invention. The imaging system includes a smartphone 100 and smart glasses 110, which are connected to each other via a wireless communication path 120 so as to be able to communicate bidirectionally.

[0013] The smartphone 100 is an example of an imaging device. The imaging device constituting the imaging system is not limited to the smartphone 100, but may be another device having an imaging function, such as a digital camera, a video camera, or an unmanned aerial vehicle (drone) equipped with an imaging function. The smart glasses 110 are an example of a head-mounted display device. The head-mounted display device constituting the imaging system is not limited to a glasses-type device like the smart glasses 110, but may be a display device having an image display function other than a glasses-type shape, such as a helmet-type display device worn over the entire head.

[0014] The wireless communication path 120 is a communication path for communication between the smartphone 100 and the smart glasses 110. Wireless communication over the wireless communication path 120 can be performed using direct communication using Bluetooth (registered trademark) or wireless LAN communication via an access point device (not shown). In the imaging system according to this embodiment, information transmitted from the smartphone 100 to the smart glasses 110 includes image information such as an image captured by the smartphone 100 (captured image) and a live view image during imaging. Information transmitted from the smartphone 100 to the smart glasses 110 also includes operation information when an imaging operation is performed on the smartphone 100 (an instruction to store image data and an instruction to prepare for imaging (an instruction to adjust focus, an instruction to adjust exposure, an instruction to adjust white balance, etc.)). In the description of this embodiment, an image transmitted from the smartphone 100 to the smart glasses 110 is referred to as a captured image when there is no need to distinguish between a captured image and a live view image.

[0015] Next, the configuration of the smartphone 100 will be described. Fig. 2(a) is a front view of the smartphone 100, and Fig. 2(b) is a rear view of the smartphone 100. The smartphone 100 has a housing 200. The housing 200 is composed of a front face 201 on the front side, a back face 202 on the rear side, and a side face 203 that connects the front face 201 and the back face 202.

[0016] The front face 201 is provided with a touch screen display 210, buttons 221, an illuminance sensor 231, a proximity sensor 232, a receiver 233, a microphone 234, and an in-camera 242. The back face 202 is provided with a speaker 241 and an out-camera 243. The side face 203 is provided with buttons 222, 223, 224, and a connector 244.

[0017] The touchscreen display 210 is composed of a display 210A and a touchscreen 210B. In this embodiment, the shapes of the display 210A and the touchscreen 210B are substantially rectangular, but are not limited thereto and may be any shape, such as a square or a circle. The display 210A and the touchscreen 210B are arranged overlapping each other in the touchscreen display 210, but are not limited thereto and may be arranged side by side or spaced apart, for example. Furthermore, the display 210A and the touchscreen 210B are arranged so that they overlap substantially the entire area when viewed from a direction perpendicular to their surfaces, but the overlapping manner of the display 210A and the touchscreen 210B is not limited thereto. For example, the display 210A and the touchscreen 210B may be arranged so that they do not overlap in part.

[0018] The display 210A is a display device such as a liquid crystal display, an organic electroluminescence (EL) display, or an inorganic electroluminescence (EL) display. Characters, images, symbols, figures, etc. are displayed on the display 210A. The touch screen 210B detects contact of a finger, pen, stylus pen, etc. (hereinafter referred to as a "touch object") with the touch screen 210B and detects the contact position of the touch object. Detection of the presence or absence of contact of the touch object with the touch screen 210B and the contact position can be performed using any method selected from well-known methods such as a capacitive resistive film method, a surface acoustic wave method (ultrasonic method), an infrared method, an electromagnetic induction method, and a load detection method. For convenience, in the following description, it is assumed that a user touches the touch screen 210B with their finger to perform various operations on the smartphone 100.

[0019] Fig. 3 is a diagram illustrating functional blocks of the smartphone 100. As shown in Fig. 2, the smartphone 100 includes a touchscreen display 210, an illuminance sensor 231, a proximity sensor 232, a receiver 233, a microphone 234, a speaker 241, an in-camera 242, an out-camera 243, and a connector 244. The smartphone 100 also includes a controller 300, a storage 310, a button 320, a communication unit 341, an acceleration sensor 331, a direction sensor 332, and a gyroscope 333. Note that, among the components of the smartphone 100 shown in the block diagram of Fig. 3, those that have already been described with reference to Fig. 2 will not be described here.

[0020] The controller 300 detects operations (hereinafter referred to as "gestures") on the smartphone 100. Specifically, the controller 300 detects gestures on the touch screen 210B (touch screen display 210) in cooperation with the touch screen 210B. The buttons 320 include buttons 221 to 224 shown in FIG. 2. A user of the smartphone 100 can issue various commands assigned to the buttons 320 to the controller 300 by operating the buttons 320. The controller 300 detects operations on the buttons 320 in cooperation with the buttons 320. Examples of operations on the buttons 320 include, but are not limited to, a click, a double click, a triple click, a push, and a multi-push. The button 221 is, for example, a home button. The button 222 is, for example, a power on / off button of the smartphone 100. The buttons 223 and 224 are, for example, volume buttons.

[0021] The illuminance sensor 231 detects the illuminance (light intensity, brightness, or luminance) of ambient light around the smartphone 100. The illuminance of ambient light detected by the illuminance sensor 231 is used, for example, to adjust the brightness of the display 210A. The proximity sensor 232 detects the presence of an object near the smartphone 100 in a non-contact manner based on a change in a magnetic field or a change in the return time of a reflected ultrasonic wave. For example, the proximity sensor 232 detects that the touchscreen display 210 has been brought close to the user's face. The illuminance sensor 231 and the proximity sensor 232 may be configured as a single sensor. The illuminance sensor 231 can also detect the presence of an object near the smartphone 100, that is, the illuminance sensor 231 can also be used as a proximity sensor.

[0022] The communication unit 341 performs wireless communication with external devices through the wireless communication path 120 in accordance with various wireless communication standards. The smart glasses 110 are one of the external devices that perform wireless communication with the communication unit 341 of the smartphone 100. In this embodiment, as will be described later, the communication unit 341 functions as an image transmitting means and an information transmitting means that transmits a live view image generated by the smartphone 100 and operation information on the smartphone 100 to the smart glasses 110.

[0023] Communication methods supported by the communication unit 341 include cellular phone communication standards such as 3G, 4G, and 5G, and representative communication standards include LTE, W-CDMA, CDMA2000, PDC, and PHS. Other wireless communication standards include WiMAX (registered trademark), IEEE802.11, IrDA, and NFC. The communication unit 341 may support one or more of the above-mentioned communication standards.

[0024] The receiver 233 and the speaker 241 are audio output means that output an audio signal transmitted from the controller 300 as sound. The receiver 233 outputs, for example, the voice of the other party during a call. The speaker 241 outputs a ringtone, music, etc. One of the receiver 233 and the speaker 241 may also serve the function of the other. The microphone 234 is an audio input unit that converts the user's voice, environmental sounds (sounds around the smartphone 100), etc. into an audio signal and transmits it to the controller 300.

[0025] The storage 310 is a storage unit that stores programs and data. The storage 310 is also used as a work area that temporarily stores processing results of the controller 300. The storage 310 may include any non-transitory storage medium such as a semiconductor storage medium or a magnetic storage medium, or may include multiple types of storage media. The storage 310 may also include a combination of a portable storage medium such as a memory card, optical disk, or magneto-optical disk, and a storage medium reader. Furthermore, the storage 310 may include a storage device used as a temporary storage area, such as RAM.

[0026] The programs stored in storage 310 include applications that run in the foreground or background and control programs that support the operation of the applications. The applications, for example, cause images and the like to be displayed on display 210A and cause controller 300 to execute processing in accordance with gestures detected via touch screen 210B. The control programs are, for example, an OS.

[0027] The application and control program may be installed in the storage 310 via wireless communication using the communication unit 341 or via a non-transitory storage medium. Furthermore, all or part of the application and control program may be stored in a non-transitory storage medium readable by a reading device included in the storage 310, or in a non-transitory storage medium readable by a reading device connected to the connector 244. Examples of non-transitory storage media include, but are not limited to, optical disks, magneto-optical disks, magnetic storage media, memory cards, and solid-state storage media. The storage 310 stores, for example, a control program 311, a face recognition program 312, contact data 313, and setting data 314.

[0028] The control program 311 provides various control functions for operating the smartphone 100. The control program 311 realizes calls (telephone functions) by controlling, for example, the communication unit 341, the receiver 233, and the microphone 234. The functions provided by the control program 311 include a function for controlling the change of information displayed on the display 210A in accordance with a gesture detected via the touch screen 210B. Note that the functions provided by the control program 311 may be used in combination with functions provided by other programs, such as a sound quality adjustment program and a telephone application.

[0029] The face recognition program 312 provides functions such as extracting a face image included in a live view image captured by the in-camera 242 and the out-camera 243, and recognizing a face from the extracted face image. The face recognition program 312 is used, for example, to identify whether the captured person is the owner. The face recognition function of the face recognition program 312 is realized, for example, by performing pattern matching between the facial features shown in the extracted face image and the facial features of the registered owner. When the captured image includes multiple face images, the face recognition function includes a function to determine whether the captured image includes a face image of a person other than the owner.

[0030] The contact data 313 includes information for the user of the smartphone 100 to contact other people and information on a predetermined registration destination in case of an emergency. The predetermined registration destination information includes, for example, an email address and a telephone number. The contact data 313 may also include various personal information such as image data, an address, a place of employment, and a date of birth. The setting data 314 includes information on various settings related to the operation of the smartphone 100.

[0031] The controller 300 is, for example, a processing device such as a CPU, SoC, MCU, or FPGA, but is not limited to these, and may also include multiple processing devices. The controller 300 comprehensively controls the operation of the smartphone 100 to realize various functions. Specifically, the controller 300 executes instructions included in a program stored in the storage 310 while referencing data stored in the storage 310 as needed, and controls and operates predetermined functional units in accordance with the data and instructions. Here, the functional units include, for example, a display 210A, a communication unit 341, a receiver 233, and a speaker 241, but are not limited to these.

[0032] Furthermore, the controller 300 appropriately changes control in accordance with the detection results of the detection unit. The detection unit includes, but is not limited to, the touch screen 210B, the button 320, the illuminance sensor 231, the proximity sensor 232, the microphone 234, the in-camera 242, the out-camera 243, the acceleration sensor 331, the orientation sensor 332, and the gyroscope 333. For example, by executing the control program 311, the controller 300 also performs control such as changing the information displayed on the display 210A in accordance with a gesture detected via the touch screen 210B.

[0033] The in-camera 242 is a camera that captures an image of an object facing the front face 201. The out-camera 243 is a camera that captures an image of an object facing the back face 202. The connector 244 is a terminal for connecting an external device, and may be, for example, a general-purpose terminal such as a USB terminal or an earphone / microphone connector, or a dedicated terminal such as a dock connector. Examples of external devices that can be connected to the connector 244 include, but are not limited to, external storage, speakers, and communication devices.

[0034] The acceleration sensor 331 detects the direction and magnitude of acceleration acting on the smartphone 100. The orientation sensor 332 detects the direction of geomagnetic field. The gyroscope 333 detects the angle and angular velocity of the smartphone 100. The detection results of the acceleration sensor 331, orientation sensor 332, and gyroscope 333 are used in combination to detect changes in the position and attitude of the smartphone 100.

[0035] Next, the configuration of the smartglasses 110 will be described. FIG. 4 is an external perspective view of the smartglasses 110. The smartglasses 110 include a frame 401, a right display unit 402R, a left display unit 402L, a right image generation unit 403R, a left image generation unit 403L, and a wireless communication unit 410. In the following description, the right display unit 402R and the left display unit 402L will be collectively referred to as the "display unit 402," and the right image generation unit 403R and the left image generation unit 403L will be collectively referred to as the "image generation unit 403." Furthermore, the "up, down, left, and right" of the smartglasses 110 refer to the up, down, left, and right from the user's perspective when the user is wearing the smartglasses 110. Therefore, when the user wears the smartglasses 110, the right display unit 402R is located in front of the user's right eye, and the left display unit 402L is located in front of the user's left eye.

[0036] The smart glasses 110 are configured as a glasses-type head-mounted display (HMD). That is, the smart glasses 110 are configured as an optically transmissive HMD that allows the user to view an image displayed on the display unit 402 (hereinafter referred to as a "display image") and simultaneously view the outside world directly.

[0037] The frame 401 includes a portion that supports the edge of the display unit 402, which corresponds to the lens of glasses, and a portion that corresponds to the temples of glasses. In the smart glasses 110, the right image generation unit 403R and the left image generation unit 403L also form part of the frame 401.

[0038] The display unit 402 is supported by the frame 401 and placed in front of the user's eyes, and displays an image generated by the image generation unit 403. The display unit 402 is made of a transparent material and does not obstruct the user's view when no display image is displayed. The transparency of the display unit 402 can be set to any desired transparency, for example, by intentionally lowering the transparency like sunglasses, within a range that can achieve the effect of display control of the display image, which will be described later. When the user wears the smart glasses 110, the right display unit 402R is placed in front of the user's right eye, and the left display unit 402L is placed in front of the user's left eye. The display unit 402 may be an integrated display placed in front of both of the user's eyes.

[0039] Furthermore, the display unit 402 may be disposed in a portion corresponding to the temple of the glasses, rather than in a portion corresponding to the lens of the glasses. In this case, a half mirror or the like is disposed in the portion corresponding to the lens of the glasses. Furthermore, instead of a display unit of the present embodiment that forms an image on the retina of the user's eye, a display unit that draws an image on the retina by scanning the retina may be used.

[0040] The image generation unit 403 is supported by the frame 401, generates an image to be displayed on the display unit 402, supplies the image to the display unit 402, and displays it on the display unit 402. The image generation unit 403 is communicably connected to the smartphone 100 via the wireless communication unit 410, and acquires images captured by the smartphone 100. The right image generation unit 403R generates an image to be displayed on the right display unit 402R and supplies the image to the right display unit 402R, and the left image generation unit 403L generates an image to be displayed on the left display unit 402L and supplies the image to the left display unit 402L. The image generation unit 403 does not need to be divided into left and right (two) units as in this embodiment, and may be configured and arranged as an integrated unit.

[0041] The wireless communication unit 410 includes an antenna (a protruding portion from the top surface of the right image generation unit 403R) that transmits radio waves to the smartphone 100, and receives data such as images captured by the smartphone 100 and various types of operation information. The antenna of the wireless communication unit 410 may be disposed inside the frame 401 or the image generation unit 403.

[0042] 5 is a diagram illustrating functional blocks of the smart glasses 110. The image generation unit 403 has an image acquisition unit 501, an image generation unit 502, and a display control unit 503. The image acquisition unit 501, the image generation unit 502, and the display control unit 503 are connected to each other so as to be able to exchange data with each other, and the display control unit 503 is connected to the display unit 402 (right display unit 402R, left display unit 402L).

[0043] The image generation unit 403 is composed of a CPU, a ROM, a RAM, etc., and the CPU executes a predetermined program stored in the ROM to realize the functions of the image acquisition unit 501, the image generation unit 502, and the display control unit 503. In other words, in the smart glasses 110, the image generation unit 403 plays the role of a controller that performs overall control of the smart glasses 110.

[0044] The image acquisition unit 501 acquires an image (original image data) that is the source of a display image displayed on the display unit 402 from the wireless communication unit 410. The original image is an image generated by an application of the smartphone 100 or an image acquired from another device. The original image includes, for example, display information generated based on a captured image captured by the smartphone 100.

[0045] Image generation unit 502 generates a display image from an original image. The display image includes a non-display area, which is an area where an image is not displayed when the display image is displayed on display unit 402, and a display area, which is an area where an image is displayed when the display image is displayed on display unit 402. Image generation unit 502 superimposes a mask layer including an image mask on the original image, so that the area of ​​the original image corresponding to the image mask can be set as a non-display area, and the area of ​​the original image not corresponding to the image mask can be set as a display area.

[0046] The image generation unit 502 can also perform image processing on the original image. Examples of image processing include, but are not limited to, processing to change the transparency of the original image and processing to change the image size of the original image. The transparency change processing can use known transparency change processing such as processing to change the brightness or luminance level of the original image and processing to thin out specific pixels so that the background is displayed transparently. The image generation unit 502 supplies the generated display image to the display control unit 503.

[0047] The display control unit 503 supplies the display image to the display unit 402, causing it to be displayed. Because the display unit 402 is a transmissive display, no image is displayed in the non-display area of ​​the display image, which corresponds to the image mask, and therefore the user can directly view the outside world through the non-display area. On the other hand, because the display area includes the original image, the user can view the display information included in the original image. In other words, the user can view the outside world through the non-display area and at the same time check the display information displayed on the display unit 402.

[0048] First Embodiment Next, an imaging processing flow according to the first embodiment of the imaging system will be described. Fig. 6 is a flowchart of processing according to the first embodiment performed by the smartphone 100 when capturing an image using the imaging system. Fig. 7 is a flowchart of processing according to the first embodiment performed by the smart glasses 110 when capturing an image using the imaging system.

[0049] Each process (step) indicated by an S number in the flowchart of Fig. 6 is realized by the controller 300 executing a predetermined program stored in the storage 310 and comprehensively controlling the operation of each unit of the smartphone 100. Also, each process indicated by an S number in the flowchart of Fig. 7 is executed by the image generation unit 403. That is, in the image generation unit 403, the CPU executes a predetermined program stored in the ROM and comprehensively controls the operation of each unit of the smart glasses 110, thereby realizing the process.

[0050] 6 and 7 show the flow of still image capturing processing, video capturing processing, and both still image and video capturing processing. The flowcharts in Fig. 6 and 7 also show processing after the pairing processing between the smartphone 100 and the smart glasses 110 is completed and an instruction to start the capturing processing is issued from the smartphone 100 side or the smart glasses 110 side.

[0051] First, the processing in the smartphone 100 (FIG. 6) will be described. In S601, the controller 300 acquires a captured image (video) of a subject, such as an object or a person, using the in-camera 242 or the out-camera 243, and generates a live view image during the capture. In S602, the controller transmits the generated live view image to the smart glasses 110 via the communication unit 341. Note that the live view image transmitted from the smartphone 100 to the smart glasses 110 by the processing in S602 serves as a criterion for the processing in S703 in the smart glasses 110.

[0052] Thereafter, in S603, controller 300 determines whether or not a touch operation has been performed on touch screen display 210 (touch screen 210B). Specifically, the touch operation here is a touch-on operation on the subject to be focused on. If controller 300 determines that a touch operation has been performed (YES in S603), the process proceeds to S604; if it determines that no touch operation has been performed (NO in S603), the process proceeds to S606.

[0053] When acquiring a live view image, the autofocus function of the smartphone 100 continuously performs a process of focusing on the main subject determined by the controller 300. The touch operation in S603 may be directed to the main subject or may be directed to a subject other than the main subject.

[0054] In S604, the controller 300 adjusts the focus of the subject displayed at the position of the touch screen display 210 where the touch operation was performed. The focus adjustment is performed by the controller 300 performing known control such as adjusting the aperture or focal length of the lens of the camera in use (the in-camera 242 or the out-camera 243) to adjust the depth of field. In S605, the controller 300 transmits operation information indicating that the focus adjustment was performed in S604 to the smart glasses 110. Note that the operation information transmitted to the smart glasses 110 by the processing of S605 is used by the smart glasses 110 in S704 and subsequent processing as needed.

[0055] In S606, controller 300 determines whether the record instruction button has been pressed. Here, it is assumed that the function of the record instruction button is assigned to button 221. Note that the process proceeds directly from S603 to S606 when the position or orientation of in-camera 242 or out-camera 243 relative to the subject is adjusted with respect to the subject without adjusting the focus, and the process of fitting the subject within the frame is continued. If controller 300 determines that the record instruction button has been pressed (YES in S606), the process proceeds to S607, and if it determines that the record instruction button has not been pressed (NO in S606), the process proceeds to S609.

[0056] In S607, the controller 300 performs storage processing of the live view image being sent to the smartphone 100. The storage processing of the live view image is performed by storing the live view image data being sent to the smartphone 100 in a data storage area of ​​the storage 310. When storing a still image, only the frame image at the timing when the button 221 is pressed is stored. When storing a moving image, the frame images for the period from the timing when the recording instruction button is pressed to the timing when an instruction to stop capturing is issued are stored.

[0057] In S608, the controller 300 transmits the captured image stored in S607 to the smart glasses 110. Note that the captured image transmitted to the smart glasses 110 by the processing of S608 is used in the smart glasses 110 for the processing of S704 and thereafter.

[0058] In S609, the controller 300 determines whether an instruction to end image capture has been received. The instruction to end image capture may be given, for example, by pressing the button 320 on the smartphone 100 or by touching a button screen displayed on the touchscreen display 210, but is not limited to these. If the controller 300 determines that an instruction to end image capture has not been received (NO in S609), the controller 300 returns the process to S601. That is, the controller 300 continues adjusting the position and orientation of the in-camera 242 or the out-camera 243 relative to the subject to fit the subject within the frame. On the other hand, if the controller 300 determines that an instruction to end image capture has been received (YES in S609), the controller 300 proceeds to S610.

[0059] In S610, the controller 300 transmits operation information indicating that an operation to end image capture has been performed to the smart glasses 110, and then terminates this processing. Note that the operation information transmitted to the smart glasses 110 by the processing of S610 is used in the processing of S709 in the smart glasses 110.

[0060] Next, the processing in the smart glasses 110 (FIG. 7) will be described. In S701, the image generation unit 403 waits in a state where it is able to receive a live view image from the smartphone 100. In S702, the image generation unit 403 determines whether or not it has received a live view image from the smartphone 100. If the image generation unit 403 determines that it has not received a live view image (NO in S702), it returns the processing to S701, and if it determines that it has received a live view image (YES in S702), it proceeds to S703.

[0061] In S703, the image generation unit 403 determines whether or not focus adjustment operation information has been received from the smartphone 100. If the image generation unit 403 determines that focus adjustment operation information has been received (YES in S703), the process proceeds to S704, and if the image generation unit 403 determines that focus adjustment operation information has not been received (NO in S703), the process proceeds to S705.

[0062] In S704 and S705, a process of changing the transparency of the received live view image is performed. Specifically, in S704, the image generation unit 403 functions as the image generation unit 502, changes the transparency of the live view image to a lower level, and then proceeds to S706. On the other hand, in S705, the image generation unit 403 functions as the image generation unit 502, changes the transparency of the live view image to a higher level, and then proceeds to S706.

[0063] Note that when a live-view image with low transparency is displayed on the display unit 402, it is difficult for the user to view the actual scenery in the area where the live-view image is displayed. In other words, the area where the live-view image with low transparency is displayed prevents the user from viewing the actual scenery. On the other hand, when a live-view image with high transparency is displayed on the display unit 402, the user can view the actual scenery through the area where the live-view image is displayed. In other words, the user can view the actual scenery superimposed on the live-view image.

[0064] In S706, the image generation unit 403 functions as the display control unit 503, and displays the live view image whose transparency has been changed in S704 or S705 on the display unit 402. On the display unit 402, the live view image is displayed near the center of the display unit 402 so as to straddle the right display unit 402R and the left display unit 402L, as will be described later with reference to FIG.

[0065] In S707, the image generation unit 403 determines whether or not a captured image stored in the smartphone 100 has been received from the smartphone 100. If the image generation unit 403 determines that a stored captured image has been received (YES in S707), the process proceeds to S708, and if the image generation unit 403 determines that a stored captured image has not been received (NO in S707), the process proceeds to S709.

[0066] In S708, the image generation unit 403 displays the captured image stored in the smartphone 100 on the display unit 402. In S709, the image generation unit 403 determines whether or not operation information to end shooting has been received from the smartphone 100. If the image generation unit 403 determines that operation information to end shooting has been received (YES in S709), it ends this processing and ends the display of the live view image and the captured image on the display unit 402. On the other hand, if the image generation unit 403 determines that operation information to end shooting has not been received (NO in S709), it returns the processing to S701 and continues displaying the live view image, etc. on the display unit 402.

[0067] Next, examples of image display on the smartphone 100 and the smart glasses 110 during processing according to the flowcharts of FIGS. 6 and 7 will be described.

[0068] Fig. 8(a) is a diagram showing an example of an image displayed on the smart glasses 110 when a touch operation in S603 has not been performed on the touch screen display 210 of the smartphone 100 (determination in S703 is 'NO'). Fig. 8(b) is a diagram showing an example of an image displayed on the smart glasses 110 when a touch operation in S603 has been performed on the touch screen display 210 of the smartphone 100 (determination in S703 is 'YES'). Note that, as described above, the touch operation in S603 is a touch operation for adjusting the focus of a subject in a live view image.

[0069] A live view display image 800 shown in FIG. 8(a) is a display image of a live view image captured by the smartphone 100 and displayed on the display unit 402. In FIG. 8(a), no focus adjustment operation has been performed on a specific subject (focusing has been performed using the autofocus function). Therefore, the live view display image 800 that has been processed to increase transparency by the processing of S705 is displayed on the display unit 402. In this case, the user can view the actual scenery in the entire area of ​​the display unit 402. In other words, in the area where the live view display image 800 is displayed, the user can view the actual scenery through the live view display image 800.

[0070] 8(a) schematically shows a state in which a user can visually recognize a first object 801A, which is an object (person or thing) around a subject captured by smartphone 100, in a real scene through live view display image 800. The user can simultaneously view both live view display image 800 and first object 801A. This allows the user to easily operate smartphone 100 to fit a desired subject (object) within the frame while viewing the surroundings of the moving subject.

[0071] The user checks the live view display image 800 displayed on the display unit 402 of the smart glasses 110, and if the user determines that the subject to be recorded is within the frame, the user operates the smartphone 100 to perform a focus adjustment operation on the subject to be recorded. Specifically, as described in the processing of S603, the user performs a touch-on operation on the touch screen display 210 at the position where the subject to be recorded is displayed.

[0072] The user checks whether or not to record the live view display image 800 after focus adjustment as an image. If the highly transparent live view display image 800 remains displayed, the first object 801A located behind the live view display image 800 will be visible through the image. As a result, it becomes difficult to distinguish whether the first object 801A is part of the real scene or the live view display image 800, which may result in the user mistaking the subject to be recorded. For example, it may become difficult to distinguish whether the first object 801A or the second object 801B is to be recorded.

[0073] Therefore, when the smart glasses 110 receive focus adjustment operation information from the smartphone 100 as a result of focus adjustment on a subject, in S704, the smart glasses 110 generate a live view display image 810 by changing the live view display image 800 to have a lower transparency. In this way, as shown in FIG. 8(b), the live view display image 810 that has been processed to have a lower transparency is displayed on the display unit 402. In the live view display image 810, almost only the live view display image 810 is displayed in the area where the live view display image 810 is displayed on the display unit 402, making it difficult to confirm the actual scenery. Therefore, the user can confirm only the live view display image 810 without being affected by objects surrounding the subject to be recorded, which makes it easy to perform a final confirmation of the subject to be recorded in the frame before recording.

[0074] If the user checks the live view display image 810 displayed on the display unit 402 of the smart glasses 110 and determines to record the subject image as a still image or a video, the user performs an operation (S607) on the smartphone 100 to record the captured image. Specifically, as described above, the user presses the button 221. Note that the function of instructing the start of recording is not limited to the button 211, and may be assigned to the other buttons 222 to 224. Also, the smartphone 100 may be configured to start recording when an operation icon displayed on the touch screen display 210 is touched on.

[0075] Once the captured image is stored, as described above, the captured image is transmitted to the smart glasses 110 by the processing of S608, and the captured image is displayed on the display unit 402 by the processing of S708. At this time, the captured image may be displayed in the area where the live view display images 800, 810 are displayed, or may be displayed in an area different from this. Furthermore, when recording a video, it is sufficient for the user to be able to determine the start and end timings of recording and whether or not recording is in progress. Therefore, it is also possible to simply display a mark on the display unit 402 in S708 indicating that the captured image is being recorded, without transmitting the recorded captured image (S608).

[0076] As described above, in the first embodiment, the imaging system controls the transparency of the live view image displayed on the smart glasses 110 so that the transparency is high before the focus adjustment operation and low after the focus adjustment operation. This makes it easy to fit a moving subject within the frame and to perform a final check of the captured image after focus adjustment and before storage. An imaging method using such an imaging system is particularly useful in shooting scenes such as high-angle shooting, where the user's line of sight from which they view the live view image displayed on the smartphone 100 is separated from the user's line of sight from which they view the subject.

[0077] Second Embodiment Next, an imaging processing flow according to a second embodiment of the imaging system will be described. In the first embodiment, the transparency of the live view image displayed on the display unit 402 of the smart glasses 110 is controlled to be changed before and after a focus adjustment operation. In contrast, in the second embodiment, the size and display position of the live view image displayed on the display unit 402 of the smart glasses 110 are controlled to be changed before and after a focus adjustment operation. Note that, in the control according to the second embodiment, the control executed by the smartphone 100 is the same as the control executed by the smartphone 100 in the first embodiment (FIG. 6), and therefore a description thereof will be omitted.

[0078] Fig. 9 is a flowchart of the processing performed by the smart glasses 110 when performing imaging processing in the imaging system. Each process indicated by an S number in the flowchart in Fig. 9 is realized by the image generation unit 403 of the smart glasses 110 executing a predetermined program and comprehensively controlling the operation of each part of the smart glasses 110.

[0079] The processes of S901 to S903 are the same as the processes of S701 to S703 in the first embodiment, and therefore description thereof will be omitted here. In S904 and S905, the image generation unit 403 performs processing to change the display size and display position of the received live-view image. Specifically, in S904, the image generation unit 403 functions as the image generation section 502, generates a live-view image with a small display size, and sets the display position of the generated live-view image to an end (left end or right end) of the display section 402. On the other hand, in S905, the image generation unit 403 functions as the image generation section 502, generates a live-view image with a larger display size than the live-view image generated in S904, and sets the display position of the generated live-view image to the center of the display section 402. Setting the display position to the center of the display section 402 brings the display position of the live-view image closer to the center of the user's field of view.

[0080] The details of the "small display size" will be described later. The live view images generated in S904 and S905 are generated with reduced transparency in accordance with the processing in S705.

[0081] In S906, the image generation unit 403 displays a live view image on the display unit 402 in accordance with the settings in S904 or S905. The processes of S907 to S909 are the same as the processes of S706 to S709 in the first embodiment, and therefore will not be described here.

[0082] Next, an example of an image displayed on the smart glasses 110 during processing according to the flowcharts of Fig. 6 and Fig. 9 will be described. Fig. 10(a) is a diagram showing an example of an image displayed on the smart glasses 110 when a touch operation in S603 has not been performed on the touch screen display 210 of the smartphone 100 (determination in S903 is 'NO'). Fig. 10(b) is a diagram showing an example of an image displayed on the smart glasses 110 when a touch operation in S603 has been performed on the touch screen display 210 of the smartphone 100 (determination in S903 is 'YES'). Note that, as described above, the touch operation in S603 is a touch operation for adjusting the focus of a subject to be recorded in a live view image.

[0083] 10(a), before the focus adjustment operation, a small live view display image 1000 is displayed at the edge of the display unit 402. The vertical size of the live view display image 1000 is, for example, ½ or less, and preferably ⅓ or less, of the height and width of the display unit 402. On the other hand, so that the subject in the live view display image 1000 can be easily confirmed, the vertical size of the live view display image 1000 is preferably ¼ or more of the height and width of the display unit 402. The horizontal size of the live view display image 1000 is adjusted in accordance with the aspect ratio of the image captured by the smartphone 100.

[0084] In the second embodiment, the live view display image 1000 before the focus adjustment operation has a small display size and is displayed at the edge of the display unit 402. Therefore, the user can view the entire real scene including the first object 801A without it being obstructed by the live view display image 1000. Note that by setting the display size of the live view display image 1000 to a small size, the display image can be clearly viewed. In this way, the user can check the live view display image 1000 and the real scene including the first object 801A and the second object 801B, which makes it easy to fit the moving subject within the frame while viewing the surroundings of the subject.

[0085] The live view display image 1010 after the focus adjustment operation shown in Fig. 10(b) is equivalent to the live view display image 810 in Fig. 8(b) in terms of display size, display position, and transparency. In other words, the live view display image 1010, which has a lower transparency, is displayed near the center of the display unit 402 at a larger display size than the live view display image 1000. Therefore, in the display area of ​​the live view display image 1010 on the display unit 402, the actual scenery is hidden by the live view display image 1010 and is essentially invisible. This makes it possible to check only the live view display image 1010 without being affected by the first object 801A behind the live view display image 1010, facilitating a final check of the subject to be recorded in the frame before recording.

[0086] As described above, according to the second embodiment, the size and position of the live view image displayed on the smart glasses 110 are controlled so that before the focus adjustment operation, the image is displayed small and on the edge, and after the focus adjustment operation, the image is displayed large and near the center. This makes it easier to fit a moving subject within the frame and to make a final check of the captured image after focus adjustment and before storing it.

[0087] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0088] For example, in the above embodiment, a smartphone is used as the imaging device, but the imaging device is not limited to this and may be a digital still camera or a digital video camera (hereinafter referred to as a "digital camera"). In the case of a digital camera, for example, an operation related to shooting preparation instructions such as focus adjustment may be replaced by a half-press of the release button, and an operation related to shooting instructions such as recording may be replaced by a full-press of the release button. On the other hand, if the digital camera is provided with a display equivalent to the touchscreen display 210 of the smartphone 100, the digital camera can be handled in the same way as the smartphone 100. Not limited to live view images, information on the imaging conditions of the digital camera (aperture value, ISO value, shutter speed, etc.) may also be configured to be displayed on the display unit 402 of the smart glasses 110 in an area that does not interfere with the visibility of the actual scenery and the displayed image.

[0089] In the above embodiment, a focus adjustment instruction is taken as an example of an operation that triggers a change in the display format (transparency, display position, display size) of a captured image such as a live view image displayed on the smart glasses. However, the operation that triggers the change is not limited to a focus adjustment instruction, and may be other instructions to prepare for shooting or an instruction to store image data.

[0090] Furthermore, in the above embodiment, a system in which the display format of a captured image is changed by an image generation unit on the smart glasses side has been taken as an example. However, a configuration in which the display format of a captured image is changed by an image generation unit on the smartphone side may also be adopted. In this configuration, it is not necessary to transmit operation information from the smartphone to the smart glasses.

[0091] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0092] 100 smartphones 110 Smart Glasses 300 Controller 341 Communication Unit 402 Display section 403 Image Generation Unit 410 Radio Communication Department 501 Image acquisition unit 502 Image generation unit 503 Display control unit

Claims

1. An imaging system having an imaging device and a head-mounted display device, The imaging device is a first image generating means for generating a live view image; an image transmission means for transmitting the live view image to the display device; an operation means for operating the imaging process; an information transmitting means for transmitting operation information of the operation by the operating means, The display device includes: A display unit; a receiving means for receiving the live view image and the operation information from the imaging device; a second image generating means for generating a display image to be displayed on the display unit based on the live view image received by the receiving means; a display control means for causing the display image generated by the second image generation means to be displayed on the display unit, The imaging system is characterized in that the second image generating means changes the display image to be displayed on the display unit in accordance with the operation information received by the receiving means.

2. 2. The imaging system according to claim 1, wherein the second image generating means changes the transparency of the display image in accordance with the operation information.

3. 3. The imaging system according to claim 1, wherein the second image generating means changes a display position of the display image in accordance with the operation information.

4. 4. The imaging system according to claim 1, wherein the second image generating means changes a display size of the display image in accordance with the operation information.

5. 3. The imaging system according to claim 2, wherein the second image generating means reduces the transparency of the display image in response to the operation information.

6. 4. The imaging system according to claim 3, wherein the second image generating means moves the display position of the display image closer to the center of the user's field of view in response to the operation information.

7. 5. The imaging system according to claim 4, wherein the second image generating means increases the display size of the display image in accordance with the operation information.

8. 8. The imaging system according to claim 2, wherein the second image generating means changes the transparency, display position, or display size of a live view image included in the display image in accordance with the operation information.

9. 9. The imaging system according to claim 1, wherein the operation by the operation unit includes an operation relating to instructions for preparation for imaging.

10. 9. The imaging system according to claim 1, wherein the operation by the operation means includes an operation relating to an instruction to store an image.

11. An imaging system having an imaging device and a head-mounted display device, The imaging device is a first image generating means for generating a captured image; an image transmitting means for transmitting the captured image to the display device, The display device includes: receiving means for receiving the captured image from the imaging device; a display means for displaying the captured image received by the receiving means to the user's eyes, wherein the display means reduces the transparency of the captured image displayed to the user's eyes when the user inputs an instruction to prepare for photography by the imaging device or an instruction to store an image by the imaging device.

12. A head-mounted display device, A display unit; a receiving means for receiving, from an external imaging device, a live view image being captured by the imaging device and operation information of the imaging device; an image generating means for generating a display image to be displayed on the display unit based on the live view image received by the receiving means; a display control means for causing the display image generated by the image generation means to be displayed on the display unit, The display device is characterized in that the image generating means changes the display image to be displayed on the display unit in accordance with the operation information.

13. A method for controlling a head-mounted display device, comprising: a receiving step of receiving, from an external imaging device, a live view image being captured by the imaging device and operation information of the imaging device; an image generating step of generating a display image to be displayed on a display unit based on the live view image received in the receiving step; a display step of displaying the display image generated in the image generation step on the display unit, The method for controlling a display device, wherein the image generating step changes the display image to be displayed on the display unit in accordance with the operation information.

Images