Information processing device and information processing method

The information processing device accurately controls imaging device operations by combining gesture detection from a head-mounted device with imaging device state information, addressing the issue of unintended movements and enhancing user experience through precise gesture recognition.

WO2024232254A1PCT designated stage expired Publication Date: 2024-11-14SONY GROUP CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015843
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-04-23
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing information processing systems face challenges in accurately controlling operations on head-mounted devices, such as imaging devices, due to unintended movements caused by the user's head and the device moving together, leading to potential erroneous operations.

Method used

An information processing device and method that utilizes a head-mounted device to detect gestures and combines this information with the state of the imaging device, allowing for precise control by canceling operations when the device and head movements match, and executing gestures when they do not, thereby reducing erroneous operations.

Benefits of technology

This approach ensures accurate operation control based on the user's intended gestures, minimizing errors and enhancing the user experience by differentiating between intended and unintended movements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015843_14112024_PF_FP_ABST
    Figure JP2024015843_14112024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing device comprises a control unit that controls at least the operation of an imaging device on the basis of gesture information indicating a gesture detected by a head-mounted device attached to the head of a user and imaging device state information indicating the state of the imaging device.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device and information processing method

[0001] The present technology relates to an information processing device and an information processing method in which, for example, an operation of a head-mounted device is controlled based on gestures.

[0002] Conventionally, a system has been proposed in which a first earphone and a second earphone each detect a physical quantity dependent on neck movement, identify the neck movement based on the physical quantity detected by each earphone, and control a device in accordance with the identified neck movement (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2016-21605

[0004] However, when a user performs a gesture operation on an imaging device by moving his or her head, the imaging device may be operated based on a movement that is not intended by the user. For example, if the user moves the imaging device in a predetermined direction and also moves his or her head in accordance with the movement of the imaging device, the operation may be controlled based on the movement of the head, even though the user simply moved his or her head in accordance with the movement of the imaging device.

[0005] The present technology has been developed in view of the above-mentioned problems, and aims to accurately control operations based on the state of the user's head.

[0006] The information processing device according to the present technology includes a control unit that controls at least the operation of an imaging device based on gesture information indicating a gesture detected by a head-mounted device mounted on a user's head and imaging device status information indicating a status of the imaging device.

[0007] 1 is a diagram showing a configuration of an information processing system according to an embodiment of the present invention. FIG. 1 is a diagram showing an external appearance of an imaging device. FIG. 2 is a diagram showing an internal configuration of an imaging device. FIG. 3 is a diagram showing an internal configuration of headphones. FIG. 4 is a diagram showing the functional configurations of a camera control unit and a headphone control unit. FIG. 5 is a diagram showing a state in which a user is capturing an image with an imaging device while wearing headphones. FIG. 6 is a diagram showing a flowchart of control processing in a first embodiment. FIG. 7 is a diagram showing an example of operation of the imaging device and headphones based on a headphone gesture. FIG. 8 is a diagram showing an example of notification of an operation based on a gesture. FIG. 9 is a diagram showing an example of a camera GUI. FIG. 10 is a diagram showing an example of a headphone GUI. FIG. 11 is a diagram showing a flowchart of control processing in a first modified example of the first embodiment. FIG. 12 is a diagram showing a flowchart of control processing in a second modified example of the first embodiment. FIG. 13 is a diagram showing an example of operation of the imaging device for combinations of headphone gestures and camera states. FIG. 14 is a diagram explaining the orientation of the imaging device when capturing an image in landscape orientation. FIG. 15 is a diagram showing a flowchart of control processing in a second embodiment. FIG. 16 is a diagram showing an example of the orientation of a screen for headphone gestures and camera states. FIG. 17 is a diagram showing a state in which a user is capturing an image with an imaging device. FIG. 18 is a diagram showing a state in which an imaging device is capturing an image using a tripod. Fig. 10 is a diagram showing a state in which a mobile terminal device and an imaging device are wirelessly connected. Fig. 11 is a diagram showing a flowchart of a control process in a fourth embodiment. Fig. 12 is a diagram showing a flowchart of a control process in a fourth embodiment. Fig. 13 is a diagram showing an example of operations of an imaging device, headphones, and a mobile terminal device based on a headphone gesture. Fig. 14 is a diagram showing an example of a notification in response to a headphone gesture.

[0008] Hereinafter, with reference to the accompanying drawings, embodiments will be described in the following order: <1. Configuration of information processing system> <2. First embodiment> <3. Second embodiment> <4. Third embodiment> <5. Fourth embodiment> <6. Modification> <7. Summary> <8. Present technology>

[0009] 1. Configuration of Information Processing System Fig. 1 is a diagram showing the configuration of an information processing system 1 according to this embodiment. As shown in Fig. 1, the information processing system 1 includes an image capture device 2, which is an embodiment of an information processing device, and headphones 3, which are an embodiment of a head-mounted device. The image capture device 2 and headphones 3 are wirelessly connected in accordance with, for example, the Bluetooth (registered trademark) standard, and are capable of wireless communication with each other. Note that the image capture device 2 and headphones 3 may be connected by wire, or may be wirelessly connected in accordance with another standard.

[0010] The imaging device 2 is a camera that captures an image of a subject in response to a user operation and generates still images and moving images. However, the imaging device 2 may be capable of generating only still images or moving images.

[0011] The headphones 3 are worn on the user's head. The headphones 3 output sound input from, for example, the imaging device 2 via wireless communication. The headphones 3 may also be wirelessly connected to a mobile terminal device such as a smartphone and output sound (e.g., music) input from the mobile terminal device. Furthermore, the headphones 3 may output sound stored in a storage unit (not shown).

[0012] FIG. 2 is a diagram showing the appearance of the imaging device 2. Note that in the following description, the subject side is the front and the photographer (user) side is the rear. As shown in FIGS. 1 and 2 , the imaging device 2 includes a camera housing 11 in which necessary components are arranged inside and outside, and a lens barrel 12 that is detachable from the camera housing 11 and attached to a front portion 11a. Note that, in the imaging device 2, an example will be described in which the lens barrel 12 is detachable as a so-called interchangeable lens, but the lens barrel 12 may also be configured to be inseparable from the camera housing 11.

[0013] A display unit 13 is disposed on the rear surface 11b of the camera housing 11. The display unit 13 displays live view images, recorded images, and the like. The display unit 13 is configured with a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display unit 13 is rotatable relative to the camera housing 11. For example, the display unit 13 is rotatable around the left end portion of the display unit 13 as a rotation axis, with the right end portion of the display unit 13 moving backward. Note that the right end portion, upper end portion, or lower end portion of the display unit 13 may be the rotation axis. Furthermore, the display unit 13 may be rotatable around multiple axes.

[0014] An EVF (Electric Viewfinder) 14 is disposed on the top surface 11c of the camera housing 11. The EVF 14 includes an EVF monitor 14a and a frame-shaped enclosure 14b that protrudes rearward and surrounds the periphery of the EVF monitor 14a. The EVF monitor 14a is configured with a display device such as an LCD or an organic EL display. Note that an optical viewfinder (OVF) may be provided instead of the EVF monitor 14a.

[0015] Various operation units 15 are provided on the rear surface 11b and the top surface 11c. Examples include a shutter button (release button), a playback menu start button, a confirmation button, a cross key, a cancel button, a zoom key, and a slide key. These operation units 15 include various types of buttons, dials, and combination controls that can be pressed and rotated. The operation units 15 also include a touch panel 15a provided on the front surface of the display unit 13. The various types of operation units 15 enable, for example, shutter operation, menu operation, playback operation, mode selection operation, focus operation, zoom operation, and parameter selection operation such as shutter speed and F-number.

[0016] Fig. 3 is a diagram showing the internal configuration of the imaging device 2. As shown in Fig. 3, in addition to the display unit 13 and the operation unit 15, the imaging device 2 includes a lens system 21, an imaging unit 22, a camera signal processing unit 23, a recording control unit 24, a communication unit 25, a camera control unit 26, a memory unit 27, a driver unit 28, an audio input unit 29, an audio processing unit 30, an audio output unit 31, and a sensor unit 32.

[0017] The lens system 21 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The lens system 21 guides light from a subject (incident light) to the imaging unit 22 and condenses the light.

[0018] The imaging unit 22 includes an image sensor 22a (imaging element), such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) type. The imaging unit 22 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signal obtained by photoelectrically converting light received by the image sensor 22a, and further performs A / D (Analog / Digital) conversion processing. The imaging unit 22 then outputs the captured image signal as digital data to the camera signal processing unit 23 and the camera control unit 26, which are provided downstream.

[0019] The camera signal processing unit 23 is configured as an image processor using, for example, a DSP (Digital Signal Processor), etc. The camera signal processing unit 23 performs various signal processes on the captured image signal from the imaging unit 22. The camera signal processing unit 23 performs, for example, pre-processing, synchronization processing, YC generation processing, resolution conversion processing, file formation processing, etc.

[0020] In the pre-processing, the captured image signal from the imaging unit 22 is subjected to clamping processing, which clamps the R, G, and B black levels to a predetermined level, and correction processing between the R, G, and B color channels. In the synchronization processing, a color separation processing is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, a resolution conversion process is performed on the image data that has been subjected to various signal processing.

[0021] In the file creation process, the image data that has undergone the various processes described above is subjected to compression encoding for recording or communication, formatting, and metadata generation and addition, etc., to generate files for recording or communication. For example, image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) are generated as still image files. It is also possible to generate image files in the MP4 format used for recording MPEG-4-compliant video and audio. It is also possible to generate image files as raw image data.

[0022] The camera signal processing unit 23 generates metadata that includes information on processing parameters within the camera signal processing unit 23, various control parameters obtained from the camera control unit 26, information indicating the operating status of the lens system 21 and the imaging unit 22, mode setting information, and imaging environment information (date, time, location, etc.).

[0023] The recording control unit 24 performs recording and playback on a recording medium, such as a nonvolatile memory. The recording control unit 24 performs processing to record image files, such as video data and still image data, and thumbnail images, on the recording medium. The actual configuration of the recording control unit 24 can be various. For example, the recording control unit 24 may be configured as a flash memory built into the imaging device 2 and its write / read circuit. The recording control unit 24 may also be configured as a card recording / playback unit that performs recording and playback access on a recording medium that can be attached to or detached from the imaging device 2, such as a memory card (such as a portable flash memory). The recording control unit 24 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 2.

[0024] The display unit 13 executes various displays on the display screen based on instructions from the camera control unit 26. For example, the display unit 13 displays a playback image of image data read from a recording medium by the recording control unit 24. The display unit 13 may also receive image data of a captured image whose resolution has been converted for display by the camera signal processing unit 23, and may perform display based on the image data of the captured image in response to instructions from the camera control unit 26. This allows a so-called through image (a monitoring image of a subject), which is an image captured while checking the composition or recording a video, to be displayed. The through image is also displayed on the EVF 14. The display unit 13 also executes displays on the screen of various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the camera control unit 26.

[0025] The communication unit 25 performs wireless communication with the headphones 3, for example, in accordance with Bluetooth (registered trademark). The communication unit 25 may also communicate with other devices via wired or wireless communication. The communication unit 25 transmits and outputs captured image data (still image files and video files) to, for example, an external display device, recording device, playback device, etc.

[0026] The camera control unit 26 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 27 stores information used by the camera control unit 26 for processing. The memory unit 27 generally refers to, for example, a read-only memory (ROM), a random access memory (RAM), a flash memory, and the like. The memory unit 27 may be a memory area built into the microcomputer chip that serves as the camera control unit 26, or may be configured as a separate memory chip. The camera control unit 26 controls the entire imaging device 2 by executing programs stored in the ROM, flash memory, or the like of the memory unit 27. For example, the camera control unit 26 controls the operation of each necessary unit, such as controlling the shutter speed of the imaging unit 22, issuing instructions for various signal processing in the camera signal processing unit 23, imaging and recording operations in response to user operations, playback of recorded image files, operations of the lens system 21 such as zoom, focus, and aperture adjustment in the lens barrel 12, user interface operations, and processing by the audio processing unit 30.

[0027] The RAM in the memory unit 27 serves as a working area for the CPU of the camera control unit 26 to temporarily store data, programs, and the like when processing various types of data. The ROM and flash memory (non-volatile memory) in the memory unit 27 are used to store an OS (Operating System) for the CPU to control each unit, content files such as image files, application programs for various operations, firmware, and various setting information. The various setting information includes exposure settings, shutter speed settings, and mode settings as setting information related to imaging operations, white balance settings, color settings, and image effect settings as setting information related to image processing, settings related to processing by the audio processing unit 30 (for example, settings of parameters related to volume, sound quality, and other processing), and custom key settings and display settings as setting information related to operability.

[0028] The driver unit 28 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, etc. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 26, and execute operations such as moving the focus lens or zoom lens and opening and closing the diaphragm blades of the diaphragm mechanism.

[0029] The sensor unit 32 collectively refers to various sensors mounted on the imaging device 2. The sensor unit 32 includes, for example, an acceleration sensor 32a, a gyro sensor 32b, and a proximity sensor 32c. The acceleration sensor 32a detects acceleration in three axes of the imaging device 2: the roll axis (front-back axis), pitch axis (left-right axis), and yaw axis (up-down axis). The gyro sensor 32b detects angular velocity of the imaging device 2 around the three axes: the roll axis, pitch axis, and yaw axis. The proximity sensor 32c is, for example, an IR (Infrared) sensor, and is provided in the enclosure 14b of the EVF 14 as shown in FIG. 2 . The proximity sensor 32c detects when the photographer (user) approaches the EVF 14. That is, the proximity sensor 32c detects whether the photographer is looking into the EVF 14.

[0030] The audio input unit 29 has, for example, a microphone and a microphone amplifier, and picks up ambient sounds and outputs them as audio signals.

[0031] The audio processing unit 30 converts the audio signal obtained by the audio input unit 29 into a digital audio signal, and performs AGC processing, sound quality processing, noise reduction processing, etc. The audio data that has undergone these processes is output to the camera signal processing unit 23 and the camera control unit 26. For example, when a moving image is captured, the audio data is processed in the camera control unit 26 as audio data accompanying the moving image.

[0032] The audio output unit 31 includes an audio signal processing circuit, a power amplifier, a speaker, etc., and plays back audio files recorded on a recording medium by the recording control unit 24. When a video is played back, the audio accompanying the video is played back by the audio output unit 31.

[0033] As shown in FIG. 1 , the headphones 3 include a headband 41 and a pair of housings 42R and 42L. The headband 41 is curved to fit the user's head and supports the entire headphones 3 by contacting the top of the user's head when worn. The housings 42R and 42L are attached to both ends of the headband 41. The housing 42R is positioned facing the user's right ear when the headphones 3 are worn on the user's head. The housing 42L is positioned facing the user's left ear when the headphones 3 are worn on the user's head. In the following, unless there is a need to particularly distinguish between the housings 42R and 42L, they will simply be referred to as the housings 42.

[0034] A touch panel 54a and a proximity sensor 56c are provided on the surface of the housing part 42, which is the outer casing of the headphones 3, that is, on the surface of the housing facing outward, opposite to the user.

[0035] 4 is a diagram showing the internal configuration of the headphones 3. As shown in FIG. 4, the headphones 3 include a headphone control unit 51, a signal processing unit 52, an audio output unit 53, an operation unit 54, a communication unit 55, and a sensor unit 56.

[0036] The headphone control unit 51 is configured by a microcomputer (arithmetic processing unit) equipped with a CPU, RAM, and ROM. The headphone control unit 51 executes programs stored in the ROM, flash memory, etc. to comprehensively control the entire headphones 3. For example, the headphone control unit 51 controls the operation of each necessary unit for sound reproduction in response to user operations, processing of the audio processing unit 30, etc.

[0037] The signal processing unit 52 is equipped with an audio signal processing circuit and a power amplifier, and converts a digital audio signal input, for example, via the communication unit 55, into an analog audio signal, amplifies it, and outputs the amplified analog audio signal to the audio output unit 53.

[0038] The audio output unit 53 is configured by, for example, a speaker, and is provided in each of the housings 42 R and 42 L. The audio output unit 53 outputs (plays) sound based on the analog audio signal input from the signal processing unit 52.

[0039] The operation unit 54 includes a touch panel 54a provided on the housing surface of the housing unit 42. The operation unit 54 enables, for example, a playback operation, a volume up operation, a volume down operation, and the like.

[0040] The communication unit 55 performs wireless communication with the imaging device 2, for example, in accordance with Bluetooth (registered trademark). The communication unit 55 may also perform wired or wireless communication with other devices. The communication unit 55 receives digital audio signals from, for example, an external mobile terminal device, a personal computer, or the like.

[0041] The sensor unit 56 collectively refers to various sensors mounted on the headphones 3. The sensor unit 56 includes, for example, an acceleration sensor 56a, a gyro sensor 56b, and a proximity sensor 56c. The acceleration sensor 56a detects acceleration in three axial directions, i.e., the roll axis, pitch axis, and yaw axis, of the headphones 3. The gyro sensor 56b detects angular velocity around three axes, i.e., the roll axis, pitch axis, and yaw axis, of the headphones 3. The proximity sensor 56c is, for example, an IR (Infrared) sensor, and is provided on the housing surface of the housing unit 42 as shown in FIG. 1 . The proximity sensor 56c detects when the user's hand approaches the housing unit 42. Therefore, the proximity sensor 56c also functions as the operation unit 54 that accepts gestures input by the user's hand.

[0042] 5 is a diagram showing the functional configuration of the camera control unit 26 and the headphone control unit 51. As shown in FIG. 5, the camera control unit 26 includes a camera state detection unit 61, an operation control unit 62, and a notification control unit 63.

[0043] The camera state detection unit 61 acquires the acceleration detected by the acceleration sensor 32a and the angular velocity detected by the gyro sensor 32b. Then, the camera state detection unit 61 detects the state (movement and attitude) of the imaging device 2 based on the acquired acceleration and angular velocity. Note that, hereinafter, the state of the imaging device 2 will be referred to as the camera state. In this way, the camera state detection unit 61 acquires information indicating the camera state (hereinafter, referred to as camera state information).

[0044] The operation control unit 62 controls the operation of at least the imaging device 2, more specifically the imaging device 2 and the headphones 3, based on headphone gesture information and camera state information, which will be described later. When operating the imaging device 2, the operation control unit 62 performs the operation based on the headphone gesture information and camera state information. When operating the headphones 3, the operation control unit 62 instructs the headphones 3 to perform the operation based on the headphone gesture information and camera state information. Upon receiving the instruction, the headphones 3 perform the operation based on the headphone gesture information and camera state information by an operation control unit 72, which will be described later.

[0045] The notification control unit 63 notifies the user of the headphone gesture information and operations based on the headphone gesture information and the camera state information. The notification control unit 63 also transmits information indicating the performed operation to the headphones 3 via the communication unit 25.

[0046] The headphone control unit 51 functions as a headphone gesture detection unit 71 , an operation control unit 72 , and a notification control unit 73 .

[0047] The headphone gesture detection unit 71 detects the state (movement and posture) of the headphones 3 based on the acceleration detected by the acceleration sensor 56a and the angular velocity detected by the gyro sensor 56b. Here, because the headphones 3 are worn on the user's head, the headphone gesture detection unit 71 detects the movement and posture of the headphones 3, thereby detecting the movement and posture of the user's head. Then, based on the detected state of the headphones 3, the headphone gesture detection unit 71 detects predetermined specific movements and postures as gestures of the headphones 3. Note that, hereinafter, gestures detected by the headphones 3 are referred to as headphone gestures.

[0048] When the headphone gesture detection unit 71 detects a headphone gesture, it transmits information indicating the detected headphone gesture (hereinafter referred to as headphone gesture information) to the imaging device 2 via the communication unit 55. As a result, the imaging device 2 acquires the headphone gesture information transmitted from the headphones 3 via the communication unit 25. Note that the headphone gesture detection unit 71 may transmit the acceleration detected by the acceleration sensor 56a and the angular velocity detected by the gyro sensor 56b as headphone gesture information to the imaging device 2. In this case, the camera control unit 26 may also function as the headphone gesture detection unit 71 and detect the headphone gesture based on the transmitted acceleration and angular velocity.

[0049] When the operation control unit 72 receives an operation instruction based on the headphone gesture information and the camera state information from the imaging device 2, it controls the operation in accordance with the instruction.

[0050] When the notification control unit 73 receives information indicating an operation performed based on the headphone gesture information and the camera state information, it notifies the user of the operation indicated in the information.

[0051] Below, the first to fourth embodiments will be described, each describing the processing performed by each functional unit of the camera control unit 26 and the headphone control unit 51. Note that in the first to fourth embodiments, some of the functional units may not perform processing.

[0052] 6 is a diagram showing a state in which a user 100 is capturing an image with an imaging device 2 while wearing headphones 3. As shown in Fig. 6, it is conceivable that the user 100 captures an image of a subject with the imaging device 2 while wearing the headphones 3 on the head.

[0053] It is assumed that the user 100 is capturing an image of a subject with the imaging device 2 while looking through the EVF 14. In this case, when the user 100 moves the imaging device 2 to a desired position, the user 100 also moves his or her head. In other words, the movement of the imaging device 2 and the movement of the headphones 3 are almost identical.

[0054] In this case, if the operation is controlled based on the movement of the headphones 3, i.e., headphone gesture information, there is a risk that an operation not intended by the user 100 will be performed even though the user 100 simply moves his or her head in accordance with the imaging device 2.

[0055] Therefore, in the first embodiment, when the headphone gesture and the camera state match, the operation control unit 62 does not perform (cancels) the operation based on the headphone gesture information.

[0056] Fig. 7 is a flowchart of a control process according to the first embodiment. Fig. 8 is a diagram showing an example of an operation of the imaging device 2 and the headphones 3 based on headphone gesture information. Fig. 9 is a diagram showing an example of a notification of an operation based on headphone gesture information.

[0057] 7 , when the control process starts, in step S1, the camera control unit 26 determines whether headphone gesture information has been acquired from the headphones 3. In the headphones 3, the headphone gesture detection unit 71 constantly acquires the acceleration detected by the acceleration sensor 56a and the angular velocity detected by the gyro sensor 56b, and detects a headphone gesture based on the acquired acceleration and angular velocity. The headphone gesture detection unit 71 detects headphone gestures such as "shake up," "shake down," "shake right," and "shake left," for example. The headphone gesture detection unit 71 then transmits headphone gesture information indicating the detected headphone gesture to the imaging device 2.

[0058] When headphone gesture information is received from the headphones 3 (Yes in step S1), in step S2 the camera state detection unit 61 detects the camera state based on the acceleration detected by the acceleration sensor 32a and the angular velocity detected by the gyro sensor 32b. Here, the camera state detection unit 61 detects state changes, for example, to "upward," "downward," "rightward," or "leftward." However, because the state of the imaging device 2 may not change, the camera state detection unit 61 does not necessarily detect these camera states.

[0059] In step S3, the operation control unit 62 determines whether the camera state and the headphone gesture match. Here, the operation control unit 62 determines whether the image capture device 2 and the headphones 3 are moving in the same direction, that is, whether the user 100 is moving while looking into the EVF 14 of the image capture device 2. For example, if the headphone gesture is "shake up" and the camera state is a state change to "upward," the operation control unit 62 determines that the camera state and the headphone gesture match. Note that "match" does not necessarily mean a perfect match, but also includes a case where it is determined that there is a match with an error within a predetermined range.

[0060] If the camera state and the headphone gesture match (Yes in step S3), that is, if the user 100 is moving while looking into the EVF 14 of the imaging device 2, the operation control unit 62 cancels the operation based on the headphone gesture information in step S4. In this case, the operation control unit 62 determines that the user 100 is simply moving while looking into the EVF 14 of the imaging device 2 and is not performing a gesture, and performs control so as not to perform the operation based on the headphone gesture information.

[0061] On the other hand, if the camera state and the headphone gesture do not match (No in step S3), that is, if the head of the user 100 and the imaging device 2 are moving separately, it is determined that the user 100 is making a gesture. Then, in step S5, the operation control unit 62 determines whether a moving image is being captured by the imaging device 2. Here, when a moving image is being captured, the imaging device 2 is operated by the headphone gesture so as not to cause so-called camera shake in the imaging device 2. Furthermore, when a moving image is not being captured, the headphones 3 are simply operated by the headphone gesture.

[0062] If a moving image is being captured (Yes in step S5), in step S6, the operation control unit 62 controls the operation of the imaging device 2 based on the headphone gesture information. Here, as shown in Fig. 8 , if the headphone gesture is "shake up," the operation control unit 62 controls the imaging device 2 to zoom in. Also, if the headphone gesture is "shake down," the operation control unit 62 controls the imaging device 2 to zoom out. Also, if the headphone gesture is "shake left," the operation control unit 62 controls the imaging device 2 to increase the ISO sensitivity. Also, if the headphone gesture is "shake right," the operation control unit 62 controls the imaging device 2 to decrease the ISO sensitivity.

[0063] In step S7, the notification control unit 63 notifies the operation of the imaging device 2 based on the headphone gesture information performed in step S6. As shown in Fig. 9 , when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output, from the audio output unit 53, a gesture sound (notification sound) notifying that the imaging device 2 has been operated based on the headphone gesture information. As a result, in the headphones 3, the notification control unit 73 causes the audio output unit 53 to output a gesture sound notifying that the operation of the imaging device 2 has been controlled based on the headphone gesture information.

[0064] Furthermore, when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 causes the display unit 13 to display a camera GUI 81 as shown in FIG. 10 , which notifies the user that the operation of the imaging device 2 has been controlled based on the headphone gesture information. FIG. 10 is a diagram showing an example of the camera GUI 81. As shown in FIG. 10 , the camera GUI 81 displays a camera icon 81a and an operation control comment 81b at the bottom, indicating that the operation of the imaging device 2 has been controlled based on the headphone gesture. The camera icon 81a is an icon that resembles the imaging device 2 so that it is easy to understand that the object of operation control is the imaging device 2, and is displayed more emphasized than a headphone icon 82a, which will be described later. For example, the camera icon 81a is displayed as a larger icon than the headphone icon 82a. Furthermore, the operation control comment 81b is displayed more emphasized than the operation control comment 82b, which will be described later. For example, the operation control comment 81b is displayed in larger characters and in a different color (more conspicuous color) than the operation control comment 82b. In this way, when the operation of the imaging device 2 is controlled based on the headphone gesture information, the display mode is emphasized compared to the headphone GUI 82 (see FIG. 11 ), so that the user 100 is notified in a more easily understandable manner.

[0065] However, when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 prevents the gesture sound notifying that the operation of the imaging device 2 has been controlled based on the headphone gesture information from being output from the audio output unit 31. This is to prevent the gesture sound from being picked up by the audio input unit 29 if it is output during imaging of a moving image.

[0066] On the other hand, if a moving image is not being captured (No in step S5), in step S8, the operation control unit 62 controls the operation of the headphones 3 based on the headphone gesture information. Here, as shown in FIG. 8 , if the headphone gesture is "shake up," the operation control unit 62 instructs the headphones 3 to increase the volume in the headphones 3. In the headphones 3, the operation control unit 72 controls to increase the volume. Also, if the headphone gesture is "shake down," the operation control unit 62 instructs the headphones 3 to decrease the volume. In the headphones 3, the operation control unit 72 controls to decrease the volume. Also, if the headphone gesture is "shake left," the operation control unit 62 instructs the headphones 3 to switch to the previous song. In the headphones 3, the operation control unit 72 controls to switch to the previous song. Also, if the headphone gesture is "shake right," the operation control unit 62 instructs the headphones 3 to switch to the next song. In the headphones 3, the operation control unit 72 controls to switch to the next song.

[0067] In step S9, the notification control unit 63 notifies the operation of the headphones 3 based on the headphone gesture information performed in step S8.

[0068] 9 , when the operation of the headphones 3 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output a gesture sound (notification sound) that notifies that the headphones 3 have been operated based on the headphone gesture information from the audio output unit 53. As a result, the notification control unit 73 in the headphones 3 causes the audio output unit 53 to output a gesture sound that notifies that the headphones 3 have been operated based on the headphone gesture information.

[0069] Furthermore, when the operation of the headphones 3 is controlled based on the headphone gesture information, the notification control unit 63 displays a headphone GUI 82 as shown in FIG. 11 on the display unit 13 to notify that the headphones 3 have been operated based on the headphone gesture information. FIG. 11 is a diagram showing an example of the headphone GUI 82. As shown in FIG. 11 , a headphone icon 82a and an operation control comment 82b, which indicate that the headphones 3 have been operated based on the headphone gesture information, are displayed at the bottom of the headphone GUI 82. The headphone icon 82a is a small icon resembling the headphones 3 so that it is easy to understand that the target of operation control is the headphones 3. Furthermore, the operation control comment 82b is displayed in smaller characters and in an inconspicuous color (e.g., black) compared to the operation control comment 81b. In this way, because the target of operation control in the headphone GUI 82 is the headphones 3, the notification is provided on the display unit 13 of the imaging device 2 in an inconspicuous manner so as not to interfere with imaging.

[0070] Furthermore, when controlling the operation of the headphones 3 based on the headphone gesture information, the notification control unit 63 causes the audio output unit 31 to output a gesture sound notifying the user that the headphones 3 are being operated based on the headphone gesture information or that the headphones 3 have been operated based on the headphone gesture information. Here, because no moving images are captured by the imaging device 2, the gesture sound is not picked up by the audio input unit 29. Therefore, the gesture sound is output from the imaging device 2 to notify the user that an operation has been performed based on the headphone gesture information.

[0071] If it is determined in step S3 that the camera state information and the headphone gesture information do not match, the operation based on the headphone gesture information is canceled. Therefore, it can be said that the operation control unit 62 determines whether or not to execute the operation on the headphones 3 based on the headphone gesture information, based on the camera state information.

[0072] 12 is a diagram showing a flowchart of the control process in Modification 1 of the first embodiment. Modification 1 differs from the first embodiment in that it determines whether to cancel the operation based on the gesture information based on the detection result of the proximity sensor 32c. Note that, here, processes different from the first embodiment will be described, and processes identical to those in the first embodiment will be assigned the same reference numerals and their description will be omitted.

[0073] 12, when the control process starts and information regarding a headphone gesture is acquired in step S1, the operation control unit 62 determines in step S11 whether the user is looking into the EVF 14. Here, whether the user is approaching the EVF 14 is determined based on whether the proximity sensor 32c is responding.

[0074] Then, if the proximity sensor 32c reacts (Yes in step S11), that is, if the user 100 is looking into the EVF 14, the operation control unit 62 determines that the user 100 is looking into the EVF 14 and therefore is not making a gesture toward the headphones 3. Therefore, if the proximity sensor 32c reacts (Yes in step S11), the process proceeds to step S4, and the operation control unit 62 cancels the gesture operation even if a headphone gesture has been detected.

[0075] On the other hand, if the proximity sensor 32c does not react (No in step S11), that is, if the user 100 is not approaching the EVF 14, the operation control unit 62 determines that the user 100 is not looking into the EVF 14 and therefore is making a gesture toward the headphones 3. Therefore, if the proximity sensor 32c does not react (No in step S11), the camera control unit 26 performs steps S2 to S9 described above.

[0076] This prevents operations based on headphone gestures detected when the user 100 is looking into the EVF 14, thereby reducing the possibility of operations being performed based on movements or postures that the user does not intend.

[0077] 13 is a diagram showing a flowchart of the control process in Modification 2 of the first embodiment. Modification 2 differs from the first embodiment in that a gesture operation is performed based on the detection result of the proximity sensor 56c. Note that, here, only processes different from the first embodiment will be described, and processes identical to those in the first embodiment will be denoted by the same reference numerals and their description will be omitted.

[0078] 13 , when the control process starts and information about a headphone gesture is acquired in step S1, the camera control unit 26 determines in step S12 whether there has been an input to the proximity sensor 56c. Here, when the proximity sensor 56c of the housing part 42R (the right proximity sensor 56c) is tapped or pressed for a long time, or when the proximity sensor 56c of the housing part 42L (the left proximity sensor 56c) is tapped or pressed for a long time, that information is transmitted from the headphones 3 to the imaging device 2. Then, in step S12, the camera control unit 26 determines whether there has been an input to the proximity sensor 56c based on the transmitted information.

[0079] As a result, if there is an input to the proximity sensor 56c (Yes in step S12), the process proceeds to step S6. In step S6, the operation control unit 62 controls the operation of the imaging device 2 based on the input to the proximity sensor 56c. For example, if the right proximity sensor 56c is tapped, the operation control unit 62 controls the imaging device 2 to zoom in. Furthermore, if the left proximity sensor 56c is tapped, the operation control unit 62 controls the imaging device 2 to zoom out. Furthermore, if the right proximity sensor 56c is pressed and held, the operation control unit 62 controls the imaging device 2 to increase the ISO sensitivity. Furthermore, if the left proximity sensor 56c is pressed and held, the operation control unit 62 controls the imaging device 2 to decrease the ISO sensitivity.

[0080] In this way, when a gesture is input to the proximity sensor 56c, the operation control unit 62 controls the operation by giving priority to the gesture, thereby enabling the operation control to better reflect the user's intentions.

[0081] 3. Second Embodiment In a second embodiment, the operation of the imaging device 2 is controlled based on a combination of headphone gesture information and camera state information.

[0082] Fig. 14 is a diagram showing a flowchart of a control process in the second embodiment. Fig. 15 is a diagram showing an example of an operation of the imaging device 2 in response to a combination of headphone gesture information and camera state information.

[0083] 14 , when the control process starts, in step S21 the camera control unit 26 determines whether headphone gesture information has been acquired from the headphones 3. Here, the headphone gesture detection unit 71 detects any of the headphone gestures, for example, “shake up and down,” “shake sideways,” “turn right,” and “turn left.”

[0084] When headphone gesture information is acquired from the headphones 3, in step S22 the camera state detection unit 61 detects the camera state based on the acceleration detected by the acceleration sensor 32 a and the angular velocity detected by the gyro sensor 32 b. Here, the camera state detection unit 61 detects a state change, for example, to one of "upward," "downward," "rightward," or "leftward."

[0085] In step S23, the operation control unit 62 controls the operation of the imaging device 2 as shown in FIG. 15 based on a combination of the headphone gesture information and the camera state information. For example, when the headphone gesture is "shake up and down" and the camera state is "upward," the operation control unit 62 controls to bring the focus closer. When the headphone gesture is "shake sideways" and the camera state is "upward," the operation control unit 62 controls to change the imaging mode to P. Note that imaging mode P is a mode in which exposure (shutter speed, aperture value) is automatically set. Furthermore, imaging mode A is a mode in which the aperture value is set as a priority and the shutter speed is automatically adjusted. Furthermore, imaging mode S is a mode in which the shutter speed is set as a priority and the aperture value is automatically adjusted. Furthermore, imaging mode M is a mode in which the aperture value and shutter speed are manually adjusted.

[0086] In this way, by controlling the operation of the imaging device 2 based on a combination of headphone gesture information and camera state information, the number of operations that can be performed by gesture operations can be increased compared to when the operation of the imaging device 2 is controlled based on only one of the headphone gesture information and camera state information. In other words, it becomes possible to control a large number of operations by gesture operations.

[0087] 16 is a diagram illustrating the orientation of the imaging device 2 when capturing an image in landscape orientation. As shown in FIG. 16 , when the user 100 captures an image in landscape orientation, the imaging device 2 may also be tilted to match the posture of the user 100. In such a case, if the orientation of the image (screen) displayed on the display unit 13 is not changed, the user 100 may find it difficult to view the screen. Therefore, in the third embodiment, the orientation of the image (screen) displayed on the display unit 13 is controlled based on a combination of headphone gesture information and camera state information.

[0088] Fig. 17 is a flowchart showing a control process in the second embodiment. Fig. 18 is a diagram showing an example of the screen orientation corresponding to the headphone gesture and the camera state. Note that the word "ICON" in Fig. 18 indicates the orientation of the icon displayed on the display unit 13.

[0089] 17 , when the control process starts, in step S31 the camera control unit 26 acquires headphone gesture information from the headphones 3. Here, since it is only necessary to know the orientation (posture) of the user 100, the posture of the headphones 3 is acquired as the headphone gesture information.

[0090] Upon receiving headphone gesture information from the headphones 3, in step S32, the camera state detection unit 61 detects the camera state based on the acceleration detected by the acceleration sensor 32 a and the angular velocity detected by the gyro sensor 32 b. Here, the camera state detection unit 61 detects the attitude of the imaging device 2.

[0091] In step S33, the operation control unit 62 displays an image on the display unit 13 in an orientation as shown in FIG. 18 based on a combination of headphone gesture information and camera state information. In FIG. 18, (A) shows the orientation of the user 100 based on the headphone gesture information, and (B) shows the orientation of the imaging device 2 based on the camera state. For example, (A) 1 in FIG. 18 indicates that the headphones 3 are facing forward and the user 100 is facing forward. Also, (A) 2 in FIG. 18 indicates that the headphones 3 are tilted 90 degrees to the right and the user 100 is facing right-sideways. Also, (A) 3 in FIG. 18 indicates that the headphones 3 are facing upside down and the user 100 is facing upside down. Also, (A) 4 in FIG. 18 indicates that the headphones 3 are tilted 90 degrees to the left and the user 100 is facing left-sideways.

[0092] 1 in (B) of Fig. 18 indicates that the imaging device 2 is facing forward. 2 in (B) of Fig. 18 indicates that the imaging device 2 has been rotated 90° to the right. 3 in (B) of Fig. 18 indicates that the imaging device 2 is facing upside down. 4 in (B) of Fig. 18 indicates that the imaging device 2 has been rotated 90° to the left.

[0093] Then, for example, when the user is facing forward ((A)-1) and the imaging device 2 is rotated 90° to the right ((B)-2), the operation control unit 62 controls the image displayed on the display unit 13 to be displayed vertically and facing forward. Also, when the user is facing right-sideways ((A)-2) and the imaging device 2 is rotated 90° to the right ((B)-2), the operation control unit 62 controls the image displayed on the display unit 13 to be displayed horizontally and rotated 90° to the right.

[0094] In this way, in the information processing system 1, the orientation of the image to be displayed on the display unit 13 is determined in accordance with the orientation of the imaging device 2 and the orientation of the headphones 3, thereby making the image easier to view.

[0095] 5. Fourth Embodiment Fig. 19 is a diagram showing how a user 100 is capturing an image of himself / herself using an imaging device 2. Fig. 20 is a diagram showing how an image is captured using the imaging device 2 while using a tripod 2a. Fig. 21 is a diagram showing how a mobile terminal device 110 and the imaging device 2 are wirelessly connected.

[0096] Here, the imaging device 2 is not only used when the user 100 captures an image of a subject from the display unit 13 or EVF 14 side, but can also be used when taking a so-called selfie, for example, by capturing an image of the user 100 himself, as shown in Figure 19.

[0097] The imaging device 2 is not only held by the user 100, but may also be used in a state where it is supported on a tripod 2a as shown in FIG.

[0098] Furthermore, as shown in FIG. 21 , the imaging device 2 may be wirelessly connected to a mobile terminal device 110 such as a smartphone via a communication unit 25 , and the operation of the imaging device 2 may be controlled based on operations on the mobile terminal device 110 .

[0099] The fourth embodiment is an embodiment in which the operation is controlled based on headphone gesture information and camera state information in such a case.

[0100] 22 and 23 are flowcharts showing control processing in the fourth embodiment. Fig. 24 is a diagram showing an example of operations of the imaging device 2, headphones 3, and mobile terminal device 110 based on headphone gestures. Fig. 25 is a diagram showing an example of notification in response to headphone gesture information.

[0101] As shown in FIG. 22, when the control process starts, the camera control unit 26 determines whether headphone gesture information has been acquired from the headphones 3 in step S41.

[0102] When information regarding the headphone gesture is acquired from the headphones 3 (Yes in step S41), in step S42 the camera state detection unit 61 detects the camera state based on the acceleration detected by the acceleration sensor 32a and the angular velocity detected by the gyro sensor 32b.

[0103] In step S43, the operation control unit 62 determines whether the acceleration detected by the acceleration sensor 32a is 0. Here, it determines whether the acceleration of the imaging device 2 is 0, that is, whether the imaging device 2 is being held by the tripod 2a, based on whether the imaging device 2 is moving or not.

[0104] If the acceleration is 0 (Yes in step S43), that is, if the imaging device 2 is supported on a tripod 2a, in step S44 the operation control unit 62 controls the operation of the headphones 3 based on the headphone gesture information. Here, as shown in Fig. 24 , similar to the first embodiment, if the headphone gesture is "shake up," for example, the operation control unit 62 issues an instruction to the headphones 3 to turn up the volume in the headphones 3. In the headphones 3, the operation control unit 72 controls to turn up the volume.

[0105] In step S45, the notification control unit 63 notifies the operation based on the headphone gesture information performed in step S44.

[0106] 25 , when the operation of the headphones 3 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output a gesture sound notifying that the headphones 3 have been operated based on the headphone gesture information from the audio output unit 53. As a result, the notification control unit 73 in the headphones 3 causes the audio output unit 53 to output a gesture sound notifying that the headphones 3 have been operated based on the headphone gesture information.

[0107] Furthermore, the notification control unit 63 causes the display unit 13 to display a headphone GUI 82 as shown in FIG. 11 that notifies that the headphones 3 have been operated based on the headphone gesture information.

[0108] Furthermore, the notification control unit 63 causes the audio output unit 31 to output a gesture sound that notifies that the headphones 3 have been operated based on the headphone gesture information.

[0109] On the other hand, if the acceleration is not 0 (No in step S43), that is, if at least the imaging device 2 is not supported on a tripod 2a, in step S46 the operation control unit 62 determines whether the camera state and the headphone gesture match.

[0110] If the camera state and the headphone gesture do not match (No in step S46), the operation control unit 62 determines in step S47 whether the imaging device 2 and the mobile terminal device 110 are connected. If the imaging device 2 and the mobile terminal device 110 are connected (Yes in step S47), the operation control unit 62 controls the operation of the mobile terminal device 110 based on the headphone gesture information in step S48. Here, as shown in FIG. 24 , if the headphone gesture is "shake up," the operation control unit 62 controls the operation of the mobile terminal device 110 to increase the shutter speed. Also, if the headphone gesture is "shake down," the operation control unit 62 controls the operation of the mobile terminal device 110 to decrease the shutter speed. Also, if the headphone gesture is "shake left," the operation control unit 62 controls the operation of the mobile terminal device 110 to increase the ISO sensitivity. Furthermore, when the headphone gesture is "shake right," the operation control unit 62 controls the portable terminal device 110 to decrease the ISO sensitivity.

[0111] In step S49, the notification control unit 63 notifies the operation based on the headphone gesture information performed in step S48. As shown in Fig. 25 , when the operation of the mobile terminal device 110 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output, from the audio output unit 53, a gesture sound that notifies that the mobile terminal device 110 has been operated based on the headphone gesture information.

[0112] Furthermore, the notification control unit 63 causes a smartphone GUI notifying that the mobile terminal device 110 has been operated based on the headphone gesture information to be displayed on the display unit 13. Note that, in the smartphone GUI, a smartphone icon and an operation control comment indicating that the mobile terminal device 110 has been operated based on the headphone gesture are displayed at the bottom, similar to the camera GUI 81 and the headphone GUI 82.

[0113] Furthermore, notification control unit 63 causes voice output unit 31 to output a gesture sound that notifies that mobile terminal device 110 has been operated based on the headphone gesture information. Furthermore, operation control unit 62 instructs mobile terminal device 110 to display, on the display unit of mobile terminal device 110, the details of the operation of mobile terminal device 110 based on the headphone gesture information. As a result, mobile terminal device 110 displays, on the display unit, the details of the operation of mobile terminal device 110 based on the headphone gesture.

[0114] On the other hand, if the imaging device 2 and the mobile terminal device 110 are not connected (No in step S47), as shown in Fig. 23 , the operation control unit 62 determines whether the display unit 13 of the imaging device 2 faces the lens barrel 12. Here, as described above, in the imaging device 2, the display unit 13 is rotatable with respect to the camera housing 11. The sensor unit 32 of the imaging device 2 is provided with a magnetic sensor (not shown) that detects the rotation of the display unit 13. The operation control unit 62 determines whether the display unit 13 faces the lens barrel 12 based on the detection result of the magnetic sensor.

[0115] If the display unit 13 is facing the lens barrel 12 (Yes in step S50), that is, if the user 100 is taking a selfie, then in step S51 the operation control unit 62 controls the operation of the imaging device 2 based on the headphone gesture information. Here, as shown in Fig. 24 , the operation control unit 62 controls the imaging device 2 to zoom in when the headphone gesture is, for example, "shake up," as in the first embodiment.

[0116] In step S52, the notification control unit 63 notifies the operation based on the headphone gesture information performed in step S51. As shown in Fig. 25 , when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output, from the audio output unit 53, a gesture sound notifying that the imaging device 2 has been operated based on the headphone gesture information. As a result, in the headphones 3, the notification control unit 73 causes the audio output unit 53 to output, from the headphone gesture information, a gesture sound notifying that the imaging device 2 has been operated based on the headphone gesture information.

[0117] Furthermore, the notification control unit 63 causes the display unit 13 to display a camera GUI 81 as shown in FIG. 10 that notifies the user that the imaging device 2 has been operated based on the headphone gesture information.

[0118] However, when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 prevents the gesture sound notifying that the imaging device 2 has been operated based on the headphone gesture from being output from the audio output unit 31. This is to prevent the gesture sound from being picked up if it is output during self-portrait shooting.

[0119] On the other hand, if the display unit 13 is not facing the lens barrel 12 (No in step S50), that is, if the user 100 is not taking a selfie, then in step S53 the operation control unit 62 controls the operation of the imaging device 2 based on the headphone gesture information. Note that, since the control in step S53 is similar to that in step S51, a description thereof will be omitted.

[0120] In step S54, the notification control unit 63 notifies the operation based on the headphone gesture information performed in step S52. As shown in Fig. 25 , when the operation of the imaging device 2 is controlled based on the headphone gesture information, the notification control unit 63 instructs the headphones 3 to output, from the audio output unit 53, a gesture sound notifying that the imaging device 2 has been operated based on the headphone gesture information. As a result, in the headphones 3, the notification control unit 73 causes the audio output unit 53 to output, from the headphone gesture information, a gesture sound notifying that the imaging device 2 has been operated based on the headphone gesture information.

[0121] 10 , which notifies that the imaging device 2 has been operated based on the headphone gesture information, is displayed on the display unit 13. Furthermore, the notification control unit 63 causes the audio output unit 31 to output a gesture sound, which notifies that the imaging device 2 has been operated based on the headphone gesture information.

[0122] On the other hand, if the camera state and the headphone gesture match (Yes in step S46), that is, if the user 100 is moving while looking into the EVF 14 of the image capture device 2, the operation control unit 62 cancels the gesture operation in step S55. In this case, the operation control unit 62 determines that the user 100 is simply moving while looking into the EVF 14 of the image capture device 2 and is not performing a gesture, and performs control so as not to perform a gesture operation based on the headphone gesture information.

[0123] If it is determined in step S46 that the camera state and the headphone gesture do not match, the operation based on the headphone gesture information is canceled. Therefore, it can be said that the operation control unit 62 determines whether or not to execute the operation on the headphones 3 based on the headphone gesture information, based on the camera state information.

[0124] 6. Modifications Note that the embodiment is not limited to the specific example described above, and various modifications may be made.

[0125] For example, in the above-described embodiment, the headphones 3 are used as a head-mounted device that detects the state (movement, posture) of the head of the user 100. However, the head-mounted device may be any device that is worn on the head of the user 100, such as earphones or a head-mounted display.

[0126] In the above-described embodiment, when music is being played through the headphones 3, the operation of the headphones 3 may be controlled based on a headphone gesture.

[0127] In the above embodiment, if it is determined that the image capture device 2 is capturing a bright scene based on the luminance value of the image, the determination threshold of the proximity sensor 56c may be lowered, thereby reducing erroneous determinations of the proximity sensor 56c due to sunlight.

[0128] Furthermore, the first to fourth embodiments described above may be executed in combination with each other, or all of the embodiments may be executed by switching between them.

[0129] Furthermore, in the above-described embodiment, the camera control unit 26 of the imaging device 2 functions as the operation control unit 62, but the headphone control unit 51 of the headphones 3 may function as the operation control unit 62. That is, the flow charts of the first to fourth embodiments described above may be performed by the headphone control unit 51. For example, in the fourth embodiment, when the display unit 13 of the imaging device 2 does not face the lens side (No in step S50), steps S5 to S9 in the first embodiment may be executed instead of steps S53 and S54.

[0130] Furthermore, in the above embodiment, the imaging device 2 has been described as an example of an information processing device, but the headphones 3 may function as the information processing device. Furthermore, instead of the imaging device 2 or the headphones 3, a mobile terminal device such as a smartphone connected to both or a computer on the cloud may function as the information processing device. In this case, the information processing device may transmit instruction information to the imaging device 2, and the camera control unit 26 of the imaging device 2 may perform various controls based on the instruction information.

[0131] Furthermore, the operations based on the headphone gesture information and the camera state information in the above-described embodiment are merely examples, and operations other than those described above may be controlled based on the headphone gesture information and the camera state information.

[0132] In the fourth embodiment described above, when the display unit 13 of the imaging device 2 faces the lens side (Yes in step S50), that is, when the user 100 is taking a selfie, the operation of the imaging device 2 is controlled based on the headphone gesture. However, when the user 100 is taking a selfie, the imaging device 2 captures an image of the user 100 himself, and the user 100 is unlikely to perform a gesture. Therefore, the operation based on the headphone gesture information may be canceled.

[0133] 7. Summary As described in the above examples, the information processing device (imaging device 2) includes a control unit (camera control unit 26) that controls at least the operation of the imaging device 2 based on gesture information (headphone gesture information) indicating a gesture (headphone gesture) detected by a head-worn device (headphones 3) worn on the user's head and imaging device state information (camera state information) indicating the state of the imaging device 2. This makes it possible to cancel the headphone gesture in accordance with the camera state when the imaging device 2 and the headphones 3 make approximately the same movement when controlling at least the operation of the imaging device 2 using a headphone gesture. Therefore, the imaging device 2 can accurately control its operation based on the state of the user's head.

[0134] The control unit cancels the operation based on the gesture information when the gesture matches the movement of the imaging device 2. This makes it possible to determine that the head of the user 100 is moving together with the imaging device 2 and not making a gesture when the imaging device 2 and the headphones 3 make approximately the same movement. Therefore, the imaging device 2 can reduce erroneous operations based on headphone gesture information.

[0135] The control unit controls the operation of the imaging device 2 and the head-mounted device based on the gesture information when the gesture does not match the movement of the imaging device 2. This makes it possible to determine that the user is intentionally making a gesture when the imaging device 2 and the headphones 3 do not make approximately the same movement. Therefore, the imaging device 2 can reduce erroneous operations based on headphone gesture information.

[0136] The control unit controls the operation of the imaging device 2 based on the gesture information when a moving image is being captured by the imaging device 2. As a result, when a moving image is being captured by the imaging device 2 and it is desired to prevent camera shake, the control unit can control the operation of the imaging device 2 based on the headphone gesture information, thereby reducing camera shake in the imaging device 2.

[0137] The control unit controls the operation of the head-worn device based on the gesture information when no moving images are captured by the imaging device 2. This makes it possible to determine that the user 100 has made a headphone gesture simply to operate the headphones 3 because no moving images are captured by the imaging device 2, and to control the operation of the headphones 3 based on the headphone gesture information.

[0138] The control unit controls to notify the user of information indicating control of the operation based on the gesture information, thereby enabling the user in the imaging device 2 to easily understand which device has been operated and how, based on the headphone gesture.

[0139] The control unit displays information indicating whether the control unit controlled the operation of the head-mounted device or the image capture device 2 on the display unit 13 of the image capture device 2. This allows the image capture device 2 to easily understand, through the screen of the display unit 13, which device was operated and how, by the headphone gesture.

[0140] The control unit displays information indicating control of operation based on the gesture information on the display unit 13 in different display modes depending on whether the control unit controls the operation of the head-worn device based on the gesture information or the operation of the imaging device 2 based on the gesture information. This allows the imaging device 2 to more easily grasp, through the screen of the display unit 13, what operation has been performed on which device by the headphone gesture.

[0141] The control unit displays icons (camera icon 81 a, headphone icon 82 a) that resemble the head-mounted device or the image capture device 2 that controls the operation based on the gesture information on the display unit 13. In this way, the image capture device 2 displays the icons on the display unit 13, allowing the user to more easily grasp the target of the operation control.

[0142] When controlling the operation of the imaging device 2 based on the gesture information, the control unit causes the display unit 13 to display the comment in a more emphasized display mode than when controlling the operation of the head-worn device based on the gesture information. For example, when controlling the operation of the imaging device 2 based on headphone gesture information, the control unit displays the characters of the operation control comment 81b larger and in a brighter color than the characters of the operation control comment 82b when controlling the operation of the headphones 3 based on the headphone gesture. This allows the imaging device 2 to more easily grasp the target of operation control. Furthermore, when controlling the operation of the headphones 3, the display can be prevented from interfering with imaging.

[0143] When controlling the operation of the head-mounted device based on the gesture information, the control unit causes the audio output unit 31 of the imaging device 2 to output a notification sound (gesture sound), and when controlling the operation of the imaging device 2 based on the gesture information, the control unit does not cause the audio output unit 31 of the imaging device 2 to output a notification sound. This makes it possible to prevent the audio input unit 29 of the imaging device 2 from picking up the notification sound when it is not desired to pick up the notification sound, such as during an operation while capturing a moving image with the imaging device 2.

[0144] When the imaging device 2 is capturing a moving image, the control unit does not output a notification sound from the audio output unit 31 of the imaging device 2. This makes it possible to prevent the audio input unit 29 from picking up a notification sound, for example, when the imaging device 2 is capturing a moving image.

[0145] The control unit cancels an operation based on the gesture information when the proximity sensor 32c provided in the imaging device 2 detects the user 100. This makes it possible to reduce the possibility of an operation based on a headphone gesture unintended by the user 100 being performed when the proximity sensor 32c detects the user 100, i.e., when the user 100 is looking into the EVF 14.

[0146] The control unit controls the operation of the head-mounted device based on the gesture information when no movement (acceleration) of the imaging device 2 is detected, thereby making it possible to control the operation of the headphones 3 based on the headphone gesture information when the imaging device 2 is supported on the tripod 2 a.

[0147] When the imaging device 2 and the mobile terminal device 110 are connected, the control unit controls the operation of the mobile terminal device 110 based on the gesture information. This makes it possible to operate the mobile terminal device 110 connected to the imaging device 2 using headphone gestures, thereby providing the user 100 with even greater ease of use.

[0148] The control unit controls the operation of the imaging device 2 based on a combination of the gesture information and the imaging device state information. This enables the imaging device 2 to control a large number of operations by gestures using a combination of the headphone gesture information and the camera state information.

[0149] The control unit determines the orientation of the image to be displayed on the display unit of the imaging device 2 based on the orientation of the head-mounted device indicated by the gesture information and the orientation of the imaging device 2 indicated by the imaging device state information. This allows the image to be displayed in an orientation that is easy for the user 100 to view, even when the user 100 is facing sideways or upside down.

[0150] The control unit cancels the operation based on the gesture information when the display unit 13 of the imaging device 2 faces the subject. This makes it possible to prevent the operation based on the headphone gesture information from being performed when taking a so-called selfie, since the subject is the user 100 and so the headphone gesture is often not performed.

[0151] In addition, the information processing method controls the operation of at least the imaging device 2 based on gesture information indicating a gesture detected by a head-mounted device (headphones 3) worn on the user's head and imaging device status information indicating the status of the imaging device 2.

[0152] The information processing device (imaging device 2) also includes a control unit (camera control unit 26) that determines whether or not to execute an operation on a head-worn device (headphones 3) worn on the user's head based on a gesture detected by the head-worn device, based on the state of the imaging device 2. This makes it possible to determine whether or not to execute a headphone gesture depending on the camera state when the imaging device 2 and the headphones 3 make approximately the same movement to control the operation of the headphones 3 using a headphone gesture.

[0153] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0154] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the various effects described above can be obtained.

[0155] <8. The Present Technology> The present technology may also have the following configurations. (1) An information processing device including: a control unit that controls operation of at least an imaging device based on gesture information indicating a gesture detected by a head-mounted device worn on a user's head and imaging device state information indicating a state of the imaging device. (2) The information processing device described in (1), wherein the control unit cancels an operation based on the gesture information when the gesture matches a movement of the imaging device. (3) The information processing device described in (1) or (2), wherein the control unit controls operation of the imaging device and the head-mounted device based on the gesture information when the gesture does not match a movement of the imaging device. (4) The information processing device described in any of (1) to (3), wherein the control unit controls operation of the imaging device based on the gesture information when a moving image is being captured by the imaging device. (5) The information processing device described in any of (1) to (4), wherein the control unit controls operation of the head-mounted device based on the gesture information when a moving image is not being captured by the imaging device. (6) The information processing device according to any one of (1) to (5), wherein the control unit performs control to notify the user of information indicating control of operation based on the gesture information. (7) The information processing device according to (6), wherein the control unit causes a display unit of the imaging device to display information indicating which operation of the head-mounted device or the imaging device has been controlled. (8) The information processing device according to (6) or (7), wherein the control unit causes a display unit of the imaging device to display the information indicating control of operation based on the gesture information in different display modes depending on whether the operation of the head-mounted device is controlled based on the gesture information or the operation of the imaging device is controlled based on the gesture information. (9) The information processing device according to (8), wherein the control unit causes the display unit to display an icon representing the head-mounted device or the imaging device that controls operation based on the gesture information.(10) The information processing device according to (8) or (9), wherein the control unit, when controlling the operation of the imaging device based on the gesture information, causes the display unit of the imaging device to display the information in a more emphasized display mode than when controlling the operation of the head-mounted device based on the gesture information. (11) The information processing device according to any of (6) to (10), wherein the control unit, when controlling the operation of the head-mounted device based on the gesture information, causes an audio output unit of the imaging device to output a notification sound, and when controlling the operation of the imaging device based on the gesture information, does not cause the audio output unit of the imaging device to output a notification sound. (12) The information processing device according to any of (6) to (11), wherein the control unit, when capturing a moving image with the imaging device, does not cause the audio output unit of the imaging device to output a notification sound. (13) The information processing device according to any of (1) to (12), wherein the control unit, when detecting a user by a proximity sensor provided in the imaging device, cancels the operation based on the gesture information. (14) The information processing device according to any one of (1) to (13), wherein the control unit controls operation of the head-mounted device based on the gesture information when no movement of the imaging device is detected. (15) The information processing device according to any one of (1) to (14), wherein the control unit controls operation of the portable terminal device based on the gesture information when the imaging device and a portable terminal device are connected. (16) The information processing device according to any one of (1) to (15), wherein the control unit controls operation of the imaging device based on a combination of the gesture information and a state of the imaging device status information. (17) The information processing device according to any one of (1) to (16), wherein the control unit determines the orientation of an image displayed on a display unit of the imaging device based on the orientation of the head-mounted device indicated by the gesture information and the orientation of the imaging device indicated by the imaging device status information. (18) The information processing device according to any one of (1) to (17), wherein the control unit cancels the operation based on the gesture information when the display unit of the imaging device is facing the subject side.(19) An information processing method for controlling at least an operation of an imaging device based on gesture information indicating a gesture detected by a head-mounted device worn on the head of a user and imaging device status information indicating a status of the imaging device. (20) An information processing device comprising: a control unit that determines whether or not to execute an operation on the head-mounted device based on a gesture detected by the head-mounted device worn on the head of a user based on the status of the imaging device.

[0156] REFERENCE SIGNS LIST 1 Information processing system 2 Imaging device (information processing device) 3 Headphones 26 Camera control unit 51 Headphone control unit 61 Camera state detection unit 62 Operation control unit 63 Notification control unit 71 Headphone gesture detection unit 72 Operation control unit 73 Notification control unit

Claims

1. An information processing device having a control unit that controls at least the operation of an imaging device based on gesture information indicating a gesture detected by a head-mounted device worn on a user's head and imaging device status information indicating the status of the imaging device.

2. The information processing device according to claim 1, wherein the control unit cancels an operation based on the gesture information when the gesture coincides with a movement of the imaging device.

3. The information processing device according to claim 1, wherein the control unit controls the operation of the imaging device and the head-mounted device based on the gesture information when the gesture does not match the movement of the imaging device.

4. The information processing device according to claim 1, wherein the control unit controls the operation of the imaging device based on the gesture information when a moving image is being captured by the imaging device.

5. The information processing device according to claim 1, wherein the control unit controls the operation of the head-mounted device based on the gesture information when the imaging device is not capturing a moving image.

6. The information processing device according to claim 1, wherein the control unit performs control to notify the user of information indicating control of an operation based on the gesture information.

7. The information processing device according to claim 6, wherein the control unit causes a display unit of the imaging device to display information indicating whether the control unit has controlled the operation of the head-mounted device or the imaging device.

8. The information processing device according to claim 6, wherein the control unit causes information indicating control of operation based on the gesture information to be displayed on a display unit of the imaging device in different display modes depending on whether the control unit controls operation of the head-mounted device based on the gesture information or whether the control unit controls operation of the imaging device based on the gesture information.

9. The information processing device according to claim 8, wherein the control unit causes the display unit to display an icon that resembles the head-mounted device or the imaging device that controls an operation based on the gesture information.

10. The information processing device according to claim 8, wherein the control unit, when controlling the operation of the imaging device based on the gesture information, causes the display unit of the imaging device to display the gesture information in a more emphasized manner than when controlling the operation of the head-mounted device based on the gesture information.

11. The information processing device of claim 6, wherein the control unit, when controlling the operation of the head-mounted device based on the gesture information, causes a notification sound to be output from an audio output unit of the imaging device, and, when controlling the operation of the imaging device based on the gesture information, does not cause a notification sound to be output from the audio output unit of the imaging device.

12. The information processing device according to claim 6, wherein the control unit, when capturing a moving image with the imaging device, does not cause a notification sound to be output from an audio output unit of the imaging device.

13. The information processing device according to claim 1, wherein the control unit cancels an operation based on the gesture information when a proximity sensor provided in the imaging device detects a user.

14. The information processing device according to claim 1, wherein the control unit controls the operation of the head-mounted device based on the gesture information when no movement of the imaging device is detected.

15. The information processing device according to claim 1, wherein the control unit, when the imaging device and a mobile terminal device are connected, controls an operation of the mobile terminal device based on the gesture information.

16. The information processing device according to claim 1, wherein the control unit controls the operation of the imaging device based on a combination of the gesture information and a state of the imaging device state information.

17. The information processing device according to claim 1, wherein the control unit determines a direction of an image to be displayed on a display unit of the imaging device based on a direction of the head-mounted device indicated by the gesture information and a direction of the imaging device indicated by the imaging device state information.

18. The information processing device according to claim 1, wherein the control unit cancels an operation based on the gesture information when a display unit of the imaging device faces a subject side.

19. An information processing method for controlling at least the operation of an imaging device based on gesture information indicating a gesture detected by a head-mounted device mounted on the user's head and imaging device status information indicating a status of the imaging device.

20. An information processing device comprising: a control unit that determines whether or not to execute an operation on a head-worn device based on a gesture detected by the head-worn device worn on a user's head, based on the state of an imaging device.

Citation Information

Patent Citations

  • Camera rotation control method based on lamp strip, intelligent sound box and storage medium

    CN111726708A

  • Control device and control method, and photographing control system

    JP2015192697A

  • Remote control method and terminal

    JP2017509034A

  • Information processing device, line-of-sight detection method, and program

    WO2022196093A1