Imaging device, and control method and program thereof.

Abstract

To provide an imaging apparatus that controls a start and an end of processing related to photographing according to the intention of a user.SOLUTION: An imaging apparatus includes: an operation member that receives an operation from a user; imaging means having a first function that executes still picture photographing in accordance with reception of a first operation to the operation member, a second function that starts recording of a moving image in accordance with the reception of a second operation to the operation member, and a third function that automatically starts recording of the moving image; and control means for performing control to stop the recording of the moving image by the third function and execute the still picture photographing by the first function in the case where the first operation is received via the operation member in a state where the third function is executed, and stop the recording of the moving image by the second function and not to execute the still picture photographing by the first function in the case where the first operation is received via the operation member in the state where the second function is executed.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an imaging device and a control method thereof.

Background Art

[0002] In recent years, a digital camera (so-called life-log camera) that automatically captures images at an appropriate timing without the user explicitly giving instructions is known. Even with such a camera, in order to be able to take pictures at the timing intended by the user, it has a function of executing a shooting process upon receiving an instruction from the user. For example, Patent Document 1 discloses a life-log camera that has a function of manually taking pictures in addition to the function of automatically taking pictures. Note that with such a life-log camera, not only still images but also moving images can be captured automatically and manually.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Life-logging cameras, as described above, are designed to be carried for extended periods, either hanging around the neck or attached to a part of the body. Therefore, portability is generally a priority. Thus, it's conceivable that multiple image-related functions, such as turning automatic shooting on and off, and manually starting and stopping still image and video recording, could be controlled using a single button. However, to distinguish and execute multiple functions with a single button press, it's anticipated that more complex operations, such as pressing twice in quick succession or holding down the button for a certain period, would be required to differentiate between functions. Furthermore, the same operation might trigger different processes depending on the state. As a result, complex operations might be required for users to initiate and terminate appropriate processes at their intended timing, potentially causing frustration.

[0005] Therefore, the present invention aims to control the start and end of the shooting process in a device that enables the start and end of the shooting process by operating a specific operating member, in a manner that more closely conforms to the user's intentions. [Means for solving the problem]

[0006] To achieve the above objective, the imaging device of the present invention is characterized by comprising: an operating member for receiving operations from a user; an imaging means having a first function for performing still image capture in response to receiving a first operation to the operating member; a second function for starting video recording in response to receiving a second operation to the operating member; and a third function for automatically starting video recording; and a control means for stopping video recording by the third function and performing still image capture by the first function when the first operation is received via the operating member while the third function is being performed, and stopping video recording by the second function and not performing still image capture by the first function when the first operation is received via the operating member while the second function is being performed. [Effects of the Invention]

[0007] According to the present invention, in a device that enables the start and end of shooting-related processing by operating a specific operating member, the start and end of shooting-related processing can be controlled more in line with the user's intentions. [Brief explanation of the drawing]

[0008] [Figure 1] This figure schematically shows the external appearance of a camera, which is one embodiment of the imaging device of the present invention. [Figure 2] A block diagram showing the overall configuration of a camera in one embodiment. [Figure 3] This diagram shows an example configuration of a wireless communication system between a camera and an external device. [Figure 4] This is a diagram showing the configuration of the external device. [Figure 5] This is a flowchart showing the automatic and manual image acquisition processes of the imaging device. [Figure 6] This is a diagram illustrating the division of areas within a captured image. [Figure 7A] This is a diagram illustrating the control of the shooting frequency. [Figure 7B] This is a diagram illustrating the control of the shooting frequency. [Figure 7C] This is a diagram illustrating the control of the shooting frequency. [Figure 7D] This is a diagram illustrating the control of the shooting frequency. [Modes for carrying out the invention]

[0009] One embodiment of the present invention will be described in detail below with reference to the attached drawings.

[0010] <Camera Configuration> Figure 1 is a schematic diagram showing the external appearance of a camera, which is one embodiment of the imaging device of the present invention. The camera 101 shown in Figure 1(a) is equipped with a power switch, an operating member for camera operation, and the like. The lens barrel 102, which integrally includes a group of photographic lenses and an image sensor as an imaging optical system for capturing an image of a subject, is movably attached to the fixed part 103 of the camera 101. Specifically, the lens barrel 102 is attached to the fixed part 103 via a tilt rotation unit 104 and a pan rotation unit 105, which are mechanisms that can rotate relative to the fixed part 103.

[0011] The tilt rotation unit 104 is equipped with a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in Figure 1(b), and the pan rotation unit 105 is equipped with a motor drive mechanism that can rotate the lens barrel 102 in the yaw direction shown in Figure 1(b). In other words, the camera 101 has a mechanism that rotates the lens barrel 102 in two axes. Each axis shown in Figure 1(b) is defined with respect to the position of the fixed part 103. The angular velocity meter 106 and accelerometer 107 are located on the fixed part 103 of the camera 101. Based on the output signals of the angular velocity meter 106 and accelerometer 107, vibrations of the camera 101 are detected, and the tilt rotation unit 104 and pan rotation unit 105 are rotated to correct the vibration or tilt of the lens barrel 102. The angular velocity meter 106 and accelerometer 107 also detect camera movement based on measurement results over a certain period of time.

[0012] Figure 2 is a block diagram showing the overall configuration of the camera 101 in this embodiment. In Figure 2, the first control unit 223 includes, for example, a CPU (MPU), memory (DRAM, SRAM), etc. It controls each block of the camera 101 and controls data transfer between each block by executing various processes according to the program stored in the non-volatile memory (EEPROM) 216. The non-volatile memory 216 is an electrically erasable and recordable memory, and as described above, it stores constants for the operation of the first control unit 223, programs, etc.

[0013] In FIG. 2, the zoom unit 201 includes a zoom lens that performs zooming (enlarging and reducing the imaged subject image). The zoom drive control unit 202 drives and controls the zoom unit 201 and detects the focal length at that time. The focus unit 203 includes a focus lens that performs focus adjustment (focusing). The focus drive control unit 204 drives and controls the focus unit 203. The imaging unit 206 includes an imaging element, receives light incident through each lens group, and outputs information on electric charges corresponding to the amount of that light as an analog image signal to the image processing unit 207. Note that the zoom unit 201, the focus unit 203, and the imaging unit 206 are arranged inside the lens barrel 102.

[0014] The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data obtained by A / D converting the analog image signal, and outputs the digital image data after the application. The digital image data output from the image processing unit 207 is converted into a recording format such as the JPEG format by the image recording unit 208 and stored in the memory 215 or transmitted to the video output unit 217 described later. Note that the function of the image processing unit 207 may be incorporated in the first control unit 223.

[0015] The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to rotate the lens barrel 102 in the tilt and pan directions. The lens barrel rotation drive unit 205 is an electronic circuit for controlling each rotation unit. The function of the lens barrel rotation drive unit 205 may be incorporated in the first control unit 223.

[0016] The device shake detection unit 209 includes an angular velocity meter (gyro sensor) 106 that detects the angular velocity of the camera 101 in three axial directions and an accelerometer (acceleration sensor) 107 that detects the acceleration of the camera 101 in three axial directions. Then, based on the signals detected by those sensors, the rotation angle of the device, the shift amount of the device, and the like are calculated.

[0017] The audio input unit 213 acquires audio signals around the camera 101 through a microphone provided in the camera 101, converts them into digital audio signals, and transmits them to the audio processing unit 214. The audio processing unit 214 is a digital signal processing circuit that performs audio-related processing such as normalization processing of the input digital audio signals. The audio signals processed by the audio processing unit 214 are transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily stores the image signals and audio signals obtained by the image processing unit 207 and the audio processing unit 214. Note that the function of the audio processing unit 214 may be built into the first control unit 223.

[0018] The image processing unit 207 and the audio processing unit 214 read out the image signals and audio signals temporarily stored in the memory 215, perform encoding of the image signals, encoding of the audio signals, etc., and generate compressed image signals and compressed audio signals. The first control unit 223 transmits these compressed image signals and compressed audio signals to the recording and playback unit 220.

[0019] The recording and playback unit 220 is a digital signal processing circuit that records the compressed image signals, compressed audio signals, and other control data related to shooting generated by the image processing unit 207 and the audio processing unit 214 on the recording medium 221. Also, when the audio signal is not compressed and encoded, the first control unit 223 transmits the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording and playback unit 220 to record them on the recording medium 221. Note that the function of the recording and playback unit 220 may be built into the first control unit 223.

[0020] The recording medium 221 may be a recording medium built into the camera 101 or a removable recording medium, and can record various data such as compressed image signals, compressed audio signals, and audio signals generated by the camera 101. Generally, a medium with a larger capacity than the non-volatile memory 216 is used for the recording medium 221. For example, the recording medium 221 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

[0021] The recording and playback unit 220 reads (plays back) the compressed image signal, compressed audio signal, audio signal, various data, and program recorded on the recording medium 221. The first control unit 223 then transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing unit 207 and the audio processing unit 214 temporarily store the compressed image signal and compressed audio signal in the memory 215, decode them according to a predetermined procedure, and transmit the decoded signals to the video output unit 217.

[0022] The audio input unit 213 is equipped with multiple microphones, and the audio processing unit 214 can detect the direction of sound relative to the plane on which the multiple microphones are installed. This is used for subject search and automatic shooting, which will be described later. Furthermore, the audio processing unit 214 detects specific voice commands. In addition to several pre-registered commands, the system may also be configured to allow users to register specific voices to the camera. The system also performs sound scene recognition. In sound scene recognition, a network trained by machine learning based on a large amount of audio data is used to determine sound scenes. For example, the audio processing unit 214 is configured with a network to detect specific scenes such as "cheers," "applause," and "speaking," and detects specific sound scenes and specific voice commands. When the audio processing unit 214 detects a specific sound scene or specific voice command, it outputs a detection trigger signal to the first control unit 223 and the second control unit 211.

[0023] In addition to the first control unit 223, which controls the entire main system of camera 101, a second control unit 211 is provided to control the power supply to the first control unit 223. The first power supply unit 210 and the second power supply unit 212 supply power to operate the first control unit 223 and the second control unit 211, respectively. When the power button on camera 101 is pressed, power is initially supplied to both the first control unit 223 and the second control unit 211. However, as will be described later, the first control unit 223 also controls the power supply to the first power supply unit 210 to turn off its own power supply. Even when the first control unit 223 is not operating, the second control unit 211 is operating and receives information from the device vibration detection unit 209 and the audio processing unit 214. Based on the various input information, the second control unit 211 determines whether or not to start the first control unit 223, and if it is determined to start it, it instructs the first power supply unit 210 to supply power to the first control unit 223.

[0024] The audio output unit 218 outputs a pre-set audio pattern from a speaker built into the camera 101, for example, during shooting. The LED control unit 224 lights up the LEDs provided on the camera 101 based on a pre-set lighting pattern or flashing pattern, for example, during shooting. The video output unit 217 consists of, for example, a video output terminal and outputs an image signal to display video on a connected external display or the like. The audio output unit 218 and the video output unit 217 may also be combined into a single terminal, such as an HDMI (registered trademark): High-Definition Multimedia Interface terminal.

[0025] The communication unit 222 is the part that communicates between the camera 101 and an external device, and transmits and receives data such as audio signals, image signals, compressed audio signals, and compressed image signals. It also receives control signals related to various shooting functions such as start and end shooting commands, pan function, tilt function, and zoom function, and drives the camera 101 based on instructions from the external device. It also transmits and receives information such as various parameters related to learning, which are processed by the learning processing unit 219 described later, between the camera 101 and the external device. The communication unit 222 includes wireless communication modules such as an infrared communication module, a Bluetooth® communication module, a wireless LAN communication module, a WirelessUSB®, and a GPS receiver.

[0026] As will be described later, the camera 101 of this embodiment can establish a wireless communication connection with an external device 301 via the communication unit 222 using a Bluetooth communication method and a wireless LAN communication method.

[0027] The camera control unit 225 includes operation buttons provided on the camera 101. The operation buttons include a power button for instructing the camera 101 to be powered on or off, and a communication button for instructing the camera 101 to initiate wireless communication with the external device 301.

[0028] When the power button is operated, power is supplied to both the first control unit 223 and the second control unit 211, and the processing of the <imaging operation sequence> described later is started.

[0029] When the communication button is operated after the camera 101 has been powered on, a process is executed to establish a wireless connection between the imaging device 101 and the external device 301. After the connection is established, various operations become possible from a dedicated application on the external device 301.

[0030] Furthermore, the camera 101 in this embodiment prioritizes portability and therefore does not have a display.

[0031] <Communication with external devices> Figure 3 shows an example of a wireless communication system configuration between camera 101 and external device 301. Camera 101 is a digital camera with a shooting function, and external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

[0032] Camera 101 and external device 301 can communicate via a first communication 302, for example, a wireless LAN compliant with the IEEE 802.11 standard series. In addition, they can communicate via a second communication 303, such as Bluetooth Low Energy (hereinafter referred to as "BLE"), which has a master-slave relationship between a control station and a slave station. Note that wireless LAN and BLE are just examples of communication methods, and each communication device may have two or more communication functions, and other communication methods may be used as long as one communication function that communicates in a relationship between a control station and a slave station can control the other communication function. However, the first communication 302, such as wireless LAN, is capable of faster communication than the second communication 303, such as BLE, and the second communication 303 consumes less power or has a shorter communication range than the first communication 302, or at least one of the other.

[0033] Here, the configuration of the external device 301 will be explained using Figure 4. The external device 301 includes, for example, a wireless LAN control unit 401 for wireless LAN and a BLE control unit 402 for BLE, as well as a public wireless control unit 406 for public wireless communication. The external device 301 also has a packet transceiver unit 403. The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, driver processing to perform various controls for wireless LAN communication compliant with the IEEE 802.11 standard series, and protocol processing related to wireless LAN communication. The BLE control unit 402 performs RF control of BLE, communication processing, driver processing to perform various controls for BLE communication, and protocol processing related to BLE communication. The public wireless control unit 406 performs RF control of public wireless communication, communication processing, driver processing to perform various controls for public wireless communication, and protocol processing related to public wireless communication. Public wireless communication is compliant with standards such as IMT (International Multimedia Telecommunications) and LTE (Long Term Evolution). The packet transmission / reception unit 403 performs processing to perform at least one of the following: transmission and reception of packets related to wireless LAN, BLE, and public wireless communication. In this embodiment, the external device 301 is described as performing at least one of the following in communication: transmission and reception of packets. However, other communication methods besides packet switching, such as circuit switching, may be used.

[0034] The external device 301 further includes, for example, a control unit 411, a storage unit 404, a GPS receiver 405, a display unit 407, an operation unit 408, an audio input / audio processing unit 409, and a power supply unit 410. The control unit 411 controls the entire external device 301 by, for example, executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, the control program executed by the control unit 411 and various information such as parameters necessary for communication. Various operations described later are realized by the control unit 411 executing the control program stored in the storage unit 404.

[0035] The power supply unit 410 supplies power to the external device 301. The display unit 407 has the function of outputting visually recognizable information, such as an LCD or LED, or outputting sound, such as from a speaker, and displays various information. The operation unit 408 includes, for example, buttons that accept user operation of the external device 301. Note that the display unit 407 and the operation unit 408 may be composed of common components, such as a touch panel.

[0036] The voice input voice processing unit 409 may be configured to acquire voice spoken by the user using, for example, a general-purpose microphone built into an external device 301, and to identify the user's operation commands through voice recognition processing. Alternatively, a dedicated application within the external device 301 can be used to acquire voice commands based on the user's pronunciation, and these can be registered as specific voice commands to be recognized by the voice processing unit 214 of the camera 101 via the first wireless LAN communication 302.

[0037] The GPS (Global Positioning System) receiver 405 receives GPS signals from satellites, analyzes the GPS signals, and estimates the current position (longitude and latitude information) of the external device 301. Alternatively, it may use WPS (Wi-Fi Positioning System) or the like to estimate the current position of the external device 301 based on information from surrounding wireless networks. If the acquired current GPS position information is within a pre-set position range (within a predetermined radius centered on the detected position), or if there is a position change of more than a predetermined amount in the GPS position information, it notifies the camera 101 of the movement information via the BLE control unit 402. This information is then used as parameters for automatic shooting and automatic editing, as described later.

[0038] As described above, the camera 101 and the external device 301 exchange data through communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, they send and receive data such as audio signals, image signals, compressed audio signals, and compressed image signals. The external device 301 also sends instructions to the camera 101 to take pictures, sends voice command registration data, sends notifications of detection of a predetermined location based on GPS location information, and sends notifications of changes in location. Furthermore, it also sends and receives training data using a dedicated application within the external device 301. The dedicated application also plays a role in displaying a GUI for viewing images in the camera and changing camera settings via the display of the external device 301, on behalf of the camera 101 which does not have a display.

[0039] Let's return to the explanation of Figure 3.

[0040] The camera 101 in this embodiment has two procedures for connecting to the external device 301 via wireless LAN.

[0041] One method is to switch from Bluetooth Low Energy to Wi-Fi.

[0042] When the camera 101 is powered on (powered up), and the communication button is briefly pressed, the camera 101 starts sending BLE advertisement packets. The external device 301 receives these advertisement packets by scanning them and responds by sending a connection request to the camera 101. When the camera 101 accepts this request, the camera 101 and the external device 301 are connected via BLE. Once connected, the camera 101 and the external device 301 remember each other's information. This state is called the paired state. When the camera 100 is powered on while in the paired state, it can start sending advertisements without waiting for the communication button to be pressed, and can automatically establish a BLE connection with the external device 301 that scans and receives these advertisements.

[0043] Furthermore, during the pairing process, camera 101 shares the wireless LAN communication parameters (SSID and password) generated by camera 101 with the external device 301 via BLE. The shared SSID and password are stored on the external device 301 until pairing is unpaired.

[0044] Additionally, you can unpair the camera 101 by simultaneously pressing and holding the power button and communication button on the camera 101.

[0045] Furthermore, when the camera 101 and the external device 301 are connected via BLE and a dedicated application is launched on the external device 301 and displayed in the foreground, the external device 301 requests the camera 101 to activate the wireless LAN via BLE, according to the control of the dedicated application. Upon receiving this request, the camera 101 terminates the BLE communication and begins transmitting a wireless LAN beacon. The beacon contains the SSID shared via BLE. Upon receiving this beacon, the external device 301 sends a request to the camera 101 to join the wireless LAN generated by the camera 101. Once this join request is accepted, the external device 301 joins the wireless LAN generated by the camera 101, and after the connection establishment process at the application layer, a wireless LAN connection between the camera 101 and the external device 301 is established.

[0046] The above is one method of connection.

[0047] Next, I will describe another connection method.

[0048] Camera 101 has a function to generate a wireless LAN without using Bluetooth, assuming that the external device 301 is a device that does not have Bluetooth. When camera 101 is powered on and not connected to external device 301, the user can change camera 101 to a mode that generates a wireless LAN without using Bluetooth by long-pressing the communication button. By long-pressing the communication button, camera 101 changes mode and powers off. The user then turns camera 101 on again by operating the power button and then short-presses the communication button. In this case, camera 101 does not start sending BLE advertisement packets, but instead activates the wireless LAN function and starts transmitting beacons. In this case, the default fixed SSID is included in the beacon. The user of external device 301 inputs this SSID using the input component of external device 301, and external device 301 sends a request to camera 101 to join the wireless LAN with the input SSID. Once this participation request is accepted, the external device 301 joins the wireless LAN generated by the camera 101, and after the connection establishment process at the application layer, a wireless LAN connection is established between the camera 101 and the external device 301. Even if the camera 101 is already paired with the external device 301 or another smartphone, the mode can be switched by long-pressing the communication button, allowing connection via wireless LAN only.

[0049] The above is an explanation of the other connection method.

[0050] In this embodiment, after wireless communication via WLAN with the external device 301 is established, the communication button can be assigned the role of a shooting instruction button for inputting shooting instructions. This assignment is performed through the GUI of a dedicated application installed on the external device 301. When the camera 101 receives an instruction from the dedicated application on the external device 301 to assign the communication button the role of a shooting instruction button for inputting shooting instructions, it enables the function of receiving shooting instructions at the communication button.

[0051] However, as mentioned above, if camera 101 is not paired or is in a mode that generates a wireless LAN without using Bluetooth, the communication button needs to function in its original role. Therefore, even if the communication button is assigned the role of receiving shooting commands, if camera 101 is not paired or is in a mode that generates a wireless LAN without using Bluetooth, that setting will be ignored. In other words, the communication button will revert to its original role.

[0052] <Sequence of imaging operations> Figure 5 is a flowchart showing the automatic and manual shooting processes of the camera 101 in this embodiment.

[0053] In the camera 101 of this embodiment, the processing of this flowchart begins when the user operates the power button on the camera 101 from on to off. Here, it is assumed that the communication button on the camera 101 has been pre-assigned the function of receiving shooting instructions, and that the communication button functions as a shooting instruction button. Furthermore, the processing of each step in the following flowchart is realized by the first control unit 223 controlling each part of the camera 101. The automatic shooting function can also be turned on and off by operation from a dedicated application on an external device 301.

[0054] First, in step S501, the image unit 206 receives an image, and the image processing unit 207 processes the captured signal to generate an image for subject detection. Furthermore, subject detection processing is performed on the generated image to detect people, objects, etc.

[0055] When detecting people, the system detects the face or body of the subject. In face detection processing, a predetermined pattern is used to identify a person's face, and areas in the captured image that match this pattern can be detected as the face region.

[0056] Furthermore, a confidence score indicating the likelihood that the subject is indeed a face is calculated simultaneously. This confidence score is calculated based on factors such as the size of the face region within the image and the degree of matching with the face pattern. Similarly, for object recognition, objects that match pre-registered patterns can be recognized.

[0057] Another method involves extracting feature subjects using histograms of hue, saturation, etc., within the captured image. For images of subjects captured within the field of view, the distribution derived from the histogram of hue, saturation, etc., is divided into multiple intervals, and the captured images are classified according to each interval. For example, histograms of multiple color components are created for the captured image, and the image is divided into bell-shaped distribution ranges. Images are classified within regions belonging to the same combination of intervals, and the image region of the subject is recognized. By calculating an evaluation value for each recognized image region of the subject, the image region of the subject with the highest evaluation value can be determined as the main subject region. In this way, information on each subject can be obtained from the captured information.

[0058] In step S502, the image blur correction amount is calculated. Specifically, the absolute angle of camera shake is first calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the tilt rotation unit 104 and the pan rotation unit 105 are moved in an angular direction that cancels out that absolute angle to determine the angle at which image blur is corrected, and this is defined as the image blur correction amount. Note that the image blur correction amount calculation process here can be changed by the learning process described later.

[0059] Step S503 determines the state of the camera. Based on the camera angle and camera movement detected using angular velocity information, acceleration information, and GPS position information, the system determines the current vibration / movement state of the camera. For example, when camera 101 is mounted on a car and taking pictures, subject information such as the surrounding scenery changes significantly depending on the distance traveled. Therefore, the system determines whether the camera is in a "vehicle movement state" (mounted on a car or other vehicle moving at high speed) and uses this determination for automatic subject search, which will be explained later. The system also determines whether the camera angle has changed significantly and whether it is in a "stationary shooting state" (camera 101 is almost stationary) with little shaking. If it is in a "stationary shooting state," it can be assumed that the position of camera 101 itself has not changed, so subject search for stationary shooting can be performed. If the camera angle has changed relatively significantly, it is determined to be in a "handheld state," and subject search for handheld shooting can be performed.

[0060] In step S504, subject search processing is performed. Subject search consists of the following processes.

[0061] (1) Area division Let's explain area division using Figure 6. As shown in Figure 6(a), the area is divided all around the camera position (with the origin O as the camera position). In the example in Figure 6(a), the tilt direction and pan direction are each divided into 22.5-degree intervals. When divided as in Figure 6(a), as the tilt angle moves away from 0 degrees, the horizontal circumference decreases, and the area decreases. Therefore, as shown in Figure 6(b), when the tilt angle is 45 degrees or more, the horizontal area range is set to be larger than 22.5 degrees.

[0062] Figures 6(c) and 6(d) show examples of areas divided within the field of view. Axis 1301 is the orientation of the camera 101 at initialization, and area division is performed using this direction as the reference position. 1302 indicates the field of view area of ​​the captured image, and an example of the image at that time is shown in Figure 6. Within the captured field of view, the image is divided based on the area division as shown by reference numerals 1303 to 1318 in Figure 6.

[0063] (2) Calculation of importance level for each area For each area divided as described above, an importance level is calculated to indicate the priority of the search, depending on the condition of the subjects and the scene within the area. The importance level based on the condition of the subjects is calculated based on, for example, the number of people in the area, the size of their faces, the direction of their faces, the certainty of face detection, their facial expressions, and the results of personal identification of the people. The importance level based on the scene is calculated based on, for example, the results of general object recognition, the results of scene discrimination (blue sky, backlight, sunset, etc.), the level of sound coming from the direction of the area, the results of speech recognition, and motion detection information within the area.

[0064] Furthermore, in the camera state determination (step S503) shown in Figure 5, if camera vibration is detected, the importance level can be changed according to the vibration state. For example, if it is determined to be in a "placed shooting state," the system will determine that subject search will be performed mainly on high-priority subjects registered by face recognition (e.g., the camera owner). Also, automatic shooting, as described later, will prioritize the face of the camera owner, for example. This means that even if the camera owner spends a lot of time wearing and carrying the camera and taking pictures, they can still capture many images of themselves by removing the camera and placing it on a table or other surface. In this case, since face search is possible by pan and tilt, images of the owner and group photos with many faces can be captured simply by placing the camera appropriately, without having to consider the camera's angle.

[0065] However, with only the above conditions, unless there are changes in each area, the area with the highest importance level will remain the same, and as a result, the areas being searched will never change. Therefore, the importance level is changed according to past photographic information. Specifically, areas that have been continuously designated as search areas for a predetermined period of time may have their importance level lowered, or areas that were photographed in step S513 described later may have their importance level lowered for a predetermined period of time.

[0066] (3) Determining the area to be searched Once the importance level of each area has been calculated as described above, the areas with the highest importance levels are selected as the areas to be explored. Then, the pan and tilt target angles required to capture the areas to be explored are calculated.

[0067] Returning to the explanation in Figure 5, in step S505, pan-tilt drive is performed. Specifically, the pan-tilt drive amount is calculated by adding the image blur correction amount at the control sampling frequency and the drive angle based on the pan-tilt search target angle. Then, the lens barrel rotation drive unit 205 drives and controls the tilt rotation unit 104 and the pan rotation unit 105, respectively.

[0068] In step S506, the zoom unit 201 is controlled to perform zoom operation. Specifically, the zoom is driven according to the state of the subject to be searched, which was determined in step S504. For example, if the subject to be searched is a person's face, if the face in the image is too small, it may fall below the minimum detectable size and may not be detected, potentially causing it to be lost. In such cases, the zoom is controlled to increase the size of the face in the image by zooming towards the telephoto side. On the other hand, if the face in the image is too large, the subject may easily move out of the field of view due to the movement of the subject or the camera itself. In such cases, the zoom is controlled to decrease the size of the face on the screen by zooming towards the wide-angle side. By performing zoom control in this way, a state suitable for tracking the subject can be maintained.

[0069] In step S507, it is determined whether or not an operation has been performed to instruct the shooting instruction member to take a picture. If an operation has been performed, the process proceeds to step S508. In this embodiment, the camera 101 is equipped with a button exposed on the outside of the housing, and in this step, this button is used as the shooting instruction member. The user can instruct the camera to take a still image by short-pressing this button. Furthermore, by long-pressing this button, the user can instruct the camera to start recording a video.

[0070] In step S508, it is determined whether or not camera 101 is recording video. If it is recording video, the process proceeds to step S509. If it is not recording video (i.e., it is in standby mode), the process proceeds to step S511. Note that if video is recording, it is either video recording started by automatic shooting in step S516 (described later) or video recording started by user instruction in step S513 (described later).

[0071] In step S509, the video recording is stopped and the recorded video file is closed.

[0072] Furthermore, it is determined whether the trigger that started the video recording that was stopped in step S510 was automatic shooting. In other words, it is determined whether the video recording was started in S516.

[0073] First, let's explain the case where, in step S510, it is determined that the stopped video recording was started by automatic shooting. In this case, we proceed to step S511, where it is determined whether the operation of the shooting instruction member determined in step S507 was a short press operation.

[0074] If, in step S511, it is determined that the operation of the shooting instruction member is a short press operation, it is determined that a still image capture has been instructed, and a still image capture is performed in step S512. On the other hand, if it is determined that the operation of the shooting instruction member is not a short press operation (i.e., it is determined to be a long press operation), video recording is started in step S513. Video recording started here continues until it is stopped by user operation or until the recording time reaches 29 minutes and 59 seconds.

[0075] Thus, if the video recording stopped in step S510 was initiated by automatic shooting, the user is highly likely to be unaware that video recording was in progress. Therefore, it is assumed that the user is operating simply with the intention of taking a picture. Accordingly, in this embodiment, if the user manually inputs a shooting command while video recording started by automatic shooting is in progress, the system will not only stop the video but also take a still image or restart video recording.

[0076] On the other hand, if it is determined in step S510 that the video recording is not by automatic shooting (but is by manual video recording), the system returns to S501 without going through S511 to S513. In this way, if the stopped video recording was started manually, the system does not take a still image or restart the video recording, but simply stops the video recording. If the video recording stopped in step S510 was started by a manual instruction, the user is aware that video recording is currently in progress. In other words, the user's instruction may be made with the intention of stopping the video recording. Therefore, in this embodiment, if the user manually inputs a shooting instruction while a manually started video is being recorded, the system simply stops the video and does not take a picture as instructed.

[0077] If it is determined in step S507 that no instruction was given, the process proceeds to step S514, where an automatic shooting determination process is executed to determine whether or not to perform automatic shooting. In the automatic shooting determination process, a determination is made as described below regarding whether or not to perform automatic shooting. At this time, autofocus control is performed by the focus drive control unit 204. In addition, exposure control is performed using an aperture control unit, a sensor gain control unit, and a shutter control unit (not shown) to ensure that the subject is at an appropriate brightness. Furthermore, after shooting, the image processing unit 207 performs various known image processing, such as auto white balance processing, noise reduction processing, and gamma correction processing, to generate an image.

[0078] If it is determined in step S514 to perform automatic shooting, the process proceeds to step S515, and the shooting of still images or video recording begins. If video recording is started here, the recording of the automatically started video will continue until it is stopped by user operation or until 30 seconds have elapsed since the start of recording. Here, an example is given in which the recording stops after a shorter time than when manually started video recording. This is based on the idea that when manually started, the user's intention to continue recording is prioritized, while when automatically started, the priority is to suppress problems such as processing load and heat generation of the casing, but this is not the only option. For example, if the casing can employ sufficient processing speed and sufficient heat dissipation mechanism, the recording of automatically started video may continue until it reaches 29 minutes and 59 seconds, just like the recording of manually started video. Alternatively, in either case, the recording may be limited to 30 seconds to prioritize the reduction of processing load.

[0079] Furthermore, during this shooting process, if certain conditions are met, the camera may notify the person being photographed that it is about to take a picture before shooting. Methods of notification may include, for example, sound output from the audio output unit 218 or LED illumination by the LED control unit 224. The predetermined conditions include, for example, the number of faces within the field of view, the degree of smiles, the degree of eye closure, the gaze angle and face angle of the subject, the face recognition ID number, the number of registered individuals, the general object recognition result at the time of shooting, the scene discrimination result, and the elapsed time since the previous shooting. It also includes the shooting time, whether the current location based on GPS information is a scenic spot, the sound level at the time of shooting, the presence or absence of a person speaking, and whether applause or cheering is present. Additionally, it includes vibration information (acceleration information, camera status), environmental information (temperature, atmospheric pressure, illuminance, humidity, ultraviolet radiation), etc. By performing notification shooting based on these conditions, it is possible to capture desirable images with the subject looking directly at the camera in scenes of high importance. Furthermore, in order to generate a suitable balance of still images and videos, if the number of still image captures reaches a predetermined number, video recording for a certain period of time may be initiated at the next shooting timing. Alternatively, video recording may be initiated when a timing is detected that should record changes in the subject's temporal movement.

[0080] <Determination of whether or not to perform automatic shooting> The determination of whether or not to perform automatic shooting (a shooting operation that records image data output by the imaging unit) is made as follows. Specifically, it is determined that automatic shooting will be performed in the following two cases. Specifically, based on the importance level for each area obtained in the subject search process in step S504, it is determined that automatic shooting will be performed if the importance level exceeds a predetermined value. Note that the recording referred to here may be the recording of image data to memory 215 or the recording of image data to non-volatile memory 216. It also includes the automatic transfer of images to external device 301 and the recording of image data on the external device 301 side.

[0081] In this embodiment, the system is controlled to automatically take pictures using the automatic shooting determination process as described above. However, depending on the situation at the location and the condition of the camera, it may be better to change the automatic shooting determination parameters.

[0082] Unlike shooting at fixed time intervals, automatic shooting control based on situational judgment tends to favor the following: (1) I want to take a large number of pictures, including people and objects. (2) I don't want to miss capturing memorable moments. (3) I want to shoot in an energy-saving manner, taking into consideration the remaining battery level and the remaining capacity of the recording media. Automatic shooting calculates an evaluation value from the subject's state, compares it to a threshold, and performs automatic shooting if the evaluation value exceeds the threshold. The evaluation value for automatic shooting is determined by the learning results up to that point.

[0083] In the example shown in Figure 7A, the horizontal axis represents the passage of time, and the vertical axis represents the threshold for automatic shooting. The system is designed so that the threshold gradually decreases if no shooting occurs for a certain period of time. This adjustment is made because, even if the subject's situation is not necessarily ideal for shooting, performing a certain number of shooting actions ensures that shooting is carried out fairly evenly throughout the entire shooting experience.

[0084] In the example in Figure 7A, the elapsed time is measured from the point where shooting is completed. After the elapsed time T3, the threshold is gradually lowered from the initial value TH_DEFAULT, and as more time passes, the judgment threshold is lowered to TH_MIN. Shooting at fixed time intervals regardless of the subject's condition may result in footage that is far removed from what the user wants to capture. By gradually lowering the judgment threshold, it is possible to control the process to more closely match the image the user wants to capture.

[0085] Furthermore, the example in Figure 7B shows the case where the shooting operation is performed at elapsed time T1. The change in the evaluation value of automatic shooting, which is stored as shooting history information, is evaluated, and if there is a downward trend or little change, the judgment threshold is set to TH_MAX. Furthermore, the judgment threshold is gradually lowered as time progresses.

[0086] On the other hand, if the evaluation value is observed to be increasing during the imaging at elapsed time T1, the judgment threshold will move in the same manner as in Figure 7A, and the judgment threshold will maintain its initial value TH_DEFAULT.

[0087] The CPU controlling camera 101 has a detection unit that detects the subject's face based on image information. Furthermore, it has a determination unit that recognizes facial expressions and calculates an evaluation value based on the state of a specific expression (for example, when the characteristic value of a state such as joy, sadness, anger, or surprise exceeds a threshold) to determine whether to take an automatic shot. Furthermore, it has a control unit that performs subject recording operations (automatic shooting) according to the determination result of the determination unit. In this case, the automatic shooting determination threshold is adjusted according to the passage of time and the evaluation value. This adjustment allows for the capture of various expressions by maintaining the determination threshold when the evaluation value calculated by the determination unit is increasing, thereby preventing missed shots. On the other hand, if the evaluation value does not change much or is decreasing, the determination threshold is temporarily increased to control the system so that the shooting operation is not performed. This helps to suppress missed shots and reduce the amount of recording memory that can be used.

[0088] Another control example is explained using Figure 7C. In Figure 7C, the judgment threshold for the period from the time the shooting operation is performed until the elapsed time T1 is set to TH_MAX. This adjustment helps to reduce the problem of too many similar images being produced when shooting operations are performed consecutively immediately after shooting. Also, Figure 7D shows a control example in which the threshold is changed according to the detection result of the camera 101's shaking state. When camera 101 is worn as a wearable camera, the shooting decision based solely on the automatic shooting judgment may not be able to connect the actions in chronological order. In Figure 10E, the threshold is lowered after the elapsed time T3, and the threshold is changed to zero at the elapsed time T4. In other words, if no shooting operation is performed until the elapsed time T4, the shooting operation will be performed regardless of the evaluation value of automatic shooting.

[0089] By controlling the shooting frequency according to the shooting conditions in this way, it is possible to perform automatic shooting that obtains an appropriate number of shots.

[0090] In the above description, the shooting frequency was controlled to change depending on the shooting conditions, but it is also possible to maintain and control separate thresholds for determining whether to take a still image and for determining whether to start video recording.

[0091] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0092] (Other embodiments) In the above-described embodiment, the shooting instruction member is a button, and its operation is explained using a short press as an example, but it is not limited to this. For example, instead of a button operation, a light tap on the camera housing with a finger or the like may be used as a shooting instruction. A shooting instruction triggered by a tap operation is determined when the user taps the camera housing, and the device shake detection unit 209, which includes an acceleration sensor, detects a continuous high-frequency acceleration over a short period of time. For example, a single tap on the camera housing may be used to instruct manual still image shooting, and a double tap may be used to instruct manual video recording, etc. Furthermore, regarding manual shooting instructions, instructions are given by short presses and long presses of the operation button on the camera 101, but the instruction method may also be implemented in a way other than that of this embodiment.

[0093] Alternatively, a touch sensor may be provided on a part of the surface of the camera 101's housing, and a shooting command may be received by accepting a specific touch operation using this sensor.

[0094] Furthermore, in the above-described embodiment, an example was given in which, if a long press operation is performed on the operating member during automatically started video recording, the video recording is stopped and then restarted in response to the operation. However, it is not limited to this. For example, it is assumed that users have lower demands regarding the timing of capture for videos compared to still images. Therefore, if a long press operation is performed on the operating member during video recording, regardless of whether the video recording was started automatically or manually, the video recording may simply be stopped in either case. In this case as well, if the video recording is stopped by a short press on the operating member to instruct the capture of a still image, then, as in the above-described embodiment, the method of stopping the video recording or stopping it and then taking a still image may differ depending on whether the stopped video recording was started automatically or manually.

[0095] Furthermore, in the embodiments described above, cameras as shown in Figures 1(a) and 3 were used as examples. However, other shapes of wearable devices may be used. For example, a device with a glasses-like casing and a built-in camera that faces the same direction as the user's face may be used. Alternatively, a head-mounted display may be used. In that case, an optical see-through type head-mounted display may be used, which uses an optical system such as a prism or half-mirror to superimpose images captured by the camera onto the surrounding scenery seen through the lenses. Or, a video see-through type head-mounted display may be used, which completely blocks the user's field of vision from the outside world and instead displays images captured by the camera on a display.

[0096] Furthermore, the camera 101 may also be equipped with a function to receive shooting instructions by voice. When a shooting instruction is given by voice, it is clear that the user intends to take a picture. Therefore, if a shooting instruction is received by voice while video is being recorded, the video recording will be stopped and the shooting process will be executed, regardless of whether the video recording was started automatically or manually. In addition, if the user operates the shooting instruction component while video recording has been started by voice instruction, the video recording may be stopped and the shooting process will be executed.

[0097] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. An operating component that receives input from the user, An imaging means having a first function that performs still image capture in response to a first operation on the operating member, a second function that starts recording video in response to a second operation on the operating member, and a third function that automatically starts recording video. An imaging device characterized by having control means that, when the third function is being executed and the first operation is received via the operating member, stops the recording of video by the third function and performs still image capture by the first function, and when the second function is being executed and the first operation is received via the operating member, stops the recording of video by the second function and does not perform still image capture by the first function.

2. The imaging apparatus according to claim 1, further comprising control means for stopping the recording of video by the third function and starting the recording of video by the second function when the second operation is received via the operating member while the third function is being performed, and stopping the recording of video by the second function and not starting the recording of video by the second function when the second operation is received via the operating member while the second function is being performed.

3. The imaging apparatus according to claim 1, further comprising control means for stopping the recording of video by the third function and preventing the start of video recording by the second function when the second operation is received via the operating member while the third function is being performed, and for stopping the recording of video by the second function and preventing the start of video recording by the second function when the second operation is received via the operating member while the second function is being performed.

4. It also has the ability to connect to external devices wirelessly, The operating member is a button used for wireless communication with the external device, The imaging apparatus according to claim 1, characterized in that the control means controls the function of accepting the first operation and the second operation via the button in response to instructions from the external device.

5. A first communication means capable of communicating with an external device using a first communication method, The system further comprises a second communication means capable of communicating with the external device using a second communication method that is faster than the first communication method, The imaging apparatus according to claim 1, characterized in that the second communication means has the function of communicating with the external device using the second communication method with the external device using communication parameters shared with the external device via the first communication means.

6. The imaging device according to claim 1, characterized in that the automatically started video recording is automatically stopped after a predetermined time has elapsed from the start.

7. The imaging device according to claim 1, characterized in that the recording of a video, which is started in response to a second operation on the operating member, is automatically stopped after a predetermined time has elapsed from the start.

8. The video recording, which was started automatically, will automatically stop once a certain amount of time has elapsed since the start. The video recording, which was started in response to the reception of a second operation on the aforementioned operating member, automatically stops after a second period of time has elapsed since the start. The imaging apparatus according to claim 1, characterized in that the first time is shorter than the second time.

9. It also has a function to accept voice-activated shooting instructions, The imaging device according to claim 1, characterized in that, when a shooting instruction is received by an audio signal during video recording, the recording of the video is stopped regardless of whether the video recording was started by the second function or the third function, and then shooting is performed based on the shooting instruction by the audio signal.

10. The imaging device according to claim 9, characterized in that, when a first operation is received on the operating member during the recording of a video initiated by a voice command, the recording of the video is stopped and still image capture is performed using the first function.

11. The imaging means further includes panning, tilting, and zooming functions. The imaging device according to claim 1, characterized in that the imaging means tracks a subject using at least one of the pan function, the tilt function, and the zoom function.

12. The imaging device according to claim 1, characterized in that the imaging device does not have a display and has a function to transmit captured images to an external device.

13. The imaging device according to claim 1, characterized in that the imaging device is an optical see-through type head-mounted display that overlays an image captured by the imaging means onto the user's field of view using an optical system, or a video see-through type head-mounted display that displays an image captured by the imaging means on a display.

14. The imaging device according to claim 1, characterized in that the third function automatically starts recording a video when predetermined conditions are met based on an image captured by the imaging means.

15. A control method for an imaging device having an operating member and an imaging means, A first function which performs still image capture in response to receiving a first operation on the operating member, A second function that starts recording a video in response to receiving a second operation on the aforementioned operating member, It has a third function that automatically starts recording video, If the first operation is received via the operating member while the third function is being executed, the system controls the system to stop recording video by the third function and to perform still image capture by the first function. A control method for an imaging device, characterized in that, when the first operation is received via the operating member while the second function is being executed, the recording of video by the second function is stopped and still image capture by the first function is not performed.

16. A computer-readable program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 14.

Images